Hi,
In few of our LPARs, we have observed a pattern that occasionally the underlying AIX process (/usr/bin/topas_nmon) called by NMON Performance Monitor for Unix and Linux Systems do not get stopped after their data collection time period is completed. Since Splunk calls nmon to collect LPAR stats every 5 minutes, the number of nmon processes keep on growing in this situation where the completed processes are not terminated. We noticed that each of the running process was consuming around 40-50MB of page space memory.
On 1st Sep, we had 300 of theses processes stuck in one of our LPAR, each consuming 40~50MB of memory totaling around 15GB of Page Space memory usage.
In other situations, the page space memory was exhausted and we lost the ability to ssh to the box and the only solution was to restart the LPAR.
**Symptoms:**
* Number of files start to grow up in nmon_temp directory. Under normal circumstances, there are no files in these directories.
* File descriptors get exhausted: Under normal circumstances, we have 10-15 open file descriptors, but in this case, all the file descriptors available for Splunk are exhausted and we start seeing an infinite number of these messages in the splunkd.log : "Resetting fd to re-extract header"
* Example of the command:
myuser 33096084 1 0 08:12:14 - 0:02 /usr/bin/topas_nmon -f -T -A -d -K -L -M -P -^ -s 60 -c 5 -youtput_dir=/splunkfwd/splunkforwarder/etc/apps/TA-nmon/var/nmon_temp/au02qdb201teax2 -ystart_time=08:12:12,Sep13,2016
Please note that:
* TA-nmon does run successfully on these boxes some times
* We have another /usr/bin/topas_nmon running by infrastructure team on these boxes that runs without issues
* AIX version: 6100-09-04-1441
* Splunk forwarder version 6.1.3 (Universal forwarder)
* TA-nmon/metadata/local.meta:version = 6.1.3, modtime = 1413521379.326427000
Is this a known behaviour? Or is there a clean up job that fails to execute occasionally? I will appreciate your help
Thanks
↧