Splunk is running on a VM with 6 virtual cores, 24 GB RAM and windows OS. We have installed Splunk Universal Forwarders one two Linux systems and we have also installed the nix add on. We have enabled selective scripted inputs, namely, ps, iostat, vmstat, top and cpu. The scripts run every minute, but the number of events indexed in case of ps, each time it runs, is over 1000. The number of events collected over the past week is quite huge. If we ever run a simple index=os search over the week's time, the RAM utilization of the splunk instance shoots to nearly 100% and the application seems to hang.
The searches do run for much shorter time ranges, like the last one hour.
I would like to understand why this is happening, because we often do run searches on bigger indexes and although it take a long time, it doesn't crash the search. We are unable to identify the root cause of this problem. Any idea?
↧