I currently have the following setup.
3 x search heads ( 8 cpu, 16gb memory)
2 x indexer ( 8 cpu, 16gb)
Currently I'm only indexing around 10GB per day worth of data, 80% is from the NetApp application "splunk app for netapp". I have datamodel acceleration enabled with a summary of 1 month history on a cron of every 5 minutes.
Now currently the datamodel acceleration runs for about 2-3 minutes and during that time, the memory usage of the splunkd process reaches 16gb and causing OOM kernel errors that kills the process. This causes splunk to crash on the indexer. I've tried the suggestion if implementing cgconfig rules that limits the splunk user to 12gb maximum memory usage but I find this to be a workaround at best that killing splunk child processes shouldn't be needed.
To see how much memory it could use, I created a 3rd indexer with double the resources of the original 2 (so 16 cpu and 32gb memory). In this case, when the datamodel acceleration job was running it was using 32GB and causing OOM errors to appear in /var/log/messages.
My questions:
1. Has anyone else seen such high memory usage on indexes when datamodel acceleration runs?
2. The splunk app for netapp datamodel is quite large which hundreds of fields. Does the amount of fields in the datamodel equate to higher memory usage during datamodel updates?
3. Does reducing the datamodel span (from 1 month to say 7 days) have an impact on memory usage during datamodel updates?
The only thing I can think of right now is creating a custom datamodel with the fields that I need. If anyone has any solutions to try other than a new datamodel, I'm all ears.
↧