Hi,
We are noticing performance issues on an Indexer. The server in question is being used to index network data so index rate is high.
Type: Physical
CPU - 32
Mem - 32 GB
Splunk Version - 6.4.1
Splunk Build - debde650d26e
Quite frequently this server stops responding to the search heads and also doesn't allow anyone to login directly. Splunk Web goes down as well. On the search head, following error is seen under peer status:
Error [00000080] Failed 12 out of 11 times.REST interface to peer is taking longer than 5 seconds to respond on https. Peer may be over subscribed or misconfigured. Check var/log/splunk/splunkd_access.log on the peer
Although, Indexing never stops and once the server normalizes, we can search on indexed data.
Attached is a screenshot of "top". ![alt text][1]
This shows the usage while server is functioning normally with all connections established. During peak, all memory usage goes to max 100%.
Has anyone experienced similar issues with indexers? And is there a way I could dive in deeper to see what is happening when the server stops responding?
Thanks,
~ Abhi
[1]: /storage/temp/160216-indexer-top.png
↧