We recently upgraded to a 3 node shcluster, of 8 core boxes. Our limits.conf across the cluster is:
max_searches_perc = 50
base_max_searches = 10
max_searches_per_cpu = 10
So according to some splunk math:
max_searches_per_cpu × number of CPUs + base_max_searches = Total number of searches
10 X 8 + 10 = 90 (x 3 SHs) = 270 concurrent searches
I've recently notice we've started to have some of our scheduled jobs skipped for some unknown reason. So I started some digging and discovered via this search:
index=_internal source=*metrics.log group=searchscheduler | timechart partial=false span=1m sum(dispatched) AS Started, sum(skipped) AS Skipped by splunk_server | table _time Started*
That ALL our scheduled jobs were running on ONE search head. I assumed that which SHClustering these scheduled searches would be divided up across the cluster based on the load of each search head. As this is NOT the case, how can I reasonably expand the cluster to allow for our increased number of scheduled reports?
![alt text][1]
[1]: /storage/temp/81219-screen-shot-2016-01-18-at-103838-am.png
↧