Several weeks ago our _internal index suddenly started receiving a massive amount of entries per day. When I do a search for the month of July, Splunk has an average of **49,000** events per day, with individual days going anywhere from a couple of hundred to 210,000. Since August 12, which was a Saturday and usually a slow day (we're mostly a M-F shop) I'm seeing approximately **50,000,000 to 60,000,000** events in the _internal per day. Since internal doesn't use our license, it's really only a problem for disk space and my sanity, but since I have a "DO NOT DELETE ANYTHING" mandate on my Splunk data so we have to retain everything that Splunk creates, the disk space is an issue.
No changes occurred on August 12th or near there on either our enterprise or on the Splunk server.
Sourcetype is 99+% splunkd. Host source is about 5% from the Splunk server, and everything else evenly distributed.
We have a 30 GB license and normally use about 20-25 of that. All Windows environment, forwarders deployed on all the endpoints. It's been running for 10 months and this is the first time I've seen this happen.
Here's a sample of what we're seeing 50-60 million times per day. I did this with 1:100,000 sampling on to explain the time discrepancies.
8/31/17
2:59:55.602 PM
08-31-2017 14:59:55.602 -0400 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=0.021421370967741934, eps=0.06451612903225806, kb=0.6640625, ev=2, avg_age=1, max_age=1
host = XXXXXXX index = _internal source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log sourcetype = splunkd
8/31/17
2:59:37.874 PM
08-31-2017 14:59:37.874 -0400 INFO Metrics - group=queue, name=fschangemanager_queue, max_size_kb=5120, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
host = XXXXXXXX index = _internal source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log sourcetype = splunkd
8/31/17
2:56:19.135 PM
08-31-2017 14:56:19.135 -0400 INFO Metrics - group=tailingprocessor, name=tailreader0, current_queue_size=0, max_queue_size=1, files_queued=2, new_files_queued=0, fd_cache_size=0
host = XXXXXXX index = _internal source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log sourcetype = splunkd
8/31/17
2:52:11.697 PM
08-31-2017 14:52:11.697 -0400 INFO Metrics - group=queue, name=parsingqueue, max_size_kb=512, current_size_kb=0, current_size=0, largest_size=2, smallest_size=0
host = XXXXXX index = _internal source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log sourcetype = splunkd
8/31/17
2:47:45.235 PM
08-31-2017 11:47:45.235 -0700 INFO Metrics - group=pipeline, name=parsing, processor=send-out-light-forwarder, cpu_seconds=0, executes=3, cumulative_hits=61223
host = XXXXXXXX index = _internal source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log sourcetype = splunkd
8/31/17
2:47:41.399 PM
08-31-2017 11:47:41.399 -0700 INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=0.24700080756584217, instantaneous_eps=0.09677164658108241, average_kbps=0.7696573743954075, total_k_processed=5602707, kb=7.6572265625, ev=3
↧