Hi,
Since upgrading to Splunk 7.1.0 from Splunk 6.5.0 I've been having issues with splunk losing access to the Web UI after some time.
I restart Splunk via /opt/splunk/bin/splunk restart and it comes back up for quite a while (usually a day) but after a period of time it will go back down again.
I've noticed that when I restart the Splunk service every time it tells me that Splunkd was not running:
splunkd 57690 was not running.
Stopping splunk helpers...
Done.
Stopped helpers.
Removing stale pid file... done.
splunkd is not running.
Splunk> The IT Search Engine.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket bro cim_modactions cim_summary endpoint_summary firedalerts history ioc main msexchange notable notable_summary os perfmon risk summary threat_activity ubaroute ueba whois windows wineventlog xtreme_contexts
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Invalid key in stanza [syslog:ubaroute] in /opt/splunk/etc/apps/Splunk_TA_ueba/default/outputs.conf, line 7: dropEventsOnQueueFull (value: 10).
Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
Waiting for web server at https://127.0.0.1:8000 to be available............... Done
I didn't see anything that stood out in Splunkd.log or Web_access.log - though in syslog I saw the following:
Out of memory: Kill process 31605 (splunkd) score 484 or sacrifice child
May 23 08:05:42 splunk kernel: [244114.815142] Killed process 31605 (splunkd) total-vm:9945680kB, anon-rss:8845420kB, file-rss:0kB
This is becoming quite an issue - any help would be appreciated.
↧