Hi Splunkers!
I've been re-writing a correlation search from Splunk Enterprise Security, "Anomalous Audit Trail Activity Detected". I wrote my version to take OS restarts and shutdowns into account, since the WinEventLog service gets stopped any time the system shuts down. Once the search was written, I went to test it!
I stopped the event logging service on one of our dev servers and waited half an hour. I checked Splunk and saw NO events about the event logging service shutdown whatsoever. I started the event logging service and still got no events mentioning the shutdown OR startup. I was able to see these events in the Windows Event Log Viewer, but again, they were not in Splunk. I restarted the forwarder on the server and BOOM, the events showing the service shutdown appeared.
This is worrisome. We can't see actual event logging service shutdown events until a forwarder restart - though MAYBE Splunk would have sorted itself out if I waited long enough...? Not sure.
Has anyone found a solution to this? I'm thinking maybe we need some intelligent logic, like "if event logging service is stopped, restart Splunk" and "if event logging service is started, restart Splunk". I never want to SHUT DOWN the forwarder so setting a 1-to-1 service dependency wouldn't work (if event logging service shuts down, forwarder shuts down).
I'll consult with my Windows server team, but I'd love any suggestions.
Thanks!!
↧