Hi all,
Have you ever seen a UF missing events? I’ve observed some of our UF’s missing ~8 seconds of events and then picking up halfway through the event they reach. The gaps are creating some muddy data and it doesn’t seem to be limited to one server, I’ve got a list of 100 or so across all of our environments and corresponding Splunk clusters.
Here's a 3 line example of what Splunk is seeing in the source(/app/search/show_source?blah). I've been able to manually confirm that there is a gap and plenty of logs between.
2017-12-03 22:25:37 GET /Something/Something/1 from=2017-12-02&to=2017-12-04 80 - 0.0.0.0 HTTP/1.1 - - Some.url.was.here.com.au 200 0 0 00000 000 00 - HasedKeyWasHere ServiceName -
0.0.0.0 HTTP/1.1 - - ome.url.was.here.com.au 200 0 0 000 000 0 - HasedKeyWasHere ServiceName -
202017-12-03 22:25:45 GET /Something/Something/1 from=2017-12-02&to=2017-12-04 80 - 0.0.0.0 HTTP/1.1 - - Some.url.was.here.com.au 200 0 0 00000 000 00 - HasedKeyWasHere ServiceName -
I've tried this with and without line breaking logic to see if it would make any difference in the props.conf with no success. Which is not entirely surprising in hindsight.
It should be worth mentioning that these are all IIS logs being forwarded to a 6 peer node cluster with no heavy forwarders inbetween.
↧