Background: I'm working off of a Splunk system that was initially installed and configured as a development / testing system. It hasn't been maintained or touched in several months.
For the most part, the system seems to be behaving properly. I can run searches and there are results. There appears to be sufficient system resources (CPU, RAM, free disk space).
In using the system to test and see specific log messaging that is occurring in an application, I can see the following problem. Here is the repeatable scenario:
1) at time 12:00pm, host1 sends a single Splunk message through UDP; the message is sent successfully as verified through Wireshark on host1
2) at time 12:05pm, searching for all messages in Splunk using either relative time or real time will not yield the message from Step1. I can wait over for over 10 minutes and the message will not show up. I can however see messages sent through from other hosts.
3) at 12:10pm, host1 sends another single Splunk message through UDP; the message is sent successfully as verified through Wireshark on host1
4) this time, searching for all messages from host1 in Splunk using either relative or real time will yield only the first message that was sent in Step1. The message sent in Step1 shows up with the proper timestamp value of 12:00pm. The event is properly processed and is now searchable.
At this point, the message sent in Step3 will not show up until another message from host1 seems to "push" it out.
Any ideas on what could be causing the issue and how to resolve it?
I have also verified that the Splunk system time is correct.
↧