Here are some pieces of info that may be relevant:
- The sourcetype in question shows no data after midnight on October 31st when searching
- Setup: 1 Splunk server, no replication or anything, 40 nodes (4 in question) reporting to splunk
- Splunk server version: 6.5.0 (installed on 10/25/16)
- Splunk server has more than 20% free disk on all drives
- All other sourcetypes on the splunk server are working, even sourcetypes that dump into the same index as the broken sourcetype
- No changes have been made on the splunk server or the nodes
- The metrics.log on the splunk server shows data coming in for that sourcetype from the 4 nodes in question
- The splunk forwarder logs on each of our nodes shows that it is sending data into the splunk server using the sourcetype
- Each node in question is reporting data to the splunk server on additional sourcetypes and they are searchable in splunk
Any ideas would be greatly appreciated. I've restarted the splunk server (full windows reboot) as well as restarting the splunk forwarder on the nodes. It seems like it's date related but none of the other sourcetypes seem to be affected.
↧