We noticed that, right after a log rotation, the data is not being indexed untill the next log rotation. That is lets say one file was rotated at 8 AM (untill which the data was already indexed). The next file is written from 8 AM to 7 PM. But this file is not indexed untill around 7 PM.
We are on Universal forwarder 7.0.3
Below is the monitoring stanza
[monitor:///opt/mapr/hadoop/hadoop/logs/*nodemanager*]
sourcetype = my_st
index = my_index
disabled = 0
ignoreOlderThan = 2h
We added `ignoreOlderThan = 2h` recently to see if it helps. But the issue still persists.
The latest file will be with `yarn-mapr-nodemanager-host_name.log` and the latest archived file be with `yarn-mapr-nodemanager-host_name.log.1`.
What is interesting is intermittently on certain servers, the current file gets indexed only at the time of its roll/archival i.e. lets say after 10-11 hours but with actual file name but not archive file name. And the issue of live/current file not getting indexed on time does **not** happen all the time. The next live file might get indexed on time. There should be ideal settings to avoid this. Any insights on this will be helpful.
Whatever Splunk says about handling log rotation files, seems to have some bug. Are we missing anything here? Please suggest.
↧