Hi
So we have a server which writes out thousands of files a day.
Over course of two months we can have 70K+ files.
We cant enforce a aggressive archival policy for the project team due to some other retention constraints.
There are also dat.gz files which may have even more CPU hit if we take into account the thousands of older files.
ignoreOlderThan = [s|m|h|d]
* Causes the monitored input to stop checking files for updates if their
modtime has passed this threshold. This improves the speed of file tracking
operations when monitoring directory hierarchies with large numbers of
historical files (for example, when active log files are colocated with old
files that are no longer being written to).
* As a result, do not select a cutoff that could ever occur for a file
you wish to index. Take downtime into account!
Suggested value: 14d , which means 2 weeks
* A file whose modtime falls outside this time window when seen for the first
time will not be indexed at all.
* Default: 0, meaning no threshold.
if ignoreOlderThan is used in the inputs.conf does this process the input at monitor time or there is a internal blacklist setup
I think there might be an internal list because below is true for ignoreOlderThan from what I read in other answers:
***Once a file is ignored, it never comes back to being monitored even if its timestamp is updated. The only way to stop this is to remove that feature and restart Splunk.***
**Now the query is : How will this affect performance of Splunk forwarder over time ( with thousands of blacklists being setup internally?)**
↧