i have multiple applications that place login information (Logon Date/Time, Logoff Date/Time, userid, etc.) into existing CSV files (one per application). I am monitoring these files, but when they are indexed, the old data is reindexed, so I have multiple events per logon. This is causing errors in reporting (I shouldn't have to do a `dedup`) and is ballooning the size of each index (wasting disk space).
My understanding is that when a file being monitored, a beginning and end CRC is generated to fingerprint the file along with a Seek Address.
**Documentation states:**
"A matching record for the CRC from the file beginning in the database, the content at the Seek Address location matches the stored CRC for that location in the file, and the size of the file is larger than the Seek Address that Splunk Enterprise stored. While Splunk Enterprise has seen the file before, data has been added since it was last read. Splunk Enterprise opens the file, seeks to Seek Address--the end of the file when Splunk Enterprise last finished with it--and starts reading the new from that point."
I take this to mean that existing events are not added and only new events are indexed. This isn't happening in my case.
I have read the questions concerning "duplicate data" and two settings keep appearing. One is "followTail", reading the doc for this, i see "WARNING: Use of followTail should be considered an advanced administrative action." and "DO NOT leave followTail enabled in an ongoing fashion.". This doesn't look to be a good fit for my problem.
The second is "crcSalt". The question I have on that setting is if I do set it, does that ignore the Seek Address causing the entire file to be indexed, which is where I am now.
Thank you in advance for any help that can be provided.
Scott
↧