I have an alert that looks like this:
index=test Operation="Add member to role." | eval lag_sec=_indextime-_time | table UserId,Operation,lag_sec,_time
Normally, if I was receiving and indexing the data in near real time, I would setup the alert to run say every 15 minutes, and set the search window to 15 minutes, and I wouldn't get any duplicate alerts. In this case, however, the search is scheduled to run every 15 minutes, but the search window is 12hrs, because the system sending Splunk the data to be indexed can be delayed up to 12 hours.
I created a Splunk case for this but they were not helpful. The only thing I could think of was maybe to use lookup tables. Where, I could save the result of a search to a lookup table and then have the search look at the lookup table for that userid and timestamp and if they match, it's a duplicate and don't alert on it.
Any ideas how else I could do this? I'm trying to eliminate duplicate alerts, and again the whole problem is that the data being indexed can be delayed up to 12hrs. Thank you in advance.
↧