Hello,
I am trying to create a search that detects when the total volume of events (the sum) from specific devices has a significant decrease or increase on a particular day. The average number of events that each device sends to splunk depends on the day of the week (weekend days send less events than weekdays, for example). What I would like to do is to use the mean and standard deviation of the sum of events on each weekday (monday, tuesday, etc), and if the sum of events on a specific day for a device in my list is more than, say, 2.5 standard deviations from the mean for that day, alert me.
A sample process for achieving this in my head:
1. Compute the sum of the events from each device on each weekday.
2. Use the information in step 1 to compute the standard deviation of the sum of of events for each device on each weekday. Keep the stdevs for each device in a lookup, and re-compute them (do step 2) each time the search is ran (daily) to keep a constantly updated stdev for each device.
4. Compute the mean of the sum of events from each device on each weekday. Keep this information in a lookup as well, and update it daily.
5. The devices that I am interested in are in a lookup table. We'll call it my_devices.csv. Search for events in the data (sourcetype=foo) that are from the devices in my_devices.csv by matching on the device IP address or host name field.
6. Compute the [z-score][1] for each device, each day. The z-score is the number of standard deviations your value is from the mean: z = (sum of events for that day - average sum of events for that weekday)/(standard deviation of number of events for that weekday). For example on Monday the 8th (picking a random day) for a particular device in my lookup: z = (sum of events for Monday the 8th - mean sum of events over all Mondays)/(standard deviation of the sum of number of events over all Mondays)
I would like for this to run daily, and to run on the data from the previous weekday. So, if it runs on Wednesday, it should be looking at the total number of events from Tuesday. If, on Wednesday, the z-score is more than 2.5 standard deviations from the mean number of events for that device on Tuesdays, alert me. I am unsure of how to bucket the sum and mean calculations into weekdays and how to perform those calculations for individual devices within my devices lookup table, as well as what the general structure of the search should be.
[1]: http://en.wikipedia.org/wiki/Standard_score
↧