New Splunk user here. I've searched and seen a lot of questions and answers that were similar to mine, but none that were close enough to help me figure out what to do in my exact scenario. So, here goes.
I have a number of hosts that have a log file that contains a heartbeat entry that typically occurs every 2 minutes. I created a chart that shows a list of each of the server hostnames along with the time of the latest heartbeat log line and a calculated value of the time since that latest heartbeat was found. This is so you can easily see if it's way over 2 minutes and know you've got some kind of issue with this server.
My challenge now is that I'd like to set up an alert for when the latest time of any server's heartbeat is greater than 30 minutes so a user can get an email with that nice chart embedded, can easily see which server is affected, and how long it's been since the last heartbeat. I just can't figure out how I can do that with the data in chart form... I know the solution is probably easy, it's just eluding me for some reason.
Here's what my search currently looks like, with some index/sourcetypes removed since it's not relevant:
"Sending heartbeat request" earliest=-4h | eval time_elapsed = round((now() - _time)/60,0) | stats latest(_time) AS "Latest Heartbeat", latest(time_elapsed) AS "Minutes since last heartbeat" by host | convert ctime("Latest Heartbeat")
And here's what my output looks like:
![Chart screenshot][1]
I've tried things like added a WHERE clause, but it just doesn't seem to output in the way I need it to. I'm hoping someone's got an easy solution. I'm sure there is one, and my noob-ness is just getting in the way. Thanks!!!
[1]: /storage/temp/136279-screenshot10-06-2016-154906.jpg
↧