Existential question here... :)
What is the appropriate mechanism in Splunk to have multiple (potentially 100s) of alerts that are based on latest events rather than realtime or timeframe searches while keeping our Splunk deployment sane and simple? (Is it even possible?)
Example: need an alert when a volume (disk) breaches an 80% used space threshold, and need it within 30 seconds of when Splunk gets an event. (Then similar alerts for NAS and SAN volumes, CPU, memory, interface utilization and a whole bunch of other metrics.) Setting up a few dozens such realtime searches and respective alerts brings our cluster to its knees. Attempting to set up auto-refreshing and *fast* dashboards with the same metrics simply knocks it out. Whereas doing something like this in Solarwinds or Datadog - peace of cake, including statistics-based metrics (e.g. if a metric exceeds a 30-minute baseline by more than 20% over the last 3 minutes).
Is Splunk not the right product for the task? If it is, what is the technical term for my problem, how can be solved in Splunk, and would you be so kind to point me to where it's discussed?
Thanks!
↧