Sometimes (like on holidays), I want to disable an alert for a period of time so that it doesn't fire and cause operators to panic. Usually, we do one of two things:
1. Manually disable the alert on the day we want it to stop running, then manually re-enable it as soon as we want it to run again. This often requires waiting until the end of the day before a holiday, then coming in as soon as possible the following work day and remembering to re-enable everything.
2. Tweak the cron schedule so the search doesn't run on the days of the week the holidays fall on. This is less transparent and still requires someone to manually alter the alert's schedule.
I'm wondering if there's a better solution, maybe something like a snooze function where we can say ahead of time that we don't want the alert to run on days x, y, z, but then resume normal functionality. This would be more like a planned outage than reactive throttling.
↧