Every so often, our search head cluster (6.5.2) switches captain. Whenever this happens, or possibly for other reasons, our realtime scheduled searches get replicated. We have three search heads, so at the moment we have three copies of each of our realtime searches. Because of this, we get flooded with extra alerts in the form of emails, or other actions that these realtime searches were configured to initiate.
Is there a way to ensure that only one copy of each search gets scheduled at a time (even for realtime searches)? Ideally, we also want the searches to fail over to an alternate search head of that one goes down. Maybe this is just a bug.
Also, what is best way (balancing for speed and safety) to clean up the dozens of duplicate realtime jobs when this does happen? If the only way is to restart one or more search heads, then so be it.
↧