One failure, one notification
The task looked like “remove the extra noise”. In practice the extra noise had to be proven extra first.
- What happened
- Two independent watchers covered the same ground. Every failure produced two messages, and some carried no cause at all. When there are many messages, people stop reading them and miss the one that mattered.
- Where the trap was
- The temptation is to switch the duplicates off straight away. But the two watchers covered different things: one had checks the other did not. Switching off a “duplicate” blindly would have traded noise for blindness.
- What I did
- First I wrote down, for every signal, which system would own it after the change: which rule, which threshold, which delay and where it is delivered. Only where coverage was proven did I remove the duplicate.
- How I verified it
- Three scenarios: a live failure, a synthetic event, and a negative one where no message should arrive. The third matters most: without it you cannot tell a working rule from one that fires all the time.
- The result
- Seven duplicate notifications removed; coverage for each signal was demonstrated before the removal, not after.
The general rule I have applied ever since: you may reduce notifications only after writing down who owns each signal. Otherwise it is not less noise, it is less sight.
If alerts arrive in batches and nobody reads them any more, this is fixable.