Monitoring that lied
Here I was investigating my own mistake rather than somebody else's. Those write-ups are the more useful kind, which is why this one is on the site.
- What happened
- The observation server lost network connectivity and declared five healthy sites down. Eighteen minutes of false alarms. The sites were opening for users the entire time.
- Why the obvious fix was wrong
- The tempting rule was “if all of them fell at once, it must be us”. But every watched address pointed at the same server, so “all of them fell” is indistinguishable from that server genuinely failing. Such a rule would have muted a real outage.
- What I did
- Added a signal that does not depend on the watched sites: connectivity checks against several independent external providers. Then verified they really were independent — one of them turned out to share infrastructure with another and was replaced. And wrote the rule so that it stays silent when data is missing instead of muting other signals.
- How I verified it
- With tests across four scenarios, including “no data at all”. Then I ran the same tests against the old version of the rule and confirmed they fail on it: a test suite that never goes red is indistinguishable from no tests.
- The result
- False alarms of this class stopped, and the rule is no longer something taken on trust.
The right answer to “the monitoring lied” was not to nudge a threshold but to find a signal worth trusting. A threshold would have hidden the symptom and kept the cause.
If you have alerts nobody believes any more, this is exactly the same task.