Monitoring from scratch without touching production

What happened

There were no metrics, logs sat in different places, and service health was judged by whether the site opened. That means every degradation was noticed by users rather than by the owner.

What was not allowed

Putting the monitoring on the same server it watches: when that server dies, the monitoring dies with it and you learn nothing. And no new doors could be opened to the internet.

What I did

A separate observation server collecting metrics and logs from five services, the host and the containers, with dashboards and alerts in a messenger. Data is PUSHED out of production over a closed channel rather than pulled in from outside — which is why production gained no new listening port.

How I verified it

With a rehearsed failure and recovery: I stopped the watched service, waited for the alert, brought it back and waited for the matching recovery message. Until both halves arrive, the check is not passed.

The result

The production server opened no new ports to the internet and restarted no service.

The rehearsed failure is not a formality. Monitoring that has never fired is indistinguishable from monitoring that is switched off, and finding that out during a real outage is the most expensive way to learn it.