The disk was filling up. It was the logs
Free space had been shrinking for weeks. The applications had nothing to do with it, and that became clear faster than expected.
- What happened
- System journals had taken 4 GB and kept growing. When space runs out, a server stops coping suddenly and completely: writes fail, and services unrelated to logging go down with them.
- What I measured
- First, where the space was actually going, by directory rather than by guess. Then how fast each source was growing. Without that second number you cannot tell whether you have a week or a day.
- What I did
- Found the source of the growth and set hard journal limits, plus a rotation that actually runs rather than merely exists in the configuration. Verified that the rotation had done its work, not just that the job had started.
- How I verified it
- By directory size before and after, by the state of the rotation timer, and by the restart counters of the applications staying at zero. That last one matters: fixing a disk by knocking over a service is not a fix.
- The result
- 4.0 GB → 31 MB. Growth stopped, the limits hold, and no application was restarted.
The wider lesson is worth more than the fix itself: “running out of space” is almost never about the disk. It is about nobody watching the rate of growth until it was too late.
If your free space is shrinking and the cause is unclear, this is a one-session job.