pegel.example.org was down for 2 hours 45 minutes
A full disk took down the site, the downloads and the status page on 2026-09-25. The download log had not been rotated since a move a month earlier.
- Status
- Resolved
- Severity
- Critical
- Started
- Resolved
- Duration
The site and the downloads were down from 21:10 to 23:55 CEST on 2026-09-25, and the status page until 00:40. Nobody could install or upgrade Pegel, and the page that should have said so was down with them. The disk of the VPS was full: the access log of the downloads had not been rotated since I moved them on 2026-08-30, and it had grown to 41 GB.
Timeline
- 2026-09-25 21:10 CEST The disk of the VPS fills up. The site and the downloads answer 500.
- 21:12 Pegel alerts for pegel.example.org and the status page.
- 23:35 Investigating: We read the alert and log in to the VPS.
- 23:41 Identified:
/varis full. The download log is 41 GB. - 23:55 Monitoring: The log is compressed and moved. The site and the downloads answer again.
- 2026-09-26 00:40 The status page is back, restored from the backup of 2026-09-25 03:00. Its database broke while writing to the full disk.
- 01:30 Resolved: Every check green for 50 minutes.
What went wrong
The alert came on time. We read it at 23:35, after an evening away from our laptops, because Pegel has no on-call rota, and with two people it will not get one. The move on 2026-08-30 copied the download server’s config but not its logrotate entry, and nothing told us the disk was filling.
$ df -h /var
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 50G 50G 0 100% /varWhat changed
- The download log is rotated daily and kept for 14 days.
- The status page runs on the second VPS, so it cannot fall with the site.
- Pegel checks the disk of the VPS through its health URL and alerts at 80 %.
subscribe
New entries and the next chapter by mail, nothing else.