Pegel

pegel.example.org was down for 2 hours 45 minutes

A full disk took down the site, the downloads and the status page on 2026-09-25. The download log had not been rotated since a move a month earlier.

Status
Resolved
Severity
Critical

The site and the downloads were down from 21:10 to 23:55 CEST on 2026-09-25, and the status page until 00:40. Nobody could install or upgrade Pegel, and the page that should have said so was down with them. The disk of the VPS was full: the access log of the downloads had not been rotated since I moved them on 2026-08-30, and it had grown to 41 GB.

Timeline

  • 2026-09-25 21:10 CEST The disk of the VPS fills up. The site and the downloads answer 500.
  • 21:12 Pegel alerts for pegel.example.org and the status page.
  • 23:35 Investigating: We read the alert and log in to the VPS.
  • 23:41 Identified: /var is full. The download log is 41 GB.
  • 23:55 Monitoring: The log is compressed and moved. The site and the downloads answer again.
  • 2026-09-26 00:40 The status page is back, restored from the backup of 2026-09-25 03:00. Its database broke while writing to the full disk.
  • 01:30 Resolved: Every check green for 50 minutes.

What went wrong

The alert came on time. We read it at 23:35, after an evening away from our laptops, because Pegel has no on-call rota, and with two people it will not get one. The move on 2026-08-30 copied the download server’s config but not its logrotate entry, and nothing told us the disk was filling.

$ df -h /var
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        50G   50G     0 100% /var

What changed

  • The download log is rotated daily and kept for 14 days.
  • The status page runs on the second VPS, so it cannot fall with the site.
  • Pegel checks the disk of the VPS through its health URL and alerts at 80 %.

Comments

Buy Logbuch