PostMembers only
Sixty days of Pegel watching itself
Our site, the status page and the mail relay from 2026-09-14 to 2026-11-12: three red days, uptime between 99.72 % and 99.81 %. The numbers per target are for members.
1 min
Outages of pegel.example.org, the downloads and our mail relay, with timelines and what we changed afterwards.
PostMembers only
Our site, the status page and the mail relay from 2026-09-14 to 2026-11-12: three red days, uptime between 99.72 % and 99.81 %. The numbers per target are for members.
1 min
Incident
For 13 hours after the release, pegel upgrade on arm64 stopped with a checksum error. The binary was fine, the checksum file was not. Two community packages still carry the old one.
Monitoring
Major
From the release at 18:04 CET on 2026-11-09 until 07:25 the next morning, The release job had no order between building and publishing. For 0.8.2 the arm64 build happened to finish first. For 0.9.0 it failed once on a runner that lost its network, restarted, and finished four minutes after the checksums were out. Nobody downloaded an arm64 binary that evening to check, so the first person to notice was a reader the next morning. The packages of two community distributions copied the checksum file within the first hour. Until their maintainers update them, installing from those packages fails the same way. The files on our downloads page are correct, and this check proves it on your machine:pegel upgrade on arm64 stopped with "checksum mismatch". The binary was fine. The checksum file next to it held the hash of an arm64 build that the release job had cancelled and started again, because the job wrote SHA256SUMS as soon as it could. amd64 was not affected.Timeline
SHA256SUMS lists the hash of the cancelled arm64 build.pegel upgrade on a Raspberry Pi stops with "checksum mismatch".SHA256SUMS signed and uploaded. pegel upgrade on arm64 works again.What went wrong
What we changed
SHA256SUMS (in 0.9.1).pegel upgrade prints both hashes on a mismatch, so a report tells us at once which file is wrong.Still open
$ sha256sum -c SHA256SUMS --ignore-missing
pegel_0.9.0_linux_arm64.tar.gz: OK
Incident
On 2026-10-24 our mail relay accepted every mail and delivered none from 09:20 to 13:20 CEST. Webhook alerts reached us the whole time.
Resolved
Minor
Our own alert mails stopped for four hours on Saturday 2026-10-24. The relay that sends them and the release announcements accepted every mail and delivered none, because its upstream provider had moved to a certificate chain the relay did not trust. The queue check alerted at 09:37, by webhook and by mail. The mail copy waited in the queue it was warning about. The relay used a CA bundle we had copied onto it by hand in 2025 and never updated. Nobody read the webhook alert until 11:50, because it was Saturday and the alert said "queue older than 15 minutes", which had always cleared on its own.Timeline
certificate verify failed for every delivery since 09:20.What went wrong
What changed
Incident
A full disk took down the site, the downloads and the status page on 2026-09-25. The download log had not been rotated since a move a month earlier.
Resolved
Critical
The site and the downloads were down from 21:10 to 23:55 CEST on 2026-09-25, and the status page until 00:40. Nobody could install or upgrade Pegel, and the page that should have said so was down with them. The disk of the VPS was full: the access log of the downloads had not been rotated since I moved them on 2026-08-30, and it had grown to 41 GB. The alert came on time. We read it at 23:35, after an evening away from our laptops, because Pegel has no on-call rota, and with two people it will not get one. The move on 2026-08-30 copied the download server’s config but not its Timeline
/var is full. The download log is 41 GB.What went wrong
logrotate entry, and nothing told us the disk was filling.$ df -h /var
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 50G 50G 0 100% /varWhat changed