Alert mails were held for four hours
On 2026-10-24 our mail relay accepted every mail and delivered none from 09:20 to 13:20 CEST. Webhook alerts reached us the whole time.
- Status
- Resolved
- Severity
- Minor
- Started
- Resolved
- Duration
Our own alert mails stopped for four hours on Saturday 2026-10-24. The relay that sends them and the release announcements accepted every mail and delivered none, because its upstream provider had moved to a certificate chain the relay did not trust. The queue check alerted at 09:37, by webhook and by mail. The mail copy waited in the queue it was warning about.
Timeline
- 2026-10-24 09:20 CEST The upstream provider switches to a new certificate chain. The relay queues every mail.
- 09:37 The queue check fails from both probes. The webhook alert arrives, the mail alert joins the queue.
- 11:50 Investigating: We read the webhook alert and log in to the relay.
- 11:52 Identified: The relay log shows
certificate verify failedfor every delivery since 09:20. - 12:40 The relay’s CA bundle is updated and the queue starts to drain.
- 13:20 Resolved: The last of 37 queued mails is delivered.
What went wrong
The relay used a CA bundle we had copied onto it by hand in 2025 and never updated. Nobody read the webhook alert until 11:50, because it was Saturday and the alert said "queue older than 15 minutes", which had always cleared on its own.
What changed
- The relay takes its CA bundle from the system packages and updates with them.
- Alerts about the relay go out by webhook only.
- The queue check alerts after 30 minutes instead of 15, so it alerts less often and means it when it does.
subscribe
New entries and the next chapter by mail, nothing else.