Pegel

Alert mails were held for four hours

On 2026-10-24 our mail relay accepted every mail and delivered none from 09:20 to 13:20 CEST. Webhook alerts reached us the whole time.

Status
Resolved
Severity
Minor

Our own alert mails stopped for four hours on Saturday 2026-10-24. The relay that sends them and the release announcements accepted every mail and delivered none, because its upstream provider had moved to a certificate chain the relay did not trust. The queue check alerted at 09:37, by webhook and by mail. The mail copy waited in the queue it was warning about.

Timeline

  • 2026-10-24 09:20 CEST The upstream provider switches to a new certificate chain. The relay queues every mail.
  • 09:37 The queue check fails from both probes. The webhook alert arrives, the mail alert joins the queue.
  • 11:50 Investigating: We read the webhook alert and log in to the relay.
  • 11:52 Identified: The relay log shows certificate verify failed for every delivery since 09:20.
  • 12:40 The relay’s CA bundle is updated and the queue starts to drain.
  • 13:20 Resolved: The last of 37 queued mails is delivered.

What went wrong

The relay used a CA bundle we had copied onto it by hand in 2025 and never updated. Nobody read the webhook alert until 11:50, because it was Saturday and the alert said "queue older than 15 minutes", which had always cleared on its own.

What changed

  • The relay takes its CA bundle from the system packages and updates with them.
  • Alerts about the relay go out by webhook only.
  • The queue check alerts after 30 minutes instead of 15, so it alerts less often and means it when it does.

Comments

Buy Logbuch