> ## Content Index
> Fetch the complete content index at: https://logbuch-changelog.nichedthemes.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Alert mails were held for four hours
- URL: https://logbuch-changelog.nichedthemes.com/alert-mails-were-held-for-four-hours/
- Published: 2026-10-26T09:00:00.000Z
- Updated: 2026-10-26T09:00:00.000Z
- Description: On 2026-10-24 our mail relay accepted every mail and delivered none from 09:20 to 13:20 CEST. Webhook alerts reached us the whole time.
- Author: David Osei
- Tags: Pegel, Incidents, #incident, #resolved, #minor, #Import 2026-11-19 07:37

Our own alert mails stopped for four hours on Saturday 2026-10-24\. The relay that sends them and the release announcements accepted every mail and delivered none, because its upstream provider had moved to a certificate chain the relay did not trust. The queue check alerted at 09:37, by webhook and by mail. The mail copy waited in the queue it was warning about.

## Timeline

- 2026-10-24 09:20 CEST The upstream provider switches to a new certificate chain. The relay queues every mail.
- 09:37 The queue check fails from both probes. The webhook alert arrives, the mail alert joins the queue.
- 11:50 Investigating: We read the webhook alert and log in to the relay.
- 11:52 Identified: The relay log shows `certificate verify failed` for every delivery since 09:20.
- 12:40 The relay’s CA bundle is updated and the queue starts to drain.
- 13:20 Resolved: The last of 37 queued mails is delivered.

## What went wrong

The relay used a CA bundle we had copied onto it by hand in 2025 and never updated. Nobody read the webhook alert until 11:50, because it was Saturday and the alert said "queue older than 15 minutes", which had always cleared on its own.

## What changed

- The relay takes its CA bundle from the system packages and updates with them.
- Alerts about the relay go out by webhook only.
- The queue check alerts after 30 minutes instead of 15, so it alerts less often and means it when it does.