Post
Release day
A signed tag starts everything else: two builds, the checksum file and the release page. Eight seconds of it, looped.
1 min
Everything about Pegel: releases, incidents and how two people build and run it.
Post
A signed tag starts everything else: two builds, the checksum file and the release page. Eight seconds of it, looped.
1 min
Changelog
Pegel refuses a config that asks more probes to agree than it has. The release job writes checksums after every build has finished.
pegel check now refuses a config whose agree is larger than the number of probes. In 0.9.0 such a config started without a word and could never alert, because one probe cannot agree with itself. Two people ran into it in the first three days, both on a Raspberry Pi with a single probe.Fixed
agree = 2 and one [[probe]] started and never fired an alert. pegel check and pegel run stop and name the line to fix.[[probe]], agree defaults to 1 again, as the notes of 0.9.0 said.pegel probe retried a refused token every second. It waits 30 seconds between attempts now and logs why.Changed
SHA256SUMS after the last build has finished. Details in the checksum incident.probes := max(1, len(cfg.Probes))
if cfg.Defaults.Agree > probes {
return fmt.Errorf("%s line %d: agree = %d needs %d probes, found %d",
cfg.Path, cfg.Line("defaults.agree"), cfg.Defaults.Agree, cfg.Defaults.Agree, probes)
}$ pegel check --config pegel.toml
error: pegel.toml line 3: agree = 2 needs 2 probes, found 1
PostMembers only
Our site, the status page and the mail relay from 2026-09-14 to 2026-11-12: three red days, uptime between 99.72 % and 99.81 %. The numbers per target are for members.
1 min
Post
Alert routes, DNS checks and a container image for the probe come next. A plugin system, a hosted Pegel and an app do not.
1 min
Incident
For 13 hours after the release, pegel upgrade on arm64 stopped with a checksum error. The binary was fine, the checksum file was not. Two community packages still carry the old one.
Monitoring
Major
From the release at 18:04 CET on 2026-11-09 until 07:25 the next morning, The release job had no order between building and publishing. For 0.8.2 the arm64 build happened to finish first. For 0.9.0 it failed once on a runner that lost its network, restarted, and finished four minutes after the checksums were out. Nobody downloaded an arm64 binary that evening to check, so the first person to notice was a reader the next morning. The packages of two community distributions copied the checksum file within the first hour. Until their maintainers update them, installing from those packages fails the same way. The files on our downloads page are correct, and this check proves it on your machine:pegel upgrade on arm64 stopped with "checksum mismatch". The binary was fine. The checksum file next to it held the hash of an arm64 build that the release job had cancelled and started again, because the job wrote SHA256SUMS as soon as it could. amd64 was not affected.Timeline
SHA256SUMS lists the hash of the cancelled arm64 build.pegel upgrade on a Raspberry Pi stops with "checksum mismatch".SHA256SUMS signed and uploaded. pegel upgrade on arm64 works again.What went wrong
What we changed
SHA256SUMS (in 0.9.1).pegel upgrade prints both hashes on a mismatch, so a report tells us at once which file is wrong.Still open
$ sha256sum -c SHA256SUMS --ignore-missing
pegel_0.9.0_linux_arm64.tar.gz: OK
Post
Every element a release note, incident report or post here can use, written the way David and I write them.
6 min
Changelog
An alert fires only when two probes agree, which ends the false alarms from a flaky home line. Since this release, interval counts seconds.
Checks can now run from two places at once, and an alert only fires when both see the outage. That ends the false alarms from a flaky home connection, which were most of our bug reports this year. The one breaking change is The probe on the second machine gets its checks from your Pegel and needs no config of its own, only the name and a token. Why two probes, and what a year of false alarms looked like: Why an alert needs two probes.interval, which counts seconds now instead of minutes. Pegel refuses to start with an old config and says which line to fix.interval counts seconds, not minutes. Multiply every interval in pegel.toml by 60 before you upgrade, or Pegel will not start.Added
[[probe]] in pegel.toml and the command pegel probe, which runs your checks on a second machine and reports to your Pegel.agree under [defaults] or per check: how many probes must see an outage before an alert fires. It defaults to 2 with two probes or more.Changed
interval is in seconds. interval = 1 meant one minute and now means one second, which Pegel refuses.pegel check names the probes each check ran from.Removed
PEGEL_CONFIG, deprecated since 0.7.0. Use --config.Fixed
Upgrading from 0.8.x
[defaults]
-interval = 1
+interval = 60
+agree = 2
+
+[[probe]]
+name = "home"
+
+[[probe]]
+name = "vps"
Post
From May to October, one probe on a home line sent us 121 false alarms. Two probes that had to agree sent one.
2 min
Letter
A security release, a mail relay that held our alerts, and two probes on main.
1 min
TIL
toml.DecodeFile from BurntSushi/toml returns a MetaData, and its Undecoded() lists every key in the file that matched no field. Pegel 0.9.0 uses it to refuse a typo such as intervall instead of ignoring it.
md, err := toml.DecodeFile(path, &cfg)
if err != nil {
return nil, err
}
if keys := md.Undecoded(); len(keys) > 0 {
return nil, fmt.Errorf("%s: unknown key %q", path, keys[0].String())
}Before, a misspelt key fell back to its default without a word, and nobody suspects a default.
Incident
On 2026-10-24 our mail relay accepted every mail and delivered none from 09:20 to 13:20 CEST. Webhook alerts reached us the whole time.
Resolved
Minor
Our own alert mails stopped for four hours on Saturday 2026-10-24. The relay that sends them and the release announcements accepted every mail and delivered none, because its upstream provider had moved to a certificate chain the relay did not trust. The queue check alerted at 09:37, by webhook and by mail. The mail copy waited in the queue it was warning about. The relay used a CA bundle we had copied onto it by hand in 2025 and never updated. Nobody read the webhook alert until 11:50, because it was Saturday and the alert said "queue older than 15 minutes", which had always cleared on its own.Timeline
certificate verify failed for every delivery since 09:20.What went wrong
What changed
Post
Two probes are on main. The home probe times out and Pegel waits. A minute later both time out and the alert fires.
1 min