Pegel

Why an alert needs two probes

From May to October, one probe on a home line sent us 121 false alarms. Two probes that had to agree sent one.

A diagram: probe home and probe vps each report down to a box labelled agree?, which points to example.org. An alert fires only when both probes see the outage.

Our own Pegel ran on one machine on Hanne’s home line for two years, and every time that line dropped, it told us that every site we watch was down. From May to October that happened 121 times. Not one of those alerts was a real outage.

So David ran a second Pegel on a small VPS from May on, and we compared afterwards: an alert counted only if the VPS had seen the same failure in the same minute. That left one alert in six months, on 2026-08-11, and it was false too. The site was up, but its DNS answered after 31 seconds and both probes had given up at 30.

A bar chart of false alarms a month from May to October 2026. One probe: 21, 17, 24, 19, 22 and 18. Two probes: none, except one in August.
False alarms a month: one probe against two that must agree.

What "agree" means

Each probe runs every check on its own schedule and reports the result with a timestamp. Pegel waits until the results of one round are in, counts the probes that saw the check fail, and alerts when that count reaches agree.

// Down reports whether at least agree probes saw the check fail in this round.
func Down(results []probe.Result, agree int) bool {
	failed := 0
	for _, r := range results {
		if !r.OK {
			failed++
		}
	}
	return failed >= agree
}
An editor with pegel.toml: interval = 60 with the comment seconds since 0.9.0, agree = 2, a check named shop and two probes named home and vps.
The config we run on main.

What it costs

A real outage reaches you a little later, because Pegel waits for the slower probe, at most the timeout. With interval = 60 and the default timeout of 10 seconds that is 70 seconds instead of 60. We take that over an alert nobody believes.

Two places, not two machines

Two probes on the same line agree about the same broken router. Put the second one where the first one’s network cannot take it down: a VPS, a friend’s house, the office. Hanne wrote the code that decides, David ran the VPS and did the counting, and the probes ship in 0.9.0.

Comments

Buy Logbuch