1. The heartbeat with a dead payload
We run a heartbeat: a scheduled message that says, in effect, “the alert channel still works.” It exists because every alarm we raise — billing failures, provider health, integrity checks — travels through one delivery path, and if that path breaks, the symptom is silence. A dead man’s switch converts silence into a signal.
The heartbeat carried a fact as well as a pulse: how many alarms had been delivered in the period. That query referenced a column called created_at. The table’s column is recorded_at. The query therefore threw on every single run, a catch block turned the error into a null, and the message dutifully reported that the count “could not be read.”
The pulse kept arriving, which is precisely why nobody looked closer. We had built the mechanism that converts silence into signal, and then failed to notice that the signal itself had been degraded since the day it shipped.
The lesson: a heartbeat proves delivery, not correctness. If the message carries data, something has to verify the data is real — otherwise you have a very reliable way of sending nothing.
The same typo had a second victim: an attribution query that ran on every completed payment, threw every time, and had its exception swallowed by a catch that dropped the attribution. A reporting feature had been live for months with no data behind it and no error anyone could see.
2. Alarms in one direction only
Our billing path holds a reservation against a customer’s balance when a scan starts, then either charges it or releases it. Both directions can fail. Only one of them was watched.
If the charge failed, the system set a flag, wrote a failure reason, sent an urgent alert, and a reconciliation job retried it later. If the release failed — the direction where the customer keeps paying for something they did not receive — the error was written to a log line and forgotten. No flag, no counter, no alert, and no query anywhere that looked for reservations left standing.
This asymmetry is not malice and it is rarely a decision. It is what happens when alerting gets built in response to incidents, and revenue loss produces an incident while customer-side loss produces a support ticket you never receive because the customer does not know either.
The lesson: list the ways money can move in your system and check that each direction has the same detection. If the asymmetry is deliberate, write down why. Ours was not deliberate.
3. The notification that reported work it had not done
This one is the most uncomfortable of the four. When a scan failed, we sent ourselves a notification that ended with a reassuring line: the reservation was fully released, the customer was not charged.
That sentence was a string constant. It was appended unconditionally, without reference to whether the release had actually succeeded — and, as above, a failed release was silently swallowed. In the exact case where a human most needed to intervene, the notification stated that no intervention was required.
A monitoring message that asserts an outcome it has not verified is worse than no message. It does not merely fail to inform; it actively creates false confidence, and it does so most convincingly in the situation you built it for. The fix was to bind the sentence to the actual result, including a third case — scans with no reservation at all — that the original text had quietly misdescribed all along.
4. The gate that cried wolf
The fourth finding is about a control we were about to build badly.
Our deployment schema had drifted from our code before, expensively: an engine was marked live in the product catalogue while the database constraint did not recognise its identifier, so every request for that engine failed at the database and orphaned the customer’s reservation. We decided to add a gate that verifies production schema before a release ships.
The obvious implementation asks the migration tool which migrations are pending and blocks if there are any. We tried it. It immediately reported a pending migration whose table was demonstrably live in production — because our convention applies migrations by file and does not write the tool’s ledger. The naive gate would have failed on almost every release, for a reason that was not a problem.
A gate that fires when nothing is wrong is not a weak control. It is a harmful one: it teaches the operator to skip the check, and it consumes the credibility the gate needs on the day it is right. We rewrote it to query the live schema instead of the bookkeeping — read the actual constraint, compare it against the product catalogue, and check that objects the pending migrations claim to create really exist. Ledger behind but schema present is now a note. Schema genuinely missing blocks the release.
The lesson: measure the thing you care about, not the record of the thing. And treat every false alarm as a defect in the alarm, because that is how it will be treated by whoever is on call.
What we changed
Concretely: the two dead queries now reference the column that exists, and both were verified against production rather than assumed. Release failures raise the same flag, failure reason and urgent alert that charge failures always did, and a sweeper looks for reservations with no corresponding scan. The failure notification states what actually happened. Integrity-check failures and partial database backups — two more alarms that only ever reached a log — now page a human. And the heartbeat moved from weekly to nightly, because a week of undetected silence consumes an entire regulatory notification window before anyone knows there is something to notify about.
The pattern worth taking away
All four findings are the same shape: a control that exists, is visible in code review, and does not do its job. That shape is much harder to catch than a missing control, because every artefact you would use to check — the code, the dashboard, the runbook — shows the control present.
The only reliable question is behavioural. Not “do we have an alarm for this?” but “when did it last fire, and what happened when it did?” If an alarm has never fired, that is worth exactly as much attention as one firing constantly. Both mean you do not yet know whether it works.
Frequently Asked Questions
What is a dead man’s switch in monitoring?
It is a scheduled signal that must keep arriving. Because a broken alerting path produces silence, and silence is indistinguishable from a healthy quiet system, the heartbeat converts that silence into something you can detect: if the regular message stops, monitoring is broken rather than idle.
How often should a heartbeat run?
Frequently enough that the detection gap is shorter than your obligations. A weekly heartbeat means a broken alert channel can go unnoticed for up to seven days, which is longer than most regulatory breach-notification windows. We moved ours to nightly for exactly that reason.
Why are silent catch blocks dangerous?
A catch that converts an error into a default value keeps the system running while hiding that a component stopped working. Two of our findings survived for months because of it: a wrong column name threw on every run and was swallowed each time. If you must swallow an error, log it loudly enough that someone sees the pattern.
Is a noisy alert better than no alert?
No. An alert that regularly fires when nothing is wrong trains people to ignore it, so it is unavailable on the day it is correct. Treat false positives as defects in the alarm and fix them with the same priority as a missed detection.
How do you test an alarm you have never seen fire?
Trigger it deliberately in a controlled way: inject the failing condition, confirm the message arrives, and confirm it says something a responder can act on. We validated one of our new deployment gates by injecting a deliberately broken configuration and checking that it blocked the release, then reverting it.