Building Blue Canoe · 13 of 19
The Report That Found a Fault in Itself
How a daily mail report exposed both a transient lookup failure and a defect in its own email.
This account was drafted on 19 August 2026. Operational states, versions and test results describe that period unless a later update is explicitly dated.
The morning mail report failed two checks on mx3. One lookup had timed out.
The PTR query through mx3's local Unbound resolver returned nothing. FCrDNS then failed because there was no PTR result to compare with the forward address. Two red lines, one underlying event.
A red report is evidence, not an instruction
By the time we inspected mx3, Unbound was active and listening. The PTR returned mx3.blue-canoe.net, the forward lookup returned the correct public address, the service had not restarted and its journal contained no event in the failure window.
The active validator then passed all 75 checks. We did not restart or reconfigure a healthy resolver to make ourselves feel useful. The incident was retained as a transient recursive timeout and the dependent FCrDNS failure was recorded as a consequence rather than a second fault.
The obvious explanation was checked and rejected
The DNS production report ran each morning as well, so overlapping diagnostics looked like a plausible source of contention. The timestamps ruled it out. DNSOps had finished at 06:10:45; the mail report began at 06:15:16; mx3 validation ran more than six minutes after DNSOps had ended.
A tidy explanation is not evidence merely because it fits the calendar.
Then the report exposed a fault in itself
The replacement PASS email reached Gmail, but DKIM was neutral because the body hash did not verify. SPF was aligned, so DMARC still passed. That made the message deliverable without making the broken signature acceptable.
The report sent its MIME bodies as 8bit. Generated HTML contained lines longer than the normal SMTP limit. The body was signed, then long lines were altered during transmission. Gmail received bytes that no longer matched the DKIM body hash.
Fix the construction, not the limit
We did not raise Postfix's line-length limit. The candidate encoded the text and HTML MIME parts as base64 with short lines before OpenDKIM signed them. Gmail then reported DKIM pass, SPF pass and DMARC pass, and its calculated body hash matched the signature.
The candidate moved through checksum verification, syntax checking, ShellCheck, a non-published test, release staging, backup, installation and active validation. The production report became v0.1.7.
Monitoring belongs inside its own model
A monitoring system can be wrong, can misunderstand a result and can damage the message carrying its conclusion. It therefore needs the same evidence, release discipline and willingness to be corrected as the infrastructure it observes.
That morning the report did three useful things: it made a transient event visible, stopped us changing a healthy resolver without evidence, and revealed a separate defect in its own output path.
Green has to mean something. So does red.