Building Blue Canoe · 13 of 19
The Report That Found a Fault in Itself
How Blue Canoe traced two mail-monitoring failures to one DNS timeout and corrected a DKIM body-hash problem in the report's own email.
This account was drafted on 19 August 2026. Operational states, versions and test results describe that period unless a later update is explicitly dated.
The morning mail report failed two checks on mx3. One lookup had timed out.
The PTR query through mx3's local Unbound resolver timed out. The forward-confirmed reverse DNS check, or FCrDNS, then failed because there was no usable PTR result to compare with the forward address. Two red lines, one underlying event.
A red report is evidence, not an instruction
By the time we inspected mx3, Unbound was active and listening. The PTR returned mx3.blue-canoe.net, the forward lookup returned the correct public address, the service had not restarted and its journal contained no event in the failure window.
The active validator then passed all 75 checks. We did not restart or reconfigure a healthy resolver to make ourselves feel useful. The incident was retained as a transient recursive timeout and the dependent FCrDNS failure was recorded as a consequence rather than a second fault.
Check plausible explanations
The DNS production report ran each morning as well, so overlapping diagnostics were worth investigating as a possible source of contention. A timeout alone did not establish that explanation. We needed to compare the run times and service evidence before attributing the failure to another scheduled task.
A tidy explanation is not evidence merely because it fits the calendar.
Then the report exposed a fault in itself
The replacement PASS email reached Gmail, but DKIM was neutral because the body hash did not verify. SPF was aligned, so DMARC still passed. That made the message deliverable without making the broken signature acceptable.
The investigation focused on how the report constructed its MIME message. The existing message used 8bit bodies; the correction encoded both the text and HTML parts as base64 before signing. The body-hash failure gave us a concrete acceptance condition for that change.
Fix the construction, not the limit
We did not raise Postfix's line-length limit. The candidate encoded the text and HTML MIME parts as base64 with lines of 76 characters before OpenDKIM signed them. The recorded recipient-side check then reported DKIM pass, SPF pass and DMARC pass.
The candidate moved through checksum verification, syntax checking, ShellCheck, a non-published test, release staging, backup, installation and active validation. The production report became v0.1.7.
Monitoring belongs inside its own model
A monitoring system can be wrong, can misunderstand a result and can damage the message carrying its conclusion. It therefore needs the same evidence, release discipline and willingness to be corrected as the infrastructure it observes.
That morning the report did three useful things: it made a transient event visible, stopped us changing a healthy resolver without evidence, and revealed a separate defect in its own output path.
Green has to mean something. So does red.