Search Blue Canoe

Enter at least two characters.

Building Blue Canoe · 7 of 19

Green Has to Mean Something

Why internal DNS agreement is not enough, and what an external production report actually proves.

A monitoring workstation with green and amber indicators beside server racks, with blue cables crossing a glass partition.

That was not what I wanted from DNS.

By the time the new authoritative platform had settled, the internal checks were quite good. Source zones were linted. BIND configuration was checked. The hidden master and both public secondaries were queried directly. SOA serials were compared. DNSSEC signing state was checked. A minute-by-minute health job could tell me whether the authoritative estate had stopped agreeing with itself.

All of that was useful. It still left one uncomfortable question.

What does the rest of the Internet see?

The first external check found something

We built DNSOps as a read-only production report rather than another management interface. Its job was to collect the facts we already trusted, add the facts we could not see from inside, and produce text, JSON and HTML from the same evidence.

For every managed domain it asked two independent public resolvers — Cloudflare and Google — for the public SOA and NS view. For domains where DS was expected, it also checked the parent DS state and required end-to-end DNSSEC validation through both resolvers.

The first run came back red.

Twenty-one domains passed. One did not.

Internally, camfrankcom.com looked fine. The source zone existed, the master and secondaries agreed, and it was locally signed. Externally, the nameserver view did not match what DNSOps had been told to expect.

My recollection is that this was around the time we were having trouble with Namecheap: some things were working and some were not. That made the registrar an obvious suspect, but a suspect is not a diagnosis.

Murphy and Sod were playing “hold my beer!” and reinforcing the belief that DNS does indeed hate you.

The report could not settle that last question. It could tell us exactly where the public view disagreed with what we expected.

Different checks answer different questions

The useful part was not that DNSOps had found a dramatic outage. It had found a disagreement between two views of reality.

Our authoritative checks answered: are the servers we operate healthy and consistent?

The external checks answered: is that the DNS state the public Internet is actually using?

Both sets of checks were correct. They were simply answering different questions.

That distinction is now baked into the report. Green does not mean 'BIND is running'. It means the source zones validate, the authoritative servers agree, the public recursive view agrees with the intended delegation, locally signed zones are actually signed, expected DS records are present, and the expected secure chains validate from outside our network.

What the recorded result establishes

The report at 10:32 on 13 August recorded a public-delegation mismatch for camfrankcom.com. At 13:15 that afternoon, all 22 managed zones passed, including that domain.

The domain was not retired; it remains on our DNS. The reports establish the mismatch and the later passing state, but not the precise cause or every action taken between them. I would rather leave that limit visible than turn a recollection of registrar trouble into a diagnosis the evidence does not support.

Trust grows through disagreement

This is not the first time in these projects that a machine's output has been right and my interpretation has been wrong. It is also not the first time a script has produced a reassuring answer while one of the numbers underneath it turned out to be misleading.

That is why I am increasingly interested in what a PASS actually proves.

I do not want to trust a script because we wrote it. I want to trust it because we have disagreed with it, checked the evidence, found out which of us was wrong, and improved the definition of the test when necessary.

That trust is growing.

Green now has a count attached

The DNSOps report at 13:15 on 13 August 2026 passed all 22 managed zones. Eleven zones expected to have a complete DNSSEC chain validated end to end through both public resolvers. Eleven were recorded as having no parent DS. That last category is not a euphemism for failure: it is an explicit statement of a different intended or externally constrained state.

A useful report does not require every object to be identical. It requires every object to be compared with the state actually intended for it.

The report is not the system

DNSOps now sends me a concise green, amber or red summary each morning. The full text, JSON, HTML and evidence stay with the DNS subsystem. The email is just the cockpit view.

The hidden master does not send mail directly. It hands a tightly constrained notification to the mail worker over the private back channel. The worker uses the mail platform's existing trusted path to MX1, so the DNS server does not acquire SMTP credentials, a new outbound-mail identity, or another SPF problem.

That arrangement matters because the worker is gradually becoming an operations integration point without becoming the place where DNS logic lives. DNS knowledge stays with DNS. Mail knowledge stays with mail. The joins between them are deliberately small.

Boring is earned

The aim was never to build a dashboard that looked reassuring.

The aim was to make DNS boring enough that I can glance at the morning report, believe what it says, and move on.

That confidence did not arrive with the architecture diagram. It came from the awkward cases: REFUSED responses, the wrong local resolver, a stale journal, registrar dependencies, a DNSKEY count that meant something other than we thought, and finally an external check that found a mismatch our internal checks could not establish.

Green has to mean something.

Otherwise it is just a colour.