Building Blue Canoe · 5 of 5
The DNS Server Nobody Can See
How Blue Canoe separated DNS source state from public secondaries, learned from migration failures, and built a checked, portable publication process.
Originally published on LinkedIn on 10 September 2026. This is the permanent Blue Canoe edition, with dated technical updates where the work subsequently progressed.
The mail platform was always intended to have three nodes and, eventually, the transport-security pieces I wanted around them. DANE was part of that plan. DANE depends on DNSSEC, and doing DNSSEC properly meant taking tighter control of authoritative DNS.
Following that dependency backwards eventually led all the way to the edge of the network. The MikroTik had to come first. Once that was in place, DNS could finally move.
The source of truth moved out of sight
The resulting design has one slightly counter-intuitive feature: the server holding the source of truth is the one the public Internet cannot normally ask.
The hidden master holds the source zone files and DNSSEC state. Public queries go to two secondaries, ns0 and ns1. Changes originate on the master, are validated there, and then transfer outward.
That separation appealed to me for a fairly simple reason. The public servers have one job: answer DNS queries. They do not need to be the place where I routinely maintain the source data.
It also makes the public pair much less precious. The important state lives behind them. If a public secondary has to be replaced, the objective is to rebuild it from known state rather than lovingly reconstruct whatever happened to be on the old machine.
The first migrations taught us what the diagram did not
We deliberately proved the process manually before trying to automate it. That turned out to be useful almost immediately.
On an early migration, the hidden master knew about the new zone and sent NOTIFY, but both public secondaries returned REFUSED. The assumption that they would simply acquire a previously unknown zone was wrong. There were no catalog zones configured; each secondary still needed an explicit declaration for a new zone. That became a formal step in the process.
Another test produced a confusing REFUSED response for a local query. The DNS server was not the problem at all: a bare dig was using the machine's configured recursive resolver rather than the authoritative BIND instance we thought we were querying. The lesson was simple enough. When testing authoritative DNS, specify the server you intend to test.
Then one zone produced the more interesting failure. blue-canoe.com passed named-checkzone, but the hidden master returned SERVFAIL and the public servers could not serve it correctly. The logs eventually showed an unsigned journal out of sync with the zone file. Removing only the stale journal state, after identifying the exact error, allowed the zone to load again.
None of those failures required redesigning the architecture. They were exactly the sort of small operational details I wanted to discover before turning the process into machinery.
A DNS move that was really a portability test
Moving live domains onto the new platform also gave us an unexpected partial test of a wider idea.
The service was not tied to the old DNS machines. We captured the existing state, constructed the replacement state, loaded it onto different infrastructure, proved the secondaries agreed, changed delegation and then validated the result from outside.
The switch went rather more smoothly than I expected.
That does not prove I could rebuild the entire datacentre from scratch over a weekend, and I am in no hurry to arrange that particular acceptance test. It does demonstrate something useful: important infrastructure roles can move when the state and the method for recreating them are understood.
The write path became a process, not a habit
As the platform settled, the manual steps became explicit tooling. A normal publication now validates the source zone and BIND configuration before mutation, takes the backup outside the live zone directory, advances the SOA serial, validates again, reloads the authoritative zone, sends NOTIFY, checks that master, ns0 and ns1 converge, and finally verifies that authoritative signing is still present.
The distinction matters to me. A command completing is not the same thing as a change being successfully published.
We also stopped leaving operational backups mixed into the live BIND directories. They belong in a backup area. A retired zone belongs somewhere else again: in an archive that records what it was, why it left production, its final configuration and DNSSEC state, and evidence of the public handoff.
Publication and observation are different jobs
The production tooling preserves that distinction. A publication can prove that the source is valid, the authoritative servers converged and signing remains present. It does not pretend that this alone proves the parent DS, recursive visibility or the complete chain seen by the rest of the Internet.
Those external questions belong to DNSOps. The daily production report checks 22 managed zones from both sides of the boundary: authoritative state under our control and the recursive public view through Cloudflare and Google. In the evidenced acceptance run, all 22 zones passed and all 11 DNSSEC chains expected to validate did so; the remaining 11 zones were explicitly recorded as having no parent DS rather than being silently coloured green.
DNS became boring
That is about where I want infrastructure to end up.
Not finished. A network is only finished when it is decommissioned and the parts are sold. But settled into its role.
The original reason for all of this was still mail. DANE demanded DNSSEC. DNSSEC demanded authoritative DNS under our control. That, in turn, demanded the network changes that came before it.
Following one requirement backwards built rather more infrastructure than I originally went looking for.
It also left us with something much more useful than a DNS server: a source of truth that can be rebuilt, checked and moved without pretending that any individual machine is precious.