The Mail Server Factory: Build for the Day It Breaks

This account was drafted on 19 August 2026. Operational states, versions and test results describe that period unless a later update is explicitly dated.

The most useful thing we built during the mail project was not a mail server. It was a way of building mail servers.

That distinction matters. A mail system is full of important state: domains, users, aliases, keys, certificates, messages, filtering decisions and the configuration that ties them together. If all of that knowledge exists only on the running machines, those machines become far too precious.

I wanted the opposite. The machines should be replaceable. The method, inputs and evidence should be the valuable parts.

Three nodes were always the plan

The platform was designed around three mail nodes from the beginning. mx1 is the primary service node. mx2 provides the secondary role and replicated mail storage. mx3 is the tertiary edge node and carries supporting public services such as autoconfiguration, autodiscovery, MTA-STS, status and setup endpoints.

The design did not appear in isolation. Mail was the reason we first followed the dependency chain backwards through DANE, DNSSEC, authoritative DNS and, eventually, the network edge. By the time we were ready to build the mail platform, much of its foundation had already been deliberately put in place.

The separate 10Gb backend also had a real job. Replication, migration, logging and worker-to-node traffic could use the internal path without competing with ordinary customer-facing service traffic.

The worker is part of the architecture

A separate worker coordinates much of the build and operational work. It connects to each node over the backend network, carries the known inputs and runs the scripts that build, inspect, validate and report on the platform.

That separation is important. If a server fails, the instructions for rebuilding it should not be trapped on the failed server. Nor should recovery depend on remembering a sequence of commands typed several months earlier.

We treated the scripts as decisions distilled into executable form. The surrounding reports, ledgers and documentation preserve why those decisions were made and what the resulting system was expected to look like.

Preflight is a different operation from apply

One of the recurring design choices was to separate inspection from change. A preflight should establish identity, inputs, reachability, storage, expected records and prerequisites without quietly altering the system it is supposed to inspect.

Only after the preflight evidence is understood should an apply or transfer operation be allowed to change production state. The change is then followed by validation against the intended result.

This sounds laborious until something is wrong. At that point it is considerably less laborious than trying to determine which of several unrecorded changes caused the new behaviour.

A console scroll is not a record

Long-running operations needed ledgers. Migration runs needed stable run identifiers, explicit mailbox states and the ability to resume. Reports needed to say what had been checked, not merely that a script had reached its final line.

That changes the useful question from ‘did it look all right?’ to ‘which inputs were used, what was attempted, what passed, what failed and can we account for every item?’

The same principle applies to rollback. A backup is only comforting if its target is known, its creation is recorded and the route back has been considered before the change begins.

The factory had to learn service identities

DANE and MTA-STS exposed another class of assumption. MX1 presents mail.blue-canoe.net to users but must also present mx1.blue-canoe.net correctly to sending MTAs. SMTP, Dovecot and Nginx do not necessarily share the same certificate role. The factory inputs were corrected to preserve those distinctions instead of treating one certificate lineage as interchangeable everywhere.

Certificate renewal became part of the build model too. DANE associates the endpoint key through TLSA, so ordinary renewal was configured to reuse that key and dry-run tested. Rebuilding a node must recreate not only a working certificate, but the renewal behaviour on which its public DNS association depends.

Reporting became another factory output

By 19 August, the released production report collected evidence from all three MX nodes, produced text, JSON and HTML from the same run, published the daily view and sent a concise operational email. Its evidence directory preserved what each PASS or FAIL was based upon. The report reached v0.1.7 after its own email construction exposed a DKIM body-hash problem and was corrected through the same candidate, test and release process used elsewhere.

Planning made execution short

By my rough reckoning, planning took far more time than execution. That is not a measured ratio or a complaint. The execution window is when production is exposed; making that part short and unsurprising is the point.

The planning also survives. The same preflight, apply, validate, report and recover pattern is now informing the next booking platform: one engine, multiple consumers, explicit tenant inputs and state that can be accounted for.

Mail servers are easy to install. Failing mail servers are a nightmare. The factory is what turns a collection of installed packages into an operational system I can understand, test and recreate.

The objective was never to make failure impossible. It was to make failure something for which we already had a method.