The Mail Server Factory: Build for the Day It Breaks

This account was first drafted on 19 August 2026 and updated for publication on 1 October 2026. Operational states and test results are described in their stated context.

The most useful thing we built during the mail project was not a mail server. It was a way of building mail servers.

That distinction matters. A mail system is full of important state: domains, users, aliases, keys, certificates, messages, filtering decisions and the configuration that ties them together. If all of that knowledge exists only on the running machines, those machines become far too precious.

I wanted the opposite. The machines should be replaceable. The method, inputs and evidence should be the valuable parts.

Why build it ourselves?

I was not trying to create another Modoboa or iRedMail. Both are strong, established solutions; neither quite matched what I wanted from this particular build. For someone approaching mail-server administration for the first time, Modoboa is certainly one I would recommend. There is still plenty to learn about operating a public mail service, whichever route you choose.

There was no shortage of guidance on building a mail server. What I found much harder to find was guidance on operating several mail servers together as one coherent system: their separate roles, shared state, replication, service identities and what should happen when one of them fails.

That was the gap I wanted to address for Blue Canoe. Installing the individual services was only part of the work. The larger question was how to build, check and recover the platform as a whole.

Three nodes were always the plan

The platform was designed around three mail nodes from the beginning. mx1 is the primary service node. mx2 provides the secondary role and replicated mail storage. mx3 is the tertiary edge node and carries supporting public services such as autoconfiguration, autodiscovery, MTA-STS, status and setup endpoints.

The design did not appear in isolation. Mail was the reason we first followed the dependency chain backwards through DANE, DNSSEC, authoritative DNS and, eventually, the network edge. By the time we were ready to build the mail platform, much of its foundation had already been deliberately put in place.

The separate 10Gb backend also had a real job. Replication, migration, logging and worker-to-node traffic could use the internal path, reducing competition with ordinary customer-facing service traffic.

The worker is part of the architecture

A separate worker coordinates much of the build and operational work. It connects to each node over the backend network, carries the known inputs and runs the scripts that build, inspect, validate and report on the platform.

That separation is important. If a server fails, the instructions for rebuilding it should not be trapped on the failed server. Nor should recovery depend on remembering a sequence of commands typed several months earlier.

We treated the scripts as decisions distilled into executable form. The surrounding reports, ledgers and documentation preserve why those decisions were made and what the resulting system was expected to look like.

Rebuilding a node and recovering its state are different jobs. Recreating the packages and configuration does not recover the messages, keys or other data that belonged to the failed machine. Those need their own recovery path. Replicated storage is useful, but it does not by itself prove that recovery will work.

Preflight is a different operation from apply

One of the recurring design choices was to separate inspection from change. A preflight should establish identity, inputs, reachability, storage, expected records and prerequisites without quietly altering the production configuration or data it is supposed to inspect.

Only after the preflight evidence is understood should an apply or transfer operation be allowed to change production state. The change is then followed by validation against the intended result.

This sounds laborious until something is wrong. At that point it is considerably less laborious than trying to determine which of several unrecorded changes caused the new behaviour.

A console scroll is not a record

Long-running operations needed ledgers. Migration runs needed stable run identifiers, explicit mailbox states and the ability to resume. Reports needed to say what had been checked, not merely that a script had reached its final line.

That changes the useful question from “did it look all right?” to “which inputs were used, what was attempted, what passed, what failed and can we account for every item?”

The same principle applies to rollback. Before making a change, I want to know what the backup contains, where it is stored and how it would be restored. Recording that a backup completed is useful evidence. Demonstrating that it can be restored answers a different question.

The factory had to learn service identities

DANE and MTA-STS exposed another class of assumption. Users connect to mail.blue-canoe.net, while other mail systems deliver to MX hostnames such as mx1.blue-canoe.net. Those services have related but distinct certificate requirements.

SMTP, Dovecot and Nginx do not necessarily share the same certificate role. The factory inputs were corrected to preserve those distinctions instead of treating one certificate lineage as interchangeable everywhere.

Certificate renewal became part of the build model too. Our TLSA 3 1 1 records associate each SMTP endpoint with a SHA-256 hash of its public key information. Ordinary renewal was configured to reuse the existing key, and renewal dry runs passed.

That setting alone does not solve a rebuild. A replacement node must either retain the matching key through secure recovery or introduce a new key through a coordinated TLSA change. The certificate actually served must then be checked against the public DNS association.

The factory therefore needs to account for both the certificate configuration and the key continuity on which DANE depends.

Reporting became another factory output

The production report grew into another output of the same process. It collects evidence from all three MX nodes, produces text, JSON and HTML from the same run, publishes the daily view and sends a concise operational email. Its evidence directory preserves the observations behind the reported results.

During development, the report’s own email construction exposed a DKIM body-hash problem. That was corrected through the same candidate, test and release process used elsewhere.

The reporting machinery needed evidence of its own correctness too. Reaching the end of a script was never going to be enough.

Planning made execution short

Across this work, planning and validation have taken much longer than the actual execution. That is not a complaint. The execution window is when production is exposed; making that part short and unsurprising is the point.

The planning also survives. The same preflight, apply, validate, report and recover pattern is informing the next booking platform: one engine, multiple consumers, explicit tenant inputs and state that can be accounted for.

Installing mail packages is only the beginning. The factory is what turns them into an operational system whose construction I can understand, inspect and repeat. Recovery still has to account for the state those services depend upon.

The objective was never to make failure impossible. It was to make failure something for which we already had a method.