The restart order, that forgotten deliverable
The problem
An organisation restores its virtual machines in order of business criticality. The most critical application starts first — and fails, because the authentication directory is not yet up, the name server does not respond and the reference database has not finished restoring.
Restart order is not criticality order. It is the order of technical dependencies, which always starts with components nobody mentions in committee.
The five waves
| Wave | Contents | Note |
|---|---|---|
| 0 — Foundation | Network, DNS, NTP, directory, PKI, secrets management | Without synchronised clocks, authentication fails |
| 1 — Platforms | Hypervisors, storage, backup, monitoring | Backup must be available before any restore |
| 2 — Data | Databases, reference data, message queues | Cross-database consistency is verified here |
| 3 — Critical applications | Services supporting prioritised activities | In the order from the BIA |
| 4 — Remainder | Deferrable applications | Can wait several days |
Wave 0 causes the most exercise failures. NTP in particular: a clock drift of a few minutes invalidates authentication tokens and fails the whole chain, for a cause nobody looks at first.
The deliverable format
An ordered list with, for each item: rank, system name, owner, estimated duration, duration observed at the last test, and the verification criterion before moving to the next.
The verification criterion is the key element. "Database started" is not enough; "a control query returns the expected row count" is verifiable.
What the test reveals
A full restoration test almost always reveals three things:
- An undocumented dependency — typically a technical service whose function nobody remembers;
- An expired credential or certificate, unusable because it was never renewed in the recovery environment;
- A gap between estimated and actual duration, usually by a factor of two to three.
Those three findings alone justify the cost of the annual test.
Key takeaways
- A service does not restart alone: it depends on a bootstrap chain
- Infrastructure services always come first
- The order is tested, otherwise it is theoretical