Writing a DR runbook you can actually execute

Most disaster recovery documents are inventories: a list of systems, a list of contacts, a diagram. A runbook is different. It is an ordered sequence of actions someone can follow at 04:00 while the primary site is dark, without needing to make architectural decisions on the spot.

Start with tiers, not systems

Sort workloads into three or four tiers and give each tier an RPO and an RTO. The recovery point is how much data you accept losing; the recovery time is how long the service can be gone. Both are business decisions, and both cost money as they approach zero — which is exactly why the business, not the infrastructure team, should sign them off.

A workable starting shape: tier 1 gets continuous replication with seconds of RPO and tens of minutes of RTO; tier 2 gets scheduled replication with a 15-minute RPO; tier 3 restores from the offsite backup chain within a working day; tier 4 is rebuilt from configuration management and does not need a replica at all. Being explicit that some systems are tier 4 is what makes the tier 1 budget defensible.

Get the dependency order right

The single most common runbook failure is booting an application before the thing it authenticates against. Write the failover order down and test it: directory and DNS, then databases, then application servers, then the public entry point.

Then handle the parts that are not virtual machines. Which DNS records change, and what is their TTL? Which certificates does the replica need? Which outbound integrations are IP-allowlisted at the other end? Which scheduled jobs must stay off until failback, so invoices are not sent twice? These are the steps that turn a successful boot into a working service, and they are usually missing.

Name the humans

A runbook needs a declared decision-maker: who is allowed to declare a disaster and start the failover, and who deputises when that person is unreachable. Without it, the first hour of every real incident is spent seeking permission.

Add the communications plan in the same document — who tells customers, who tells staff, which channel is used when email is part of the outage — and keep an offline copy. A runbook stored only in the environment it recovers is not a runbook.

Rehearse in a sandbox

Test failover brings replicas up in an isolated network where they cannot reach production, so you can rehearse on a normal Tuesday. Boot the tier 1 group in dependency order, log in as a real user, run a genuine transaction, and record the wall-clock time from decision to service.

Compare that number with the RTO in the document and correct whichever is wrong. Then rehearse the part everyone skips: failback. Returning to the primary site with the changes made during the outage is the harder direction, and the first time should not be for real.

Keep it short and current

A runbook that has not been executed in a year is a work of fiction. Review it after every significant infrastructure change, keep it to actions rather than explanations, and attach the evidence from the last rehearsal — the date, who ran it, what broke, what was fixed. That evidence is also what an insurer or an auditor asks for.

Antyxsoft DRaaS keeps replicas of critical machines warm in EU data centres with per-workload RPO and RTO targets, sandboxed test failover and orchestrated failback — see how Veeam backup and DRaaS works.