Let’s run your first failover drill together. Nothing here needs a special product; you need a service with a standby, a stopwatch, a few colleagues and about half a day. I’ll go step by step, and along the way I’ll lean on the Ops Atlas, a free reference from xOps School, for the key ideas behind DisasterRecoveryOps. If you have only ever read about recovery plans, this is the gentlest way to make one real.
Step 1: Choose one service and write down two numbers
Start small. Pick a service that matters but will not hurt anyone if the drill goes sideways, ideally in a staging environment. Then write down its recovery time objective (how quickly it must be back) and its recovery point objective (the moment in time to which data must be restored). For a practice run, something like sixty minutes and five minutes is perfectly reasonable.
Put both numbers in a document everyone on the drill can see. Numbers in someone’s head do not count.
Step 2: Start the clock at the problem, not at the keyboard
Here is the idea most beginners miss. The Atlas teaches that the recovery clock includes the people part: noticing the issue, gathering the right person and deciding to invoke the plan. Their sample drill spent 31 minutes on those human steps and 33 on the technical ones, so the total slipped past a one-hour target.
For your drill, decide on a start signal, say a message that says “the primary is down”, and start timing the instant it is sent. Everything after that counts, including the pauses while people check chat.
Step 3: Calculate a decision deadline
Now do a tiny bit of arithmetic. The Atlas gives a recipe: start from the objective, subtract how long your slowest recent drill took, then subtract a cushion. Take a one-hour objective, a technical run of 33 minutes and five minutes of margin, and you end up with 22 minutes. That is how long the team may spend deciding before the target is already at risk.
On your first drill you will not have a “slowest recent drill”, so use your best estimate and update it afterwards. Write the deadline on the same page as the objective.
Step 4: Name who may call it
Pick one person with authority to declare a failover, plus a deputy. Check that both can be reached on a weekend. Teams that skip this step spend their precious minutes asking whose job it is. If your organisation is still building the habits that make that kind of clarity normal, the discussion of how culture change shapes day-to-day DevOps practice is a friendly companion read.
Step 5: Fence the failing side
Time to act, and the order matters. The Atlas gives a three-part sequence: fence, then promote the data, then move traffic. Fencing means making sure the old primary can no longer accept work. Be careful, though. Turning off a DNS route only helps with traffic that follows DNS. Cron-style jobs, workers draining queues, and callers holding cached addresses can still hit the old side, so stop each of those by hand.
Step 6: Promote the standby data
Next, make the standby copy the one that accepts writes. This is also the moment to note the effective loss window. Your true data-loss window follows whichever dependency lags furthest behind, as the Atlas explains, and that may not be the main database at all. Look at each store and jot down what you find.
Step 7: Move traffic and confirm from the user’s side
Only now do you point users at the standby. Then verify the way a customer would: log in, perform a typical action, check that the result is saved. “The dashboard is green” is not the same as “a user can finish what they came to do”. The Atlas makes a related point using a large cloud disruption in 2025, where finding the cause and restoring every service were separated by many hours.
Step 8: Stop the clock and write the scorecard
Record the total time and compare it with your objective. Record the decision time and compare it with your deadline. Record how stale the data was against the point objective. In the Atlas sample, the drill missed its time target by four minutes, overshot the decision allowance by nine, but met its data target. Mixed results like those are normal, and a mixed scorecard is a fine first outcome.
Step 9: Plan the trip home
Failing over is only half the journey. In the Atlas’s framing, going home is a migration in itself: data written on the standby has to be carried back, ownership has to be switched again, and traffic has to follow. If you did not plan this, spend an hour now writing the steps. Delaying failback planning is one of the anti-patterns the entry lists.
If some writes never reached the standby, handle them carefully. Preserve the old primary first, since snapshots can expire; compare against a restored copy instead of live production; and replay differences through the application in a way that is safe to repeat.
Step 10: Fix three things and schedule the next drill
Finish by choosing the three biggest surprises from the day and turning each into a task with a named owner. Common finds include credentials that lived only in the failed environment, outdated container images in the standby, and steps that required clicking around in a console. The Atlas recommends holding recovery-only logins somewhere that will not fail along with the main site, and scripting the steps so memory is not required.
Book the next drill before anyone leaves the room. Risk work does not stay done; this approach is explored from the delivery angle in a practical view of risk management in reliable delivery.
What good looks like after a few rounds
After three or four drills you should see the margin improve, the decision time shrink and the list of surprises get shorter. At that point, consider the next level: rehearse failback, then try a planned switchover in production for your most important service. If you want to know where that road leads, the Atlas page about DisasterRecoveryOps lays out five maturity levels, ending with routine production switchovers.
For a wider view of how practices like this are spreading, see what lies ahead for the field in an essay on DevOps futures across Japan and other markets. And remember, the first drill is meant to be imperfect. Its job is to show you where the plan is thinner than you believed, while it is still cheap to find out.









