Every engineering team has one: a test that fails now and then for no reason anyone can explain, and that is rerun until it passes. Rerunning costs a few minutes each time and wears down trust in the whole pipeline. This how-to takes a single flaky test and walks through fixing it at its root in a handful of steps, as an example of the larger habit of choosing one recurring engineering annoyance per month and removing it properly rather than working around it again. The method applies equally to a slow build or a stuck deployment step; the flaky test is simply easy to describe.
Before you start: confirm it is worth a month’s fix
Check three facts. The test must fail intermittently, not constantly. It must have failed at least three times in the recent past. And people must be doing something about it that costs time, such as rerunning jobs or ignoring red builds. If those hold, proceed. If the failure is constant, it is a bug, not an annoyance, and belongs in the normal queue.
Step 1: Gather the failure history
Pull the last several failures from your build history and put them side by side. For each one, note the date, the runner or machine, the exact error line and how long the test took. Intermittent failures leave patterns that are invisible one at a time. You may notice, for instance, that failures cluster on a particular runner, or that the failing runs all took longer than usual. Write the observations in a plain list; do not theorise yet.
Step 2: Reproduce it on demand
A fix you cannot test is a guess. Try to make the failure happen deliberately. Run the test many times in a loop on your own machine, then with the same constraints as the build runner, such as limited CPU or parallel execution. If it never fails locally, vary one thing at a time: ordering, parallelism, system load, the clock. Many intermittent failures reveal themselves when you shuffle test order or run tests simultaneously. When you can reproduce it in, say, one of every ten runs, you have a usable experiment.
Step 3: Identify the category of cause
Most flaky tests fall into a short list of causes. Compare your evidence against them:
- Timing: the test waits a fixed period for something that sometimes takes longer.
- Shared state: another test leaves data behind, or two tests write to the same place.
- Ordering: the test relies on another test having run first.
- External dependency: a network call, a clock or a randomly generated value changes the outcome.
- Resource limits: memory, file handles or ports are exhausted on busy runners.
Pick the category that best fits the pattern from Step 1. If two seem plausible, test them in turn rather than changing both at once.
Step 4: Make the smallest root-level change
Now fix the cause, not the symptom. If the test waits a fixed time, replace the wait with a check on the actual condition, with a generous upper limit as a safety net. If tests share data, give each its own isolated data and clean it up deterministically. If an ordering dependency exists, make the test set up its own prerequisites. Resist the temptation to add an automatic retry as the fix: a retry hides the problem and lets the flakiness grow elsewhere. Keep the change as small as possible so that a reviewer can see exactly what you did and why.
Step 5: Prove the fix against the reproduction
Run the same loop that used to fail one time in ten, but run it many more times, enough that the old failure rate would have produced several failures. If it passes cleanly, repeat under the stressed conditions from Step 2. Only then merge. Note the numbers you observed in the pull request description so a later reader can see the evidence.
Step 6: Write the note and close the loop
Add a short comment near the code explaining what used to go wrong and why the new approach avoids it. Update the issue or ticket with the cause category, so that if a similar failure appears elsewhere, someone can search for it. Tell the team the test is fixed and that reruns for this one should no longer be needed. That last message matters more than it seems: habits of rerunning outlive the problem that created them.
Where this sits in a wider practice
Fixing a single flaky test is a small act, but it is part of how teams build a healthy delivery culture. Our write-up on steering the people side of a DevOps shift argues that visible, repeated improvements matter as much as the formal process around them, and the outlook for delivery practices at home in Japan and abroad gives useful context on why teams are investing in dependable delivery.
A note on cataloguing ideas
Teams sometimes keep a list of candidate annoyances and glance at how other sites present technology collections for structure ideas. BestToolsX is one example, a trial-run directory of software and tech products whose pages mark the entries as placeholder data, offering sections for browsing listings by product type and by the problem each solves. It illustrates one way to organise a list. The flaky test itself needs nothing from a directory.
Common pitfalls
- Skipping reproduction. Without it you cannot tell a real fix from a lucky streak.
- Adding retries. They reduce visible failures and increase hidden ones.
- Widening the fix. A month’s fix is one test or one small family of tests, not the entire suite.
- Forgetting the note. Without it, a later refactor can bring the problem straight back.
Applying the steps elsewhere
The same six steps carry over to annoyances that have nothing to do with tests. A slow build can be treated with the same history-first approach: collect the durations, reproduce the slowness, classify whether it is download time, compilation or waiting for a shared runner, then change the smallest thing that addresses the cause. A deployment step that needs a manual confirmation can be examined by asking what the confirmation protects against and whether a check could protect against it directly. In each case the discipline is identical: evidence first, a minimal change, and proof before you merge. Over time the team builds a shared habit of treating friction as something with a findable cause.
What to do next month
When this one is closed, return to your list of irritations and choose the next. Perhaps it is a build step that downloads the same files every time, or a deployment that needs a manual approval nobody remembers the reason for. Apply the same six steps: gather the history, reproduce, categorise, make the smallest root-level change, prove it and write it down. Taken together, a year of such fixes gives a team a pipeline that people trust, and that trust is the real return.









