Error Budgets Explained Simply: A Story About Honest Failure

Rajesh Kumar

Rajesh Kumar is a leading expert in DevOps, SRE, DevSecOps, and MLOps, providing comprehensive services through his platform, www.rajeshkumar.xyz. With a proven track record in consulting, training, freelancing, and enterprise support, he empowers organizations to adopt modern operational practices and achieve scalable, secure, and efficient IT infrastructures. Rajesh is renowned for his ability to deliver tailored solutions and hands-on expertise across these critical domains.

Let me tell you about Kenji, a fictional engineer whose story I use whenever someone asks me to explain an error budget without jargon. Kenji ran the online ordering system for a restaurant chain. Every time the system hiccupped — a slow page, a failed payment — his manager would appear at his desk with the same question: “Why isn’t it perfect yet?” And every time, Kenji gave the same honest, doomed answer: “It will never be perfect. Nothing is.” They had this conversation perhaps forty times before either of them realised the actual problem: nobody had ever decided how good the system was supposed to be.

That missing decision has a name, and making it well is one of the most useful skills in modern operations work. This is a plain-language tour of the idea — what it is, why it feels backwards at first, and why it quietly changes how whole teams behave.

The decision nobody makes

Here is the uncomfortable truth Kenji stumbled into: every system already has a reliability level. His ordering platform worked, let’s say, 99.2 percent of the time. The question was never whether failure existed — it demonstrably did — but whether anyone had chosen that number on purpose. Nobody had. So 99.2 percent was simultaneously unacceptable (to his manager, after each incident) and perfectly fine (to everyone, in between incidents). The system’s quality was a matter of mood.

An error budget replaces the mood with a decision. The team, together with the business, picks a target: our ordering system should complete 99.5 percent of orders successfully each month. Not 100 — a number a human chose and can defend. The remaining 0.5 percent is the budget: the amount of failure the business has agreed it can live with.

Why would anyone budget for failure?

This is where newcomers flinch, so let’s take it slowly with an analogy. Imagine a railway that promises its passengers absolute, literal perfection: no train ever late, by even one second, in any typhoon. To keep that promise, the railway would have to stop running trains — standing still is the only way to never be late. The promise of perfection and the act of doing anything are fundamentally in tension.

Software is the same. Every change — new feature, upgrade, configuration tweak — carries a small chance of breaking something. If your target is perfection, the only rational policy is to change nothing, forever. But a frozen system serves the business badly too; competitors ship, customers’ expectations move. So the real question was never “failure or no failure?” It was always “how much failure buys us the right amount of progress?” The budget is simply that trade-off, written down. Reliability and speed of change sit on opposite ends of one see-saw, and the budget marks where you’ve chosen to balance it.

How the error budget changes daily life

Back to Kenji. His team adopted a 99.5 percent target with a monthly window. Here’s what changed, in the order he noticed it.

The arguments got shorter. When a minor outage ate a slice of the budget in week one, there was no desk visit, no mood. The numbers said: budget remains, keep shipping. When a bad deploy in week three nearly emptied it, there was also no argument — the pre-agreed rule said feature releases pause and the team spends its time on stability until the month resets. The same events that used to produce blame now produced a lookup.

Risk-taking became honest. With budget to spare near a quiet month’s end, the team finally attempted a database upgrade they’d been postponing out of vague fear. The budget gave them a vocabulary for courage: we can afford this experiment. That’s the counter-intuitive gift — a failure allowance doesn’t make teams sloppier, it makes them braver at the right times and more careful at the right times.

Incidents turned into lessons. Because each failure now had a measurable cost against a shared budget, writing up what happened stopped feeling like confession and started feeling like accounting. The post-incident review asked what the system allowed, not who to scold — and fixes actually landed, because stability work had earned official time on the calendar.

Teams that reach this state often describe a cultural shift bigger than the mechanism itself — developers and operators stop being natural enemies, because both answer to the same number. If that transformation interests you, it connects deeply to how organisations manage the human side of a DevOps culture change, which is usually the harder half.

Choosing the number (the part that’s genuinely hard)

So who picks 99.5 rather than 99.9? This is a business conversation wearing an engineering costume. The honest method is to ask: who suffers at each level of failure, and how much? For a hobby blog, an hour of downtime costs embarrassment. For an ordering system on a festival evening, minutes cost real revenue and — worse — the kind of customer disappointment that never files a complaint, just quietly orders elsewhere next time. The link between reliability targets and what customers actually feel is the whole game; it’s the same thread explored in why modern delivery practice is ultimately about customer satisfaction.

Two practical tips make the choice easier. Start from today’s measured reality, not from aspiration — a target slightly below your current performance teaches the habit without instant failure. And remember each extra nine costs disproportionately more: moving from 99 to 99.9 percent might mean redundant infrastructure, on-call rotations, and months of engineering. Sometimes that’s worth it. Often it isn’t. The budget forces someone to actually decide instead of assuming.

Three questions people always ask next

“What if the business insists on 100 percent?” Then show them last quarter’s actual numbers. Every running system has a track record below perfection; the only choice is whether to acknowledge it. In Kenji’s case, the chain’s directors discovered they had been operating at 99.2 percent for two years without the company collapsing — which made agreeing to an honest 99.5 surprisingly easy.

“Doesn’t this give engineers an excuse to be careless?” The opposite happens in practice, because the budget is finite and visible. An engineer who burns it on sloppy work has publicly spent the team’s shared allowance for experiments and upgrades. Peer pressure does the policing far better than any manager’s desk visits ever did.

“What if we never use the budget at all?” A chronically untouched allowance is its own warning sign. It usually means the target is too loose, or the team has become so cautious it isn’t shipping enough to matter. Unspent failure allowance is unclaimed progress — the signal to attempt the scary upgrade, not to congratulate yourselves.

Where to learn error budgets properly

Everything above is the intuition; the professional practice has real depth — how to pick the right indicators to measure, how fast an alert should fire when the budget drains, how managers run the pause-and-stabilise conversation without drama. This is now taught as a formal discipline: SRESchool offers site reliability engineering courses and certifications structured as a path, where a beginner starts with the foundation-level CSRE credential and its matching five-day instructor-led course, available online or on-site. For someone building an operations career, it’s a natural next rung after general DevOps study, and the ideas compound with everything else in the field — the reliability mindset shows up in every serious discussion of where DevOps is heading in Japan and globally.

The sentence that changes everything

Kenji’s story ends with his manager — the desk-visitor — presenting the monthly reliability report himself, explaining to the executives that the team had spent 80 percent of its failure allowance and would therefore delay one launch. Same man, same system, completely different conversation. The difference was one written sentence: this is how much failure we accept, and here is what we do when we reach it.

If your team has never written that sentence, you’re living Kenji’s first act — perfection demanded, perfection impossible, everyone pretending otherwise between incidents. Writing it down costs an afternoon. What it buys is honesty, and everything else follows from there.

Leave a Reply