Diagnosing a Broken Incident Response: Seven Symptoms, Seven Fixes

Rajesh Kumar

Rajesh Kumar is a leading expert in DevOps, SRE, DevSecOps, and MLOps, providing comprehensive services through his platform, www.rajeshkumar.xyz. With a proven track record in consulting, training, freelancing, and enterprise support, he empowers organizations to adopt modern operational practices and achieve scalable, secure, and efficient IT infrastructures. Rajesh is renowned for his ability to deliver tailored solutions and hands-on expertise across these critical domains.

Haruka runs the platform group at a mid-sized online retailer. Her team is skilled, her tooling is modern, and yet every serious outage leaves everyone more exhausted than it should. She describes the same pattern each time: too many voices, thin updates, and a review that produces a long list nobody finishes. Her story is a composite built to illustrate common trouble, not a real company. What follows is a diagnosis in the style of a clinic visit: symptoms first, then likely causes, then the treatment. The treatment comes from the discipline called IncidentOps.

Symptom one: the chat room fills with a crowd

What it looks like. The moment a major alert fires, dozens of people join the channel. The few who are actually changing production retreat into private messages, and the main thread turns into commentary.

Diagnosis. Nobody has defined who belongs where. The Ops Atlas, summarising field research on large outages, describes exactly this crowd effect, with hundreds of participants forcing the core group to hide in side channels. Its own case from a cloud provider’s cluster failure saw 41 people in the chat for more than six hours.

Treatment. Split the space in two: a working room limited to the people hands-on, and a read-only stream for everyone else. Add a coordinator whose job is to keep the working room small. The Atlas also borrows an emergency-services rule of thumb, which is one supervisor for about five people, with sub-teams beyond that.

Symptom two: updates arrive late and say little

What it looks like. Customers hear nothing for a long stretch, then a vague note. Support staff improvise answers.

Diagnosis. No one owns communication, so everyone assumes someone else will write. The most striking example in the Ops Atlas entry covering IncidentOps is a 2022 data-deletion event at a software company in which the first broad public message, a social-media post, appeared about 41 hours after deletion began, even though a status page had been touched earlier.

Treatment. Appoint a communications lead at the start. Aim to publish an opening note while the call is barely under way, then refresh no more than twenty minutes apart early on. Close with a plain statement about whether data was lost. Haruka’s team wrote three template sentences for each stage so the writer never faced a blank page.

Symptom three: the status page is down when you need it

What it looks like. The public page runs on the same platform as the product, so an outage takes both away.

Diagnosis. Shared fate. The Atlas gives a famous 2017 storage outage as the reference case: the provider’s service health dashboard relied on the very service that was failing, and it was unreliable until late morning, so the company turned to social media and banners instead.

Treatment. Host the status page and the paging path outside the systems they report on. Test it. A heartbeat alert, a signal that fires if the alerting chain itself falls silent, tells you the safety net is still there.

Symptom four: everybody waits for the cause

What it looks like. The room debates theories for an hour while customers keep failing at checkout.

Diagnosis. Confusing understanding with stopping. The commander never asked the simplest question: can we undo the last change?

Treatment. Teach the team the short menu of generic mitigations: roll back, drain, switch a feature off, fail over, shed load. They only demand that you know roughly where the fault sits. In the cluster case above, the write-up estimates that reverting once the fault had been found would have ended things around ten in the morning rather than near noon.

Symptom five: the review produces a long, vague list

What it looks like. Twenty action items, equally weighted, several of them “improve monitoring”. Three months later half remain open.

Diagnosis. The Atlas lists the warning signs: actions that are mostly mitigative, vague wording, equal priority for everything, reviews that merely count tickets, and closing items by quietly downgrading them.

Treatment. Cut the list to a few actions. Give each one owner and one date. Agree a service-level target for completing the important ones, in the manner of one big software vendor, whose window stretches from one to two months by service tier, and track the proportion closed on time. Ask someone who was not deeply involved to facilitate, and write down the owner of the review before the incident call even ends.

Symptom six: the baton drops at shift change

What it looks like. A long incident crosses into the next person’s morning, and the newcomer spends an hour rebuilding context while the old team has already gone offline.

Diagnosis. The handover was casual. Nobody confirmed who held command, and the notes lived in someone’s head. The Atlas mentions a real handover between London and Seattle in the middle of a long outage as an ordinary, not exceptional, event.

Treatment. Make the transfer explicit and spoken: the outgoing commander tells the incoming one that command is theirs now and waits for a clear yes. Keep a running timeline in the incident channel so the scribe’s notes carry the context across the gap.

Symptom seven: the numbers look wonderful, then terrible

What it looks like. Quarterly average recovery time swings wildly, so leaders either celebrate or panic.

Diagnosis. Averages of skewed data. The Atlas notes that incident durations are lopsided, that the common abbreviation covers at least four different measurements, and that one large outage in a small sample can move the mean by tens of minutes.

Treatment. Report distributions and always state the phase and population. Add pager load over a trailing window and the share of actions closed on time.

Why the cure works better than the pill

It is tempting to answer each symptom with a purchase: a bigger chat licence, a nicer dashboard, another alert rule. The diagnosis above suggests otherwise. Every one of the seven problems is about agreements between people, such as who speaks, who writes, who decides and who confirms. Software can carry those agreements, but it cannot invent them. Teams that start with the agreements find that even modest tools suddenly feel powerful, while teams that start with tools often discover they have bought an expensive way to be confused faster.

What changed for Haruka’s team

They did not buy a new product. They wrote one page of declaration criteria, printed role cards, moved the status notice to a separate host and held a ninety-minute tabletop exercise with a made-up failure. At the next real incident, the coordinator said her role aloud, the communications lead posted within minutes and the group rolled back before debating causes. Nobody had to be a hero, and the people who left at a normal hour went home feeling the system had worked.

If you would like to place this work in a wider picture of delivery practice in the region, a look ahead at where DevOps is going in Japan and abroad is a good starting point, while the practical guide on managing risk for dependable delivery explains why these routines belong in a risk plan.

Check yourself

Count how many of the seven symptoms feel familiar. One or two is normal. Four or more suggests the team has been surviving on goodwill. Pick the symptom that hurts most, apply its treatment for a month, and review again.

Leave a Reply