From Ticket Desk to SRE: A Fictional Team’s Six Chapters

Rajesh Kumar

Rajesh Kumar is a leading expert in DevOps, SRE, DevSecOps, and MLOps, providing comprehensive services through his platform, www.rajeshkumar.xyz. With a proven track record in consulting, training, freelancing, and enterprise support, he empowers organizations to adopt modern operational practices and achieve scalable, secure, and efficient IT infrastructures. Rajesh is renowned for his ability to deliver tailored solutions and hands-on expertise across these critical domains.

Nobody in this story is real, and the employer is invented too. Treat both as a composite sketch assembled to illustrate how site reliability engineering, or SRE, tends to unfold inside an ordinary company. The details of the method come from the SRE entry in the Ops Atlas, which is quoted here for definitions and arithmetic; everything else is invention for teaching purposes. Names are replaced with roles so that no one in the real world is implicated.

Chapter one: the helpdesk with a new badge

A mid-sized online retailer, call it the shop, had an operations group of seven people who answered alarms, restarted services and shepherded releases. Leadership decided to “adopt SRE” and changed the group’s name on the org chart. For a quarter, nothing else moved. The tickets kept coming, releases stayed scheduled by committee, and morale dipped because the new title raised expectations without changing the workload.

The Atlas has a label for this situation: a renamed operations team, sometimes called fake SRE. The team lead, whom we will call the lead engineer, read the entry on a train and underlined a different idea: the practice begins with a user-facing promise and a number, not a title.

Chapter two: one promise, measured honestly

The lead engineer picked the checkout path. The indicator was the proportion of checkout requests that completed successfully, observed from a probe outside the shop’s own network. A second indicator counted the share of requests answered within a fixed time limit, expressed as a proportion instead of an average. Initially the team proposed a 99.99 percent objective because it sounded serious. A quick sum cooled that enthusiasm: the allowance over four weeks would be about four minutes, and the checkout depended on a payment provider promising less than that.

They settled on 99.9 percent across a rolling 28 days. The budget came to about forty minutes of bad time, or one thousand failures per million attempts. The sum was written on a whiteboard by the door and stayed there for a year.

Chapter three: the argument that changed shape

The first test arrived within a fortnight. A product manager wanted a new recommendation panel released before a sale. In the old world, the discussion would have been a contest of seniority. In the new one, the lead engineer opened the budget chart: two incidents earlier in the month had already used well over half of it. The panel could still ship, but behind a canary reaching five percent of visitors first. Had a defect in the panel produced errors on a fifth of those requests, the overall error rate would have risen by only one percentage point, an effect small enough to detect and cheap to undo.

The panel shipped, a defect did appear, and the canary caught it. The team noted a statistic from the Atlas on why they bothered: roughly seven out of ten outages trace back to changes made on live systems. That figure stuck with the product manager more than any lecture could have.

Chapter four: the alert that finally made sense

The old alarm setup had dozens of threshold rules. The team deleted most of them and replaced the set with budget-based alerts. A fast drain, around fourteen times the sustainable pace for an hour, would page someone. A moderate one, about six times for six hours, would page too. Slow leaks, three times over a day or the steady pace over three days, would create tickets only. During a test replay of a month of traffic, a short severe incident would have paged within roughly a quarter of an hour, while a long mild one would have sat quietly in the ticket queue.

That quiet incident taught the team something they would not have guessed: a half-percent error rate lasting sixteen hours can consume a large share of a month’s budget without ever waking anyone. They added a weekly review of ticket-lane burn to make sure slow leaks did not go unseen. Readers curious about how a delivery culture and customer outcomes relate might enjoy this discussion of modern delivery and customer satisfaction.

Chapter five: toil, counted at last

The lead engineer asked each person to log manual, repetitive work for four weeks. Seven people, each with roughly eighty hours of working time in the period, made the total about 560 hours; the log showed around 230 spent on repeated chores. That is slightly above forty percent, comfortably under the guideline ceiling of fifty yet higher than the typical third seen in industry surveys. The team picked the three biggest chores and turned them into scripts, and they negotiated one structural change: tickets that belonged to the development teams went back to them rather than landing on the shop’s reliability group by default.

Chapter six: a review that closed

Six months in, a configuration mistake took checkout down for twenty-five minutes. The review followed the blameless pattern: no search for a culprit, but a firm list of actions with a single accountable person and a deadline beside each. One of those actions, a validation step before configuration reaches production, was finished within the week. The budget policy, signed earlier by a senior executive, meant that releases paused for the rest of that window without anyone needing to argue. For related thinking on how reliability and risk planning fit together, see a practical piece on balancing risk and dependable delivery.

Epilogue: a deliberate dip

A year on, the checkout objective had been met so comfortably that the budget was nearly untouched each month. The lead engineer treated that as a signal, not a victory. Either the target was too loose or the team was being more cautious than customers required. After a discussion with the product group, they raised the bar slightly and used the extra headroom to ship two long-delayed improvements in the same quarter. The experience echoed a story in the Atlas about an internal lock service that was so dependable that its owners scheduled planned outages to keep dependants honest. The shop never went that far, but it did start a quarterly drill in which one non-critical dependency was switched off for a few minutes to see who noticed.

Lessons readers can borrow

  • A title change alone does not alter outcomes; objectives, budgets and policies do.
  • Choosing a sensible target means checking dependencies first, not copying an impressive-looking figure.
  • A number turns release arguments into planning conversations.
  • Different incident shapes need different alert lanes.
  • Toil must be counted before it can be reduced.
  • Reviews only matter if their actions are finished.

Fictional as it is, the arc resembles what many real teams report. If you try it, start with one path, one target, and one signed policy, and let the second and third services wait until the first has delivered a lesson worth repeating.

Leave a Reply