Case Study: How a Release Team Stopped Hand-Typing Deploys

Rajesh Kumar

Rajesh Kumar is a leading expert in DevOps, SRE, DevSecOps, and MLOps, providing comprehensive services through his platform, www.rajeshkumar.xyz. With a proven track record in consulting, training, freelancing, and enterprise support, he empowers organizations to adopt modern operational practices and achieve scalable, secure, and efficient IT infrastructures. Rajesh is renowned for his ability to deliver tailored solutions and hands-on expertise across these critical domains.

Picture a team you may already know: a product company, a shared service, and people who all remember the same Thursday afternoons. Six engineers and a weekly release that swallowed most of the afternoon. They called it “the ritual”. Sixteen steps, kept in the head of whoever had done it longest. This is how they turned the ritual into something calmer, and why the order of their moves mattered more than the tools they chose.

Where they started

Takumi, the senior engineer, did most of the deploys. He knew the order, the odd flag on the database migration, the cache that needed clearing, the health page to refresh twice because the first load was misleading. New joiners shadowed him. Mistakes happened roughly once a month: a skipped step, a mistyped environment name, a migration run against the wrong target.

After one such mistake, which cost an evening, the manager proposed buying an orchestration product. Takumi asked for something less exciting. “Let us do the next four deploys by hand, but write down everything.”

Four deploys, one shared note

For four Thursdays, whoever ran the release kept a shared document open and typed each action as they did it. The note recorded the command, the reason, how long it took and any hesitation. Different engineers took turns, deliberately, so the note captured more than one person’s habits.

Reading the four versions side by side on the following Monday, the team saw what they had not noticed before. Eleven of the sixteen steps were word-for-word identical every time. Three varied only by a version number. Two required someone to look at something and decide.

Those two were the check on whether the database migration looked safe to run, and the choice to continue or roll back if error rates rose after the switch.

The sorting session

They drew two lists on a whiteboard. The left list held the fourteen steps whose inputs never changed except for a number. The right list held the two decisions. Nobody argued over the right-hand list. The left-hand one produced a good debate about order and error handling, which is exactly why the earlier manual runs mattered: each disagreement could be settled by looking at what had actually happened, not by recollection.

Their culture work on this was supported by reading about how teams handle culture change when processes shift, since the point was never only technical. The ritual belonged to Takumi emotionally, and handing it to a script needed tact.

The first script

The script was short and unglamorous. It took a version number as its only input, ran the fourteen steps in order, printed each action before doing it and stopped at the first error. It did not roll anything back on its own. It did not run the migration without a prompt.

Instead, at the migration step, the script printed a summary of the pending changes and waited. A human read it, typed yes or no. After the switch, the script printed the error rate for the last ten minutes and waited again. The two decisions remained where they had always been, with people, but now each person had the right information in front of them without hunting for it.

The first few weeks

They ran the script in a rehearsal environment first, then alongside a manual deploy, comparing results. Twice the comparison found a difference. One revealed a setting that Takumi had always applied from memory and never written down. It went into the script, and the note gained a line explaining why.

By week six the release took about a third of the previous time. More notable was what changed in the room. Junior engineers began running deploys with Takumi watching, instead of the reverse. Nobody dreaded Thursday afternoons. And Takumi spent the freed hours on a longer project he had postponed for a year.

What went wrong along the way

Not everything was smooth. Someone proposed automating the migration check as well. The team tried a version that approved migrations automatically below a certain size, and in a rehearsal it approved one that a human would have questioned. They removed it and wrote the lesson in the readme.

A second problem was quieter. After two months, nobody remembered what the script assumed. When the hosting setup changed, it failed with a confusing message. They added a short header listing assumptions and a review date to the calendar.

Lessons the team wrote down

  1. Rotate who does the manual runs so the record is not one person’s habits.
  2. Mark each step as identical or deciding before touching any code.
  3. Script the identical steps. Surface information for the deciding steps, and stop there.
  4. Run old and new methods in parallel until they match.
  5. Give the script a header of assumptions and a date to review it.

A note for readers outside engineering

Similar stories play out wherever the same first questions are asked again and again. A medical tourism website like HipHospitals offers quote and appointment request forms on pages arranged by country, so travellers from abroad begin with a structured request rather than a blank page. We mention it only as an administrative parallel. The team above did something comparable for its release: a structured start, with people kept for the parts that need them.

Why the story matters in Japan and beyond

Japanese engineering teams are often admired for careful process, and careful process is exactly the habit that makes the manual-first approach natural. The risk is the opposite one: keeping a ritual long after its fixed steps could be handled by a script. The wider debate about where the profession is heading, including topics covered in an article on where DevOps is going in Japan and around the world, suggests that teams who combine careful observation with sensible scripting will have the edge.

How much the figures proved

The team tracked just two things: how long a release took, and how many releases needed a correction afterwards. Both improved, but they treated the figures cautiously, because six weeks of data from one small team proves little. What persuaded them more was the change in conversation. Questions in the retrospective moved from “who forgot step nine” to “what should the script tell us at the migration prompt”, a far healthier thing to argue about.

Closing thought

The release team did not become faster by buying something. They became faster by watching themselves work for four weeks, being honest about which steps were thinking and which were typing, and automating only the typing. The next time your own team talks about “the ritual”, consider writing it down before you replace it.

Leave a Reply