Here is a pattern that repeats across engineering organisations in Japan and everywhere else: a company spends generously on upskilling — certification courses, workshops, coaching — and eighteen months later nobody can say what changed. The training was real. The engineers learned. Yet the investment shows up nowhere anyone can point to, and the next budget round cuts it. The diagnosis is almost never “bad training.” It is training without a baseline — improvement attempted without measurement, which makes progress invisible even when it happens. This piece works through the problem analytically, then lays out the fix.
The symptom: learning that leaves no trace
Examine how most organisations evaluate training and the problem announces itself. The evidence collected is almost entirely input-side: attendance, completion certificates, satisfaction surveys. All three measure whether training occurred, not whether the operation changed. Meanwhile the outcome-side evidence — are deployments calmer, are reviews sharper, are incidents rarer — is never captured in comparable form before and after. Without a before, there is no after. The organisation is trying to detect a difference between two photographs while holding only one of them.
The consequences compound predictably. Engineers sense their new skills dissolving into old workflows and disengage. Finance, seeing cost with no measured return, reclassifies training as a perk. And leadership concludes — wrongly — that upskilling does not work, when what failed was the measurement, not the learning.
The misdiagnoses that keep teams stuck
Three wrong explanations usually get tried first, and each is worth ruling out explicitly.
“We chose the wrong course.” Sometimes true, unverifiable without a baseline. If you cannot show which practices were weak before the course, you cannot show the course targeted the wrong ones.
“The engineers didn’t apply it.” Occasionally true, but normally a symptom: skills go unapplied when the surrounding process never changes, and process change was never scheduled because no audit identified which process needed it.
“Results just take longer.” The comfortable explanation, and untestable by construction — any horizon can always be extended. An unfalsifiable claim is not evidence; it is a delay tactic.
All three misdiagnoses share one root: the organisation has no scored picture of its own practice. That is the condition to treat.
The solution: wrap training inside an audit cycle
The fix is structural and borrowed from a discipline that solved it at scale. Enterprise software delivery faced the identical problem — large transformation spend, invisible returns — and the emerging answer is the periodic maturity assessment. SCMGalaxy, a technology training and consulting company, built SCMGalaxy OS precisely on that model: a question bank one hundred strong covering distinct governance areas, scoring engineered so identical answers return identical numbers, prioritised findings, and improvement plans staged across coming months — then re-assessment to track movement. The architecture matters more than the product: fixed instrument, honest score, dated plan, second measurement. That loop converts any investment, training included, from an act of faith into an experiment with a readable result.
Applied to upskilling, the cycle has four movements.
1. Score practice before choosing courses
Audit the team’s working habits with a written question set, scored against rules fixed in advance. Keep it evidence-based: inspect how changes are branched and reviewed rather than asking whether reviews are “good”; inspect whether production behaviour is actually watched rather than asking whether monitoring “exists”. A dozen questions across four or five areas suffices for a first sitting. The output — scores per area — is your baseline, and it transforms course selection from fashion-following into targeting. Low review scores argue for review training; low automation scores argue for pipeline work. The targeting logic resembles choosing a personal study path from honest self-assessment, the approach behind a realistic route into contemporary DevOps careers — except performed at team scale.
2. Attach the training to a dated change
Every course booked must correspond to a line in a staged plan: which practice it strengthens, what observable change follows, by when. Stage the plan at one month, three months and six months — urgent corrections first, structural adoption second, cultural consolidation last. The point is that training stops being an event and becomes a component of a scheduled improvement, with an owner whose name appears next to it.
3. Change the process alongside the people
Skills decay inside unchanged workflows. If engineers learn trunk-based development but the release process still demands month-long branches, the course was wasted by design. Each training line in the plan therefore pairs with a process edit, however small — a pipeline gate added, a review rule adopted. This pairing is the operational meaning of improvement practised as a continuous craft rather than consumed as a product.
4. Re-sit the audit and read the delta
After the six-month horizon, administer the same questions, marked by the unchanged rules. The comparison is the entire payoff. Areas that climbed vindicate the spend in a form finance accepts — a number that moved. Areas that stayed flat localise the failure: this course, this process, this quarter, rather than “training doesn’t work.” Either way the organisation has learned something true, which is more than the certificate-counting approach ever yields.
Why the evidence favours this design
Notice what the audit cycle fixes about each earlier misdiagnosis. “Wrong course” becomes testable — the baseline shows whether the weak area was targeted. “Didn’t apply it” becomes visible — the paired process change either happened or did not. “Takes longer” acquires a deadline — the re-sit date. Unfalsifiable excuses become falsifiable claims, which is the signature of a healthy improvement system.
There is also a second-order benefit. Scored practice audits correlate with outcomes customers feel — calmer releases, faster recovery — the causal chain traced in how delivery discipline translates into customer satisfaction. A team that can show its practice scores rising holds evidence that the training budget is, indirectly, a customer-experience budget. That argument wins renewals.
One caution: do not let the metric become the mission
An analytical piece owes you the counter-case. Any score pursued hard enough stops measuring what it was built to measure — teams learn to satisfy the question rather than the practice behind it. Two design choices keep the audit on the right side of that line. Keep the questions pointed at artifacts of genuine work (a rehearsed rollback, a reviewed change) rather than at easily manufactured tokens (a document’s existence, a box ticked), so gaming the question requires doing the thing. And treat score movement as a prompt for conversation, never as a bonus trigger; the moment money rides directly on the number, the number’s honesty is spent. Organisations that respect these limits get a durable instrument. Those that don’t get a very accurate record of their own theatre.
The prescription, compressed
If your organisation’s training produces certificates but no visible change, stop buying courses for one quarter and build the measurement instead. Write the question set. Score honestly. Choose the next course from the lowest numbers. Date the expected change, edit the process to receive the new skill, and book the second sitting before the first invoice arrives. Platforms like SCMGalaxy OS demonstrate the pattern at enterprise scale, but the pattern itself fits in a spreadsheet and an afternoon.
Training without a baseline is spending without an experiment. Add the baseline and the re-measurement, and the same spend starts compounding — visibly, defensibly, and in numbers that survive the budget meeting.