Before-and-After or Side-by-Side? Comparing Two Ways to Test

Rajesh Kumar

Rajesh Kumar is a leading expert in DevOps, SRE, DevSecOps, and MLOps, providing comprehensive services through his platform, www.rajeshkumar.xyz. With a proven track record in consulting, training, freelancing, and enterprise support, he empowers organizations to adopt modern operational practices and achieve scalable, secure, and efficient IT infrastructures. Rajesh is renowned for his ability to deliver tailored solutions and hands-on expertise across these critical domains.

Once a team decides to learn through small experiments, a practical question follows at once: how should the test be arranged? Two designs cover most everyday situations. In a before-and-after test, you make a change and compare the period that followed with the period that preceded it. In a side-by-side test, you run the old and new ways at the same time on two similar groups and compare them directly. Each has a place. Below, each is weighed on the criteria a DevOps practitioner actually cares about, so the choice rests on reasons rather than habit.

The two designs in brief

Before-and-after needs only one environment. You record how things stand, change one element, wait, and record again. It is the natural choice when there is no way to split your users or workload.

Side-by-side needs two comparable streams. Perhaps two services of similar shape, two build agents, or a random share of requests. One stream keeps the old behaviour, the other gets the change, and both are measured over the same hours.

Criterion one: fairness of the comparison

This is where the designs differ most. A before-and-after test is exposed to everything else that changed in the meantime: a busy season, a holiday, an unrelated release, a staffing gap. If response times improved after your change, how much of that came from the change and how much from a quieter week? You cannot be sure.

A side-by-side test absorbs those outside influences because both groups experience the same calendar. Whatever disturbs one disturbs the other. The remaining risk is that the two groups were not truly alike to begin with, so choose them carefully and, where possible, assign members at random.

Verdict: side-by-side is the fairer design whenever you can manage it.

Criterion two: cost and effort

Before-and-after wins here. It requires no parallel set-up, no splitting logic and no second pipeline. A single engineer can run one on an afternoon. Side-by-side tests demand a mechanism to separate groups, such as a flag or a routing rule, and a way to keep the two measurements apart.

For a first experiment, that difference is considerable. Teams new to the habit often begin with the cheaper design to build confidence, then graduate to side-by-side as their tooling matures.

Verdict: before-and-after is cheaper and simpler to start.

Criterion three: risk if the change is bad

In a before-and-after test, everyone receives the change, so a faulty one affects the whole population until you notice and revert. A side-by-side test limits exposure to the test group, which can be kept deliberately small. When a mistake would be costly, that containment is a strong argument for the second design.

Verdict: side-by-side contains damage; before-and-after does not, unless the change itself is trivial.

Criterion four: speed of learning

Before-and-after must wait for enough time to pass in each period to give a stable reading. Side-by-side collects both readings simultaneously, so a given amount of evidence arrives sooner. If your traffic is thin, however, splitting it further makes each half even thinner, and a before-and-after test over a longer stretch may be the more practical choice.

Verdict: depends on volume. High volume favours side-by-side; low volume favours patience with before-and-after.

Criterion five: ease of explaining the result

Stakeholders grasp a before-and-after story effortlessly: “this was the figure last month, this is the figure now.” Side-by-side results require a sentence or two of explanation about how the groups were chosen. Yet the second is harder to dispute, because it answers the usual objection, “but surely something else changed?”

One scenario, two designs

Suppose a team wants to know whether a lower timeout on an internal service reduces the number of stuck requests. Under a before-and-after design, they note the count of stuck requests for a representative stretch, lower the timeout, and count again for a similar stretch. The weakness is plain: if the second stretch happened to include a quiet period, the improvement might be illusory.

Under a side-by-side design, they apply the lower timeout to half the instances behind the same balancer and leave the rest alone, then count stuck requests on each half over the same hours. The calendar now affects both halves equally, so a difference between them is far more likely to come from the timeout. The price is a little configuration work and a need to confirm the two halves are comparable, for instance in hardware and typical load.

A hybrid worth knowing

Some teams combine the two: they run a short before-and-after check to see whether anything dramatic happens, then follow with a side-by-side test only if the first look is promising. This staged approach spends little on ideas that fizzle and saves the rigorous design for the ones that justify it.

A summary at a glance

  • Fairness: side-by-side stronger.
  • Cost: before-and-after cheaper.
  • Containment of risk: side-by-side stronger.
  • Speed with ample volume: side-by-side faster.
  • Simplicity of storytelling: before-and-after easier.

Choosing between them in practice

A few rules of thumb help. If you can split cleanly and the change could do harm, go side-by-side. If you cannot split, or the change is cosmetic and trivially reversible, before-and-after is perfectly acceptable, provided you note any other events during the test window and treat the result with appropriate caution. If you are unsure, run before-and-after first as a rough look, then confirm any promising finding with a side-by-side test.

Whichever you pick, the discipline is common to both: pose one question, change one thing, fix the measure and the duration in advance, and write down what you saw. A badly designed side-by-side test is no better than a careful before-and-after one.

Where organisational context enters

Experiments do not happen in a vacuum. The culture around them determines whether results are believed and acted on. A thoughtful discussion of managing cultural change when adopting DevOps ways of working is relevant here, because people accept evidence more readily when they trust the process that produced it. Similarly, an outlook piece on where DevOps practice is heading, locally and globally reminds us that the tooling will keep evolving while the logic of comparison stays the same.

A parallel from another field

The same choice appears in any community that tests ideas. Consider DronesBee, a drone-focused site with blog and news pages, listings, a shop, a forum and directories for service providers, operators and pilots. Members who post about trying something new in their own practice are in effect sharing before-and-after notes, and those who compare two approaches in one thread are sharing side-by-side findings. Different subject, identical reasoning.

Common mistakes in both designs

  • Changing more than one thing. It makes either design uninterpretable.
  • Choosing the measure afterwards. Selecting the favourable figure once the data is in invalidates the comparison.
  • Stopping at the first good result. Short runs are vulnerable to chance.
  • Mismatched groups. In side-by-side tests, a quiet service compared with a busy one proves nothing.
  • Forgetting the exit. Know how to revert before you begin.

Recording the design alongside the result

Whichever design you use, the note you keep afterwards should state which one it was. A reader who finds a result without its design cannot judge how much weight it deserves. Add one line to your template: “design: before-and-after” or “design: side-by-side, groups chosen by this rule”. It takes seconds and makes the record far more useful to the colleague who consults it a year from now.

Conclusion

Neither design is superior in every case. Before-and-after is the pragmatic starting point: quick, cheap and good enough for low-stakes changes. Side-by-side is the more rigorous instrument and the better guard against harm. A mature team keeps both within reach and picks according to the question, the risk and the traffic available. What matters most is that the decision about the design is made deliberately, in writing, before the experiment starts, so that the result, whichever way it points, can be trusted and put to use.

Leave a Reply