Keep the gains: performance regression testing without fooling yourself

TL;DR: Keep only the benchmarks that protect important hot paths. Use baseline/diff comparisons to catch regressions, but do not trust noisy shared runners blindly. Performance regression testing needs stable machines, clear thresholds, and restraint.

After a successful performance investigation, the benchmark folder looks valuable.

It contains the history of the work: experiments, false starts, before-and-after comparisons, warmup cases, exception cases, and small probes that helped explain the code. Keeping all of it feels safe, and deleting any of it feels strangely reckless.

That instinct leaves the team maintaining experiments long after they have answered their questions.

Benchmarks have a maintenance cost. They take time to run, understand, and fix when the production code changes. The useful measure is whether the suite protects important performance promises, not how many benchmark classes survive in the repository.

Most benchmark experiments should expire

Many benchmarks are investigation tools. They help answer a question during the improve-and-compare loop. Once the question is answered, the benchmark may no longer deserve permanent continuous integration time.

Keep the benchmarks that protect core hot paths. In the NServiceBus pipeline case, a benchmark for pipeline invocation can make sense because every message goes through that path. A regression there can affect many users across many transports.

Protecting that benchmark is worth the maintenance cost.

A one-off benchmark that tested an abandoned idea is different. Keep the lesson, not necessarily the code. Move the finding into the pull request, a decision record, or a short note near the optimized code. Future readers need to know why the code looks the way it does. They do not need every experiment to run forever.

• Keep benchmarks for shared infrastructure hot paths.
• Keep benchmarks that protect public contracts or expensive operations.
• Delete or archive one-off experiments after the lesson is captured.
• Document what was measured, what changed, and why the trade-off was accepted.

Catch regressions by comparing benchmark history

Performance regression testing usually compares two versions of the same benchmark. Run the benchmark at a known baseline commit. Store the artifacts. Move to the candidate commit or branch. Run the same benchmark again. Compare the results with a threshold.

The .NET performance repository includes a ResultComparer tool that can compare BenchmarkDotNet artifacts. The exact commands depend on your repository layout, but the workflow looks like this:

git checkout baseline-sha

dotnet run -c Release --artifacts "C:\results\before"

git checkout candidate-sha

dotnet run -c Release --artifacts "C:\results\after"

dotnet run --project C:\Projects\performance\src\tools\ResultsComparer \
  --base "C:\results\before" \
  --diff "C:\results\after" \
  --threshold 2%

The threshold is policy. Two percent might be reasonable for one benchmark and meaningless for another. Choose the threshold based on observed variance and business impact, not because the number looks tidy.

Thresholds are policy, not math magic

A benchmark result is a distribution, not a single truth. The same code can produce slightly different measurements from run to run. The machine, operating system, runtime, background processes, CPU frequency, temperature, and neighboring workloads all get a vote.

Before turning a benchmark into a failing gate, learn its natural variance on the machine that will run it. If a benchmark moves by three percent when nothing changed, a two percent regression threshold will create noise. The team will learn to ignore it, and then the gate has failed socially even if it works technically.

CPU-bound benchmarks are often more stable than memory-bound or disk-bound benchmarks, but stability is not guaranteed.

Measure the variance before deciding what the gate should enforce.

Shared runners can lie to you

Shared continuous integration runners are convenient. They are also shared. Another build on the same host can affect your measurements. The hardware can differ between runs. Power settings and virtualization layers can change the timing profile. That is the noisy-neighbor problem.

Two subsequent builds on the same revision can have ranges of 1.5..2 seconds and 12..36 seconds. CPU-bound benchmarks are much more stable than Memory/Disk-bound benchmarks, but the “average” performance levels still can be up to three times different across builds.

Andrey Akinshin, quoted in the BeyondSimpleBenchmarks talk material

That does not mean shared runners are useless. They can still compile benchmarks, run smoke checks, or provide a rough signal. But a flaky performance gate is worse than no gate.

Developers learn to rerun jobs until they pass, and performance work starts to feel like superstition. Eventually the team assumes every benchmark failure is noise.

Performance culture is social as much as technical. If people stop trusting the signal, the tooling has already lost.

Use stable hardware when the gate matters

If the benchmark is important enough to block a pull request, the machine should be stable enough to support that decision. That may mean a dedicated bare-metal runner, a controlled virtual machine, a lab machine, or a manually triggered benchmark pipeline for risky changes.

Not every team needs a performance lab. A team that is just starting can run benchmark experiments locally, review the results manually, and build shared knowledge. That is already progress. Automating noisy benchmarks too early can create more frustration than value.

Running every benchmark on every pull request is automation, but it is not necessarily maturity.

A mature team knows which performance promises are stable and important enough to defend automatically.

Regression tests still need human judgment

A regression is not always a bug. Sometimes a slower implementation fixes correctness, improves security, removes a dangerous shortcut, or makes the system easier to maintain. The benchmark should start a conversation, not replace one.

When a benchmark fails, ask what changed. Did the hot path slow down because of accidental allocations? Did a new feature add necessary work? Did the benchmark become invalid because the production code changed shape? Did the machine have a bad run?

The best performance gates make accidental regressions cheap to catch and intentional trade-offs explicit. They should not make teams afraid to improve the design.

Close the loop with continuous improvement

The full performance loop is still the foundation: profile with a profiling harness, improve a hot path, benchmark and compare, profile again, ship, and observe production. Regression testing is a maturity step that protects the gains after the team knows what matters.

This approach also pushes against rewrite culture. It is easy to look at old code and say, “This is slow. We should rewrite it.” Sometimes a rewrite is right. Most of the time, the team needs more knowledge first.

Otherwise, the rewrite repeats the old mistakes with newer code and better formatting.

Profiling and benchmarking build the missing knowledge. They show which paths matter, which assumptions were wrong, and which trade-offs paid off. After a few loops, the team may not need a rewrite. If it still does, the rewrite starts with evidence instead of frustration.

Performance knowledge accumulates. A small improvement can expose the next bottleneck, a useful benchmark protects the path, and production observations correct assumptions made in the lab. The team gradually has less reason to guess.

Start with one hot path and build a profiling harness around it. Take memory and CPU profiles, improve one thing, benchmark it, and profile again. Then write down what you learned.

The profile points at the work and the benchmark tests the change. Production then exposes whatever the lab missed.

Repeated often enough, this becomes ordinary engineering work rather than a rescue mission after performance has already collapsed.

Further reading

• .NET ResultComparer
• Preventing regressions in the .NET runtime benchmarking workflow
• Andrey Akinshin on GitHub Actions performance stability
• BeyondSimpleBenchmarks repository

Common questions

Should performance benchmarks run on every pull request?

Only if the benchmark is stable, fast enough, and important enough. Otherwise, run it on demand, nightly, before release, or when a change touches the protected hot path.

What should I do if continuous integration results are noisy?

Measure variance, loosen or remove the gate, use dedicated hardware, or treat the result as a signal rather than an automatic failure. Do not keep a flaky performance gate just because it feels rigorous.

How many benchmarks should a team keep?

Keep the benchmarks that protect meaningful performance promises. If nobody can explain what decision a benchmark supports, it probably does not belong in the permanent suite.

Performance loop status

• [x] Understand the loop
• [x] Build profiling harness
• [x] Profile
• [x] Improve / Benchmark
• [x] Profile again / Ship and observe
• [x] Protect the gains