TL;DR: Useful benchmarks are controlled experiments. Copy the relevant production code, trim away unrelated work, choose realistic parameters, measure one responsibility, and use short runs for direction before spending time on full runs.
Most benchmark examples look cleaner than the code we work with.
Suspiciously cleaner.
They compare string concatenation with StringBuilder. They call a static method. They pass one value in and return one value out. Those examples are useful for learning BenchmarkDotNet, but production code rarely looks like that.
Production software is usually a disgusting festering mess, but it makes money. It has dependency injection, callbacks, options, asynchronous boundaries, global state, caches, feature flags, and a decade of decisions hiding in the corners.
That mess does not mean we cannot benchmark it. It means we need to treat the benchmark as a controlled experiment.
Benchmarks only look like unit tests
The first trap is mental. A BenchmarkDotNet benchmark looks a little like a unit test: a class, attributes, methods, and a runner. But the result is completely different.
A unit test gives a pass or fail signal. A benchmark produces measurements: mean, median, standard deviation, allocation counts, ratios, outliers, and distributions. To get those measurements, the benchmark runs many times.
That changes what we should measure. Unit tests should cover edge cases. Benchmarks should focus on common hot-path cases unless an edge case is operationally important.
A benchmark should answer one cost question clearly. If it tries to answer five, the result becomes hard to trust.
Copy the code, then remove the noise
For the pipeline investigation, I copied the relevant pipeline infrastructure into a dedicated benchmark project. That sounds wrong because copy-paste is how many codebases get into trouble.
Here, copying the code made the experiment safer.
We could freeze the code under investigation and control exactly what the experiment included.
Once the code was copied, the cutting started. Irrelevant behaviors came out. The dependency injection container came out. Real input/output was replaced with completed tasks. The benchmark did not need to compare containers, transports, databases, serializers, or cloud services. It needed to compare pipeline invocation before and after the proposed optimization.
This kind of benchmark is not a second implementation of the product. It is a lab bench. The value comes from making the experiment small enough to understand while keeping enough real code that the result still means something.
The result was controlled without being detached from the production code.
Measure one thing at a time
A good benchmark follows the single responsibility principle. If the question is “how fast does the pipeline execute,” then pipeline construction should not be part of the measured method. Setup belongs in [GlobalSetup]. The benchmark method should execute the pipeline.
[ShortRunJob]
[MemoryDiagnoser]
public class PipelineExecutionBenchmark
{
BaseLinePipeline<IBehaviorContext> pipelineBeforeOptimizations;
PipelineOptimization<IBehaviorContext> pipelineAfterOptimizations;
BehaviorContext behaviorContext;
[Params(10, 20, 40)]
public int PipelineDepth { get; set; }
[GlobalSetup]
public void SetUp()
{
behaviorContext = new BehaviorContext();
pipelineBeforeOptimizations = CreateBeforePipeline(PipelineDepth);
pipelineAfterOptimizations = CreateAfterPipeline(PipelineDepth);
}
[Benchmark(Baseline = true)]
public Task Before()
{
return pipelineBeforeOptimizations.Invoke(behaviorContext);
}
[Benchmark]
public Task After()
{
return pipelineAfterOptimizations.Invoke(behaviorContext);
}
}
The shape matters. [MemoryDiagnoser] adds allocation data. [Params] runs the same benchmark at several pipeline depths. [GlobalSetup] keeps setup out of the measured path. The two benchmark methods compare old and new invocation.
If construction time matters, write a separate benchmark for construction time.
Choose parameters from reality
The pipeline benchmark used depths of 10, 20, and 40. Those values were not random. They came from real customer cases and support knowledge about how deep pipelines tend to get after NServiceBus and customers add behaviors.
Could the benchmark use 5, 10, 15, 20, 25, 30, 35, 40, and 50? Sure. It would also take longer and probably teach less.
Keep the matrix small enough that developers still run it while they are thinking.
Use production evidence where you can: telemetry, logs, support cases, known customer configurations, or load-test data. If you have none of that, start with a small set of plausible values and write down the assumption.
A benchmark can survive imperfect assumptions when they are written down. Accidental assumptions are harder to spot and much easier to trust by mistake.
Use short runs to steer and long runs to trust
The improve-and-benchmark part of the performance loop is iterative. Try a change. Run the benchmark. Learn something. Try another change. If each run takes fifteen minutes, that loop becomes painful.
[ShortRunJob] is useful while steering. It gives quicker feedback, which is enough to tell whether an idea is obviously wrong or promising. It is not the final proof.
Once the direction looks good, remove the short-run shortcut and run a longer benchmark. Let BenchmarkDotNet do its job: warmups, iterations, statistical analysis, and allocation diagnostics.
Short runs keep the investigation moving. Longer runs provide the evidence needed before accepting the result.
They also help with coffee.
When the serious benchmark is running, do not start a video call, a build, or three browser-heavy dashboards on the same machine. Get a beverage of your choice. I prefer tasty espresso. If your boss sees you at the coffee machine for the tenth time that day, you have a perfectly reasonable answer: “I am measuring the performance optimization that should reduce our cloud bill.”
Protect benchmark correctness
Benchmark code needs tests around it too. The benchmark itself does not ship to customers, but broken setup still produces confident nonsense.
Watch for side effects. If a collection grows across iterations, the first iteration may measure a different operation than the last one. If a cache warms during the measured method and never resets, the benchmark may measure the cache state more than the code under investigation.
Also prevent dead code elimination. The just-in-time compiler is allowed to remove work that has no observable effect. BenchmarkDotNet helps by consuming return values, and it provides consumer helpers for cases where returning a value is not enough.
- Keep setup outside the measured method unless setup is the thing being measured.
- Avoid state that grows across iterations unless growth is the scenario.
- Make sure the measured work is consumed and cannot be removed.
- Prefer explicit types in benchmark code when it reduces confusion.
- Do not run heavy applications, video calls, or builds during serious benchmark runs.
BenchmarkDotNet protects against many common mistakes. It cannot protect against measuring the wrong thing.
Sometimes edge cases deserve a benchmark
The general rule is to benchmark common hot-path cases. The pipeline investigation had one exception: exception handling.
Normally, exception paths should not dominate a throughput benchmark. But in a messaging system, production incidents can produce thousands of exceptions while messages move to an error queue. Framework overhead still matters in that path because the system may need to fail fast, move messages safely, and recover capacity.
Common cases come first, while production behavior decides whether an edge case also deserves a benchmark.
Exception handling qualified because incidents made its cost operationally important.
The benchmark is not the finish line
After the benchmark shows an improvement, the temptation is to celebrate. Hold that thought.
The benchmark compared a controlled slice. The system still has serializers, transports, handlers, persistence, transactions, logging, and runtime behavior.
A benchmark can prove a local improvement. It still cannot prove the system became faster.
The next step is to put the improvement back into the profiling harness and check the larger picture again.
A benchmark tells us the change worked in isolation. The second profile tells us whether it still matters in context.
The profile found the target and the benchmark compared the change. Now the profiling harness has to show whether the result survives contact with the rest of the system.
Further reading
Common questions
Is copy-pasting production code into a benchmark repository acceptable?
Yes, for a controlled experiment. Do not confuse the copy with production source. Keep the benchmark isolated, document what was removed, and avoid treating the copied code as another product implementation.
How many parameters should a benchmark have?
As few as possible while still representing the important production cases. If adding a parameter does not change a decision, it probably does not belong in the first benchmark.
Should benchmarks run in continuous integration right away?
Not necessarily. Local benchmark experiments are a good way to build performance awareness. Automated regression testing is a later maturity step, and it has its own problems.
Performance loop status
- [x] Understand the loop
- [x] Build profiling harness
- [x] Profile
- [x] Improve / Benchmark
- [ ] Profile again
- [ ] Ship and observe
