BenchmarkDotNet is the standard micro-benchmarking library for .NET, used inside the runtime itself, ASP.NET Core and thousands of open-source libraries to answer one question with statistical rigor: is this code actually faster, and by how much? This guide is for intermediate and senior C# developers who need to compare two implementations, catch a performance regression before it ships, or justify an optimization with numbers instead of intuition. You will learn how to set up a benchmark project, use [Benchmark], [MemoryDiagnoser] and [Params], compare baselines and runtimes, read a results table correctly, avoid the pitfalls that produce misleading numbers, and wire benchmarks into CI.

What Is BenchmarkDotNet and Why Do You Need It?#

A Stopwatch wrapped around a loop feels like a benchmark, but it rarely measures what you think it does. The JIT recompiles hot methods with better code partway through the run (tiered compilation), the garbage collector can pause the process at an unlucky moment, the OS can scale the CPU clock up or down, and an optimizing compiler can notice that a computed value is never used and delete the whole loop. A single timed run also has no error bars, so a 5% difference between two implementations could be real or could be noise.

BenchmarkDotNet, currently at version 0.15.8 on NuGet and targeting .NET Standard 2.0 and .NET 6.0 and later, exists to remove that guesswork. It generates an isolated console project for each benchmark job, builds it in Release, runs a pilot phase to work out how many iterations are needed, warms up the JIT, runs the measured workload multiple times, and reports a mean with a confidence interval instead of a single number. It can also target multiple runtimes, such as .NET Framework, .NET 8 and .NET 10, from a single benchmark project, which is invaluable when you need to know whether an optimization still helps after an upgrade.

How BenchmarkDotNet Works#

You reference the BenchmarkDotNet package, write a class whose public methods are marked [Benchmark], and call BenchmarkRunner.Run<T>(). From there, BenchmarkDotNet does not simply call your methods in a loop inside your process. It generates a separate, minimal project per job, so your host process's JIT state, loaded assemblies and GC history never leak into the measurement. That generated project is compiled and launched as a child process, which is why a full benchmark run takes noticeably longer than the code itself would suggest.

Each benchmark goes through several phases: a pilot stage estimates how many operations fit in one iteration; a warmup stage runs the workload repeatedly until throughput stabilizes, similar to what happens in a real, long-running process; and the actual workload stage runs the timed iterations that are used for the statistics. An engine subtracts the overhead of the measurement loop itself, so the reported time reflects your code, not the harness. Optional diagnosers, such as [MemoryDiagnoser], run additional passes to collect data like allocations without disturbing the timing run.

Getting Started: Your First Benchmark#

Create a dedicated console project for benchmarks; never benchmark from inside a unit test project or the application itself.

Bash
dotnet new console -n StringBenchmarks
cd StringBenchmarks
dotnet add package BenchmarkDotNet
C#
using System.Text;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<StringConcatBenchmarks>();

[MemoryDiagnoser]
public class StringConcatBenchmarks
{
    private readonly string[] _parts =
        Enumerable.Range(0, 200).Select(i => $"part-{i}").ToArray();

    [Benchmark(Baseline = true)]
    public string Concatenation()
    {
        var result = string.Empty;
        foreach (var part in _parts)
        {
            result += part;
        }
        return result;
    }

    [Benchmark]
    public string StringBuilderJoin()
    {
        var sb = new StringBuilder();
        foreach (var part in _parts)
        {
            sb.Append(part);
        }
        return sb.ToString();
    }
}

Run it with dotnet run -c Release. BenchmarkDotNet refuses to run, or prints a loud warning, for Debug builds and when a debugger is attached, because both disable JIT optimizations and would make the numbers meaningless. A full run of this simple example still takes on the order of a minute, because of the pilot, warmup and multiple iterations described above.

Core Attributes: Benchmark, MemoryDiagnoser and Setup Methods#

[Benchmark] is the only attribute a method needs to be measured; everything else refines what gets measured or how the benchmark class is prepared.

  • [MemoryDiagnoser] adds Gen0, Gen1 and Gen2 columns, showing collections per 1,000 operations, plus an Allocated column with bytes allocated per operation. Allocation counts are exact, not sampled, so they are often more reliable evidence than the timing numbers.
  • [GlobalSetup] and [GlobalCleanup] run once before and after all iterations of a benchmark method, ideal for loading fixed test data that should not be part of the measurement.
  • [IterationSetup] and [IterationCleanup] run before and after every iteration; use them sparingly because they add overhead and can themselves skew results if they allocate or trigger a GC.
  • [Benchmark(Baseline = true)] marks one method in a class as the baseline. Every other benchmark's results are then shown relative to it in a Ratio column.
C#
[MemoryDiagnoser]
public class LookupBenchmarks
{
    private Dictionary<string, int>? _dictionary;
    private string[]? _keys;

    [Params(100, 10_000)]
    public int Count { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        _keys = Enumerable.Range(0, Count).Select(i => $"key-{i}").ToArray();
        _dictionary = _keys.Select((k, i) => (k, i)).ToDictionary(x => x.k, x => x.i);
    }

    [Benchmark(Baseline = true)]
    public int LinearScan() => _keys!.Select((k, i) => (k, i)).First(x => x.k == _keys[^1]).i;

    [Benchmark]
    public int DictionaryLookup() => _dictionary![_keys![^1]];
}

Parameterizing Benchmarks with Params#

[Params] on a public field or property turns a single benchmark method into a matrix of cases: one run per value, and one run per combination when several [Params] members are declared, because BenchmarkDotNet builds the Cartesian product of all parameter sets. [ParamsSource] pulls the value list from a member instead of hard-coding it in the attribute, which is useful when the values come from a shared constant or need to be built programmatically. In the LookupBenchmarks example above, [Params(100, 10_000)] on Count produces two rows per benchmark method, so you can see immediately whether an approach that wins at 100 items still wins at 10,000. This is usually more informative than a single, arbitrarily sized input, because many algorithms have different winners at different scales.

Comparing Runtimes and Jobs#

Stacking multiple [SimpleJob] attributes on a class runs every benchmark once per job, which is the standard way to compare .NET versions, garbage collector modes or JIT settings side by side in one report:

C#
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;

[MemoryDiagnoser]
[SimpleJob(RuntimeMoniker.Net80)]
[SimpleJob(RuntimeMoniker.Net10_0)]
public class JsonSerializationBenchmarks
{
    // From .NET 10 onward the monikers use an underscore (Net10_0) to avoid ambiguity.
    // Newer runtimes are added to RuntimeMoniker as BenchmarkDotNet releases ship.

    [Benchmark]
    public string Serialize() => System.Text.Json.JsonSerializer.Serialize(Payload.Sample);
}

Each [SimpleJob] needs its own <TargetFramework> entry (or a multi-target <TargetFrameworks>) in the project file so the generated child projects can build and run under each runtime. You can also compare Server GC against Workstation GC, or Concurrent GC on and off, by deriving a class from ManualConfig and adding two jobs built with Job.Default.WithGcServer(true) and Job.Default.WithGcServer(false). This is the fastest way to answer "does Server GC actually help this workload" with evidence instead of a rule of thumb.

Reading BenchmarkDotNet Results#

A typical summary table has columns like these:

Text
| Method             | Count | Mean       | Error    | StdDev   | Ratio | Gen0    | Allocated |
|------------------- |------ |-----------:|---------:|---------:|------:|--------:|----------:|
| LinearScan         | 100   |   1.812 us | 0.021 us | 0.019 us |  1.00 |  0.0668 |     424 B |
| DictionaryLookup   | 100   |   9.912 ns | 0.084 ns | 0.079 ns |  0.01 |       - |         - |
  • Mean is the arithmetic mean of every measured operation, in whatever unit keeps the number readable (ns, us or ms).
  • Error is half the width of a 99.9% confidence interval around the mean by default; treat two means as indistinguishable if their intervals overlap.
  • StdDev is the standard deviation across iterations; a StdDev that is large relative to the Mean usually means the environment was noisy or the workload has multiple execution paths.
  • Ratio (and RatioSD) appear once a Baseline = true benchmark exists, comparing every other method's mean, and its variability, against the baseline's mean.
  • Gen0/Gen1/Gen2 and Allocated, from [MemoryDiagnoser], show garbage collector pressure per 1,000 operations and bytes allocated per operation.

The Disassembly Diagnoser#

[DisassemblyDiagnoser(maxDepth: 3)] asks BenchmarkDotNet to capture the actual JIT-compiled assembly of the benchmarked method and the methods it calls, up to the requested depth. It runs on Windows, Linux and macOS, though how deep it can follow inlined calls varies by platform and JIT tier. This is the authoritative way to answer questions a timing number cannot: did the JIT actually inline this call, did it vectorize the loop with SIMD instructions, or did a seemingly cheap helper method turn into a much larger block of code than expected.

C#
[MemoryDiagnoser]
[DisassemblyDiagnoser(maxDepth: 3, printSource: true)]
public class SumBenchmarks
{
    private readonly int[] _values = Enumerable.Range(0, 10_000).ToArray();

    [Benchmark]
    public long Sum()
    {
        long total = 0;
        foreach (var v in _values)
        {
            total += v;
        }
        return total;
    }
}

The output is large, so reach for it only after the summary table already points at a specific method; running it on every benchmark in a suite would be slow and mostly noise.

Running BenchmarkDotNet in CI for Regression Detection#

Benchmarks are too slow, and too sensitive to the machine they run on, to execute on every pull request the way unit tests do. A practical pattern is to run the suite on a schedule, on a consistent, dedicated runner, export the results as JSON, and keep them as a build artifact or push them to a small dashboard:

YAML
name: benchmarks
on:
  schedule:
    - cron: '0 3 * * 1'
  workflow_dispatch: {}

jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '10.0.x'
      - name: Run benchmarks
        run: >
          dotnet run -c Release --project bench/Bench.csproj --
          --filter '*' --exporters json
      - uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: bench/BenchmarkDotNet.Artifacts/results/*.json

--filter accepts a glob against namespace, type and method (for example *JsonSerialization*), so a faster smoke suite can run more often than the full one. To turn this into an actual regression gate, compare the new Mean and Allocated values for each benchmark against the previous stored run and fail the job if either regresses beyond a threshold you choose, such as 10%; BenchmarkDotNet reports the raw numbers, but the comparison and threshold logic is something you own, typically a small script that reads the exported JSON. The .NET Diagnostics Toolkit guide covers the complementary, lower-overhead tools you can leave running in production to catch regressions BenchmarkDotNet never sees. CI pipeline design in general is covered in CI/CD for .NET with GitHub Actions and Azure DevOps.

Best Practices#

  • Always build Release and run outside a debugger. Debug builds and attached debuggers disable JIT optimizations that production code relies on.
  • Return or otherwise consume the computed value from every benchmark method, so the JIT and any future ahead-of-time compiler cannot reason that the result is unused and remove work.
  • Add [MemoryDiagnoser] by default. Allocation counts are cheap to collect and often explain a performance difference better than the timing alone.
  • Use [Params] to cover realistic input sizes, not just one convenient size, especially for collection and algorithm comparisons.
  • Mark a Baseline = true benchmark whenever you are comparing alternatives, so the report includes a Ratio instead of forcing readers to do the division themselves.
  • Run on quiet, dedicated hardware for numbers you intend to publish or gate a build on; a laptop on battery power with background processes running will not reproduce.
  • Keep benchmark projects separate from application and test code, with their own .csproj, so a dotnet test run never accidentally executes a multi-minute benchmark suite.

Common Pitfalls#

  • Benchmarking a Debug build or with a debugger attached, which silently produces numbers that do not reflect production performance.
  • Letting dead-code elimination win by writing a void benchmark that computes a value and never returns, stores or otherwise uses it.
  • Trusting a single run. Environment noise, especially on shared CI runners and laptops with dynamic CPU frequency scaling, can shift results by double digits between runs; look at the confidence interval, and re-run when it looks too wide.
  • Skipping warmup by using [SimpleJob(RunStrategy.ColdStart)] for everyday comparisons. Cold start jobs exist for measuring startup scenarios specifically, not as a shortcut to faster benchmark runs.
  • Ignoring [MemoryDiagnoser] output and optimizing only for the Mean column, which misses allocation-heavy code that happens to run fast on the benchmark machine but pressures the GC in production under load.
  • Benchmarking a method in isolation that behaves completely differently in context, for example a cache lookup benchmarked with a single key repeated forever, which will look unrealistically fast.

BenchmarkDotNet vs Stopwatch vs a Profiler#

ToolAnswersStatistical rigorTypical use
Stopwatch loopRough order of magnitudeNone; single sample, no warmup controlQuick sanity check while coding
BenchmarkDotNetIs A faster than B, and by how much, with allocationsHigh: warmup, multiple iterations, confidence intervalsDeciding between implementations, tracking regressions
Sampling profiler (dotnet-trace, PerfView)Where does an entire application spend its timeStatistical sampling across real workloadsFinding which method to benchmark next

Profiling and benchmarking answer different questions and work best together: profile a running application to find the hot method, then benchmark that method in isolation to compare alternatives. The Diagnostics Toolkit guide covers dotnet-trace and the rest of the profiling toolchain, and High-Performance .NET shows the allocation-reduction and collection-choice techniques that benchmarks like these are typically used to validate.

Frequently Asked Questions#

Why not just use Stopwatch in a loop instead of BenchmarkDotNet?#

A hand-rolled loop has no warmup phase, so early iterations run against not-yet-optimized JIT code; no statistical treatment, so you cannot tell noise from a real difference; and no protection against dead-code elimination, so the compiler can remove work it decides is unused. BenchmarkDotNet automates all three and adds memory diagnostics for free.

How many iterations does BenchmarkDotNet actually run?#

It varies per benchmark. A pilot stage estimates how many invocations fit into one iteration so each iteration takes a reasonable amount of wall-clock time, then the default engine runs enough iterations, generally at least fifteen, to get a stable estimate, extending further if the results are still noisy. You rarely need to set this manually, but [SimpleJob(iterationCount: 20)] overrides it when you do.

Can I benchmark async methods?#

Yes. A [Benchmark] method can return Task or Task<T>, and BenchmarkDotNet awaits it correctly as part of the measured operation. Keep in mind that this measures the whole asynchronous flow, including any real I/O it performs, so async benchmarks over network or disk calls are typically far noisier than CPU-bound ones and need more iterations to reach a stable confidence interval.

Why did my benchmark report an unrealistically small time, like a few nanoseconds?#

This is almost always dead-code elimination: the JIT determined that the computed value was never used and removed the work. Make sure every benchmark method returns its result, or assigns it to a field marked volatile or otherwise observable, so the computation cannot be proven unnecessary.

Should benchmarks run on every pull request like unit tests?#

Usually not. Benchmarks take much longer than unit tests and are sensitive to whatever else is running on the machine, so shared, variably loaded CI runners produce noisy, hard-to-trust numbers on every commit. Run a full suite on a schedule or on demand, on consistent hardware, and keep pull requests fast with unit and integration tests instead.

Summary#

  • BenchmarkDotNet removes the warmup, statistics and dead-code-elimination problems that make hand-rolled timing loops unreliable.
  • [Benchmark], [MemoryDiagnoser] and [Params] cover most day-to-day comparisons; add Baseline = true to get a Ratio column for free.
  • Stack [SimpleJob] attributes to compare .NET runtimes or GC modes in one report.
  • Read Mean alongside Error, StdDev and Allocated; allocation counts are usually the most reproducible signal.
  • Reach for [DisassemblyDiagnoser] only once you already know which method to investigate.
  • Run benchmarks on a schedule against consistent hardware, not on every pull request, and compare exported JSON results over time to catch regressions.

Further Reading#