Benchmarking and BenchmarkDotNet Interview Questions

Q1 What makes a microbenchmark trustworthy, and what does BenchmarkDotNet do that a hand-rolled Stopwatch loop doesn't?#

Short answer: A trustworthy microbenchmark isolates the measurement from JIT warm-up, tiered compilation, dead-code elimination and garbage collection noise, runs enough iterations to produce a stable statistical estimate, and reports an uncertainty range rather than a single number; BenchmarkDotNet automates all of that, and a hand-rolled loop typically gets every one of those wrong by default.

// Unreliable: no warm-up, a single measurement, and the result is never
// used, so the JIT is free to optimize the entire loop body away.
var sw = Stopwatch.StartNew();
for (int i = 0; i < 1_000_000; i++)
{
    ParseOrderLine(Line);
}
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);

// Reliable: BenchmarkDotNet handles warm-up, iteration count, outlier
// removal and statistics, and consumes the return value so it can't vanish.
[MemoryDiagnoser]
public class ParseBenchmarks
{
    [Benchmark]
    public OrderLine Parse() => ParseOrderLine(Line);
}

The Stopwatch version fails in ways that are easy to miss. The method is not fully optimized until the runtime has called it enough times to promote it past tiered compilation's early, unoptimized tier, so the first iterations measure the wrong code. Because the loop's result is discarded, the JIT is free to prove the whole computation has no observable effect and remove it, at which point the benchmark measures nothing at all; BenchmarkDotNet requires [Benchmark] methods to return a value precisely to close that loophole. A single elapsed-time reading also has no error bar, so there is no way to tell a real difference between two versions from ordinary run-to-run noise. BenchmarkDotNet addresses each of these: a warm-up phase before measurement, a separate process per benchmark to avoid cross-contamination, many measured iterations, statistical outlier handling, and a reported mean with an explicit error range.

What interviewers look for: naming the specific failure modes, dead-code elimination and insufficient warm-up especially, rather than a general "Stopwatch is inaccurate." Candidates who explain why BenchmarkDotNet's design choices exist are showing real understanding, not tool familiarity.

Common mistakes: assuming a for loop with enough iterations is "close enough"; forgetting that a discarded result can let the compiler eliminate the code being measured.

Q2 Walk through designing a BenchmarkDotNet class to compare two implementations, including their memory behavior.#

Short answer: Give each implementation its own [Benchmark] method, mark one Baseline = true so the report shows a ratio, add [MemoryDiagnoser] for allocation data, and use [Params] with [GlobalSetup] when you need to see how the comparison changes with input size.

[MemoryDiagnoser]
public class MembershipBenchmarks
{
    private List<int> _list = null!;
    private HashSet<int> _set = null!;

    [Params(10, 1_000, 100_000)]
    public int N;

    [GlobalSetup]
    public void Setup()
    {
        _list = Enumerable.Range(0, N).ToList();
        _set = [.. _list];
    }

    [Benchmark(Baseline = true)]
    public bool ListContains() => _list.Contains(N - 1);

    [Benchmark]
    public bool SetContains() => _set.Contains(N - 1);
}

[GlobalSetup] runs once per parameter combination, not once per measured invocation, so the cost of building _list and _set is excluded from the timing entirely; forgetting this distinction is a common way candidates accidentally benchmark their setup code instead of the operation they meant to test. [Params(10, 1_000, 100_000)] runs the whole benchmark class once for each value, which is how you show that a List<T>.Contains linear scan degrades with size while a HashSet<T>.Contains lookup does not, in one report instead of three separate ones. Marking ListContains as the baseline adds a Ratio column, so the report states directly how many times faster or slower SetContains is at each size, rather than making the reader do that division by hand. [MemoryDiagnoser] adds an Allocated column, which in this particular comparison should show that neither method allocates, correctly showing that the difference here is purely algorithmic, not allocation-driven.

What interviewers look for: correct use of [GlobalSetup] to exclude fixture cost, a baseline for readable ratios, and [Params] for size-dependent behavior, which together show you have actually built more than a single fixed-input benchmark.

Follow-up questions:

• What is the difference between [GlobalSetup] and [IterationSetup], and when would you need the more expensive one?
• How would you benchmark a method that depends on I/O without the I/O latency swamping the result?

Q3 How do you interpret BenchmarkDotNet's statistical output, and what does "noise" mean in this context?#

Short answer: Mean and Median describe the central estimate, Error and StdDev describe how much to trust it, and Allocated comes from the memory diagnoser; "noise" refers to everything outside your code that perturbs a measurement, other processes, CPU frequency scaling, background GC and shared, busy hardware, all of which BenchmarkDotNet's statistics are designed to make visible rather than hide.

A StdDev that is a large fraction of the Mean is the signal to treat a result cautiously: it usually means the host machine had other activity during the run, the CPU throttled or boosted unevenly, or the benchmarked code itself has data-dependent timing, such as a branch or a GC pause that only sometimes triggers. BenchmarkDotNet's multiple iterations and outlier detection reduce the effect of a single bad measurement, but they cannot remove noise that is present throughout the run, which is why a quiet, dedicated machine produces far more trustworthy numbers than a busy laptop or a shared, bursty CI runner.

What interviewers look for: knowing what each column actually measures, and specifically that a wide Error or StdDev is a reason to distrust a result rather than something to ignore in favor of the headline Mean.

Q4 What are the most common mistakes that invalidate a microbenchmark?#

Short answer: Benchmarking a Debug build, letting the compiler eliminate dead code by discarding a result, comparing methods that don't do equivalent work, running on a noisy or shared machine, and skipping [MemoryDiagnoser] so an "optimization" that traded allocations for CPU time goes unnoticed.

BenchmarkDotNet itself guards against the first mistake: it detects an unoptimized, Debug-mode build and refuses to run by default rather than silently producing misleading numbers, which is a strong hint that Release mode matters enough to enforce. Dead-code elimination is the second recurring trap; a [Benchmark] method must return a value the harness actually consumes, or the JIT can prove the computation has no effect and remove it, leaving you with a very fast benchmark of nothing. A subtler mistake is an asymmetric comparison: two methods that look parallel in a report but do different amounts of work, one returning a value the other only computes as a side effect, or one that materializes a collection while the other does not, will produce a "result" that has no real meaning. Running comparisons on a laptop with other applications open, on battery power with CPU throttling active, or on a shared, bursty CI runner all inflate StdDev enough that small, real differences disappear into noise. Finally, treating raw timing as the whole picture misses regressions that trade time for memory; without [MemoryDiagnoser], a change that is faster but allocates far more can look like a pure win when it is really shifting cost onto the garbage collector.

What interviewers look for: a list that goes beyond "use Release mode," touching asymmetric comparisons and environment noise, which are the mistakes that survive past a first code review.

Common mistakes: treating a benchmark result from a shared CI runner as authoritative; adding [MemoryDiagnoser] only after a regression is already suspected instead of by default.

Q5 How do you use BenchmarkDotNet to compare behavior across .NET runtimes or configurations?#

Short answer: Stack multiple [SimpleJob(RuntimeMoniker...)] attributes on the same class so the identical benchmark methods run under each targeted runtime, and BenchmarkDotNet reports every job as its own row in the same table for direct comparison.

[MemoryDiagnoser]
[SimpleJob(RuntimeMoniker.Net80)]
[SimpleJob(RuntimeMoniker.Net10_0)]
public class RuntimeComparisonBenchmarks
{
    [Benchmark]
    public int[] AllocateArray() => new int[1024];
}

Each [SimpleJob] attribute tells BenchmarkDotNet to build and run the benchmarks against a specific target, using the SDKs installed on the machine, so a class like this produces one table with a row per runtime per benchmark method, making it straightforward to see whether a .NET 10 upgrade actually changed the numbers for code you care about. The same mechanism extends to comparing GC configurations, such as Server versus Workstation GC or concurrent GC on and off, by configuring a custom Job with the corresponding characteristics instead of, or alongside, a runtime moniker. This is the right tool for "did upgrading help" questions raised by a runtime upgrade, and it produces a defensible, reproducible answer instead of an anecdote from one developer's machine.

What interviewers look for: knowing the attribute-based mechanism for multi-runtime comparisons specifically, not just "I'd run it twice manually on two SDKs and compare by eye."

Follow-up questions:

• What has to be true about the machine running this benchmark for a multi-runtime comparison to be valid?
• How would you extend this class to also compare Native AOT against the JIT-compiled runtime?

Q6 How would you add benchmarks to CI without making builds flaky or slow?#

Short answer: Run a fast, reduced-iteration smoke pass on every pull request to catch outright regressions and compile errors in the benchmarks themselves, and reserve the full, statistically rigorous suite for a schedule or a pre-release gate on dedicated, consistently sized hardware, comparing against a rolling baseline with tolerance rather than a hard pass or fail on a single run.

on:
  pull_request:
  schedule:
    - cron: "0 3 * * *"   # nightly, on dedicated runners with stable hardware

jobs:
  benchmarks:
    runs-on: [self-hosted, perf-runner]
    steps:
      - run: dotnet run -c Release --project bench -- --filter "*" --job short
        if: github.event_name == 'pull_request'   # fast smoke pass on every PR
      - run: dotnet run -c Release --project bench -- --filter "*"
        if: github.event_name == 'schedule'        # full statistical run, nightly

Running BenchmarkDotNet's default, thorough statistical methodology on every pull request is usually too slow to be practical, and running it on shared, bursty CI infrastructure makes the numbers themselves unreliable, since noisy-neighbor variance can dwarf a real regression. The pattern that works in practice is two tiers: a short job configuration on every PR that mainly proves the benchmarks still compile and run, catching gross regressions without claiming statistical rigor, and a full run on a schedule or before a release, on hardware reserved for exactly this purpose so run-to-run comparisons are meaningful. Comparing against a single previous run invites false alarms from ordinary noise, so track results over time and alert on a sustained deviation from a rolling baseline rather than any single run crossing a fixed line.

What interviewers look for: the two-tier design and, specifically, the reasoning that shared CI hardware undermines absolute numbers, which is why dedicated runners and trend-based comparison matter more than a single threshold.

Common mistakes: running the full benchmark suite on every commit and accepting the resulting slow pipeline; comparing a new run only against the immediately preceding one instead of a rolling baseline.

Q7 What's the difference between a microbenchmark and a macro-benchmark or load test, and when do you need both?#

Short answer: A microbenchmark measures a single method or algorithm in isolation, in-process and single-threaded, answering "which implementation is faster"; a load test exercises the whole running system over the network under realistic concurrency, answering "does this system meet its latency and throughput targets in production-like conditions," and the two catch entirely different classes of problems.

A microbenchmark cannot see connection pool exhaustion, thread-pool starvation under genuine concurrent load, lock contention between real concurrent requests, or a downstream dependency that degrades under sustained traffic, because none of those exist inside a single-threaded, in-process loop. A load test cannot tell you whether one parsing algorithm beats another by a few hundred nanoseconds, because that difference is invisible against network latency and disables the kind of controlled, repeatable comparison a microbenchmark provides. You need both because they answer different questions at different points in the development cycle: microbenchmarks give fast, local feedback while writing or reviewing a specific piece of hot-path code, and load tests validate, before a release, that a system built from individually fast pieces still meets its targets once real concurrency, shared resources and network effects are in play. A codebase full of benchmarked-fast methods can still fall over under load if they all contend for the same lock or database connection pool, which is precisely the failure a microbenchmark is structurally unable to reveal.

What interviewers look for: a clear statement that these tools answer different questions and catch different bug classes, not a ranking of one as more important than the other.

Q8 How do k6, NBomber and Bombardier differ, and how would you choose between them for a load test?#

Short answer: k6 is a scriptable, JavaScript-based open-source load-testing tool with a strong ecosystem around dashboards and thresholds; NBomber is a .NET-native framework where scenarios are written in C# or F# and can drive any protocol, not just HTTP; Bombardier is a single dependency-free binary for a fast, no-scripting HTTP throughput check.

k6 fits teams that want load tests as versioned scripts independent of the application's implementation language, with rich built-in support for checks, thresholds and integration with existing observability stacks. NBomber fits teams that want the test written in the same language as the system under test, able to reuse the application's own DTOs and serialization code directly, and that need to load-test something beyond plain HTTP, since an NBomber step is just an asynchronous function, which means it can exercise a message queue, a gRPC endpoint or a custom protocol as naturally as a REST call. Bombardier fits the narrowest but very common need: a quick, disposable answer to "how many requests per second can this one endpoint sustain," with no script to write at all, at the cost of far less flexibility for multi-step scenarios with think-time or data-driven workloads. The practical decision usually comes down to whether you need multi-step, stateful scenarios (k6 or NBomber), whether the test needs to live alongside .NET application code and cover non-HTTP protocols (NBomber), or whether you just need a fast sanity check on raw throughput for a single endpoint (Bombardier).

What interviewers look for: matching the tool to the actual need, scripting language, protocol coverage, and scenario complexity, rather than naming a single favorite tool for every situation.

Q9 Walk through writing a simple NBomber load-testing scenario for an HTTP endpoint.#

Short answer: Define a scenario as an async function that makes the request and reports Response.Ok() or Response.Fail(), attach a load simulation describing the injection rate over time, and register it with NBomberRunner to execute and produce a report.

using NBomber.CSharp;

using var httpClient = new HttpClient { BaseAddress = new Uri("https://localhost:5001") };

var scenario = Scenario.Create("get_order_summary", async context =>
{
    var response = await httpClient.GetAsync("/api/orders/42/summary");
    return response.IsSuccessStatusCode ? Response.Ok() : Response.Fail();
})
.WithLoadSimulations(
    Simulation.Inject(rate: 50, interval: TimeSpan.FromSeconds(1), during: TimeSpan.FromMinutes(2))
);

NBomberRunner
    .RegisterScenarios(scenario)
    .Run();

Scenario.Create takes a name and an async step; the return value, Response.Ok() or Response.Fail(), is what NBomber uses to compute success rate and latency statistics separately from failures, so a scenario that silently swallows an error instead of returning Response.Fail() will under-report problems. Simulation.Inject(rate, interval, during) describes a constant-rate load, fifty requests injected per second for two minutes in this example; NBomber also supports ramping simulations for gradually increasing load, which is usually the more realistic shape for finding the point where a system's latency starts to degrade rather than jumping straight to peak load. NBomberRunner.RegisterScenarios(...).Run() executes the configured scenarios and produces a report with throughput, error rate and latency percentiles, which is the artifact you compare against the system's stated performance budget.

What interviewers look for: the correct shape of the API, Scenario.Create, a load simulation, and NBomberRunner, and understanding that Response.Fail() is what makes failure tracking meaningful rather than an optional detail.

Follow-up questions:

• How would you extend this scenario to log in once and reuse an authentication token across requests, rather than authenticating on every call?
• What would a ramping load simulation tell you that a constant-rate one does not?

Q10 How do you interpret p50, p95 and p99 latency from a load test, and why is p50 alone often misleading?#

Short answer: p50 is the typical request, p95 means one request in twenty is at least that slow, and p99 means one in a hundred is; p50 alone hides the tail, and the tail is usually both what breaches your SLO and what a meaningful fraction of real users actually experience at any real traffic volume.

The math that makes this concrete: at a million requests a day, a p99 of two seconds against a p50 of fifty milliseconds still means roughly ten thousand requests a day take at least two seconds, a volume of bad experiences a p50-only dashboard hides completely. Averages are worse than even p50 for this purpose, because a mean is pulled by outliers in a way percentiles deliberately resist, which is why "average response time" is rarely the number an SLO is written against. A less obvious but interview-relevant trap is that percentiles do not average: you cannot take the p99 reported by each of several service instances and average them to get a fleet-wide p99, because that arithmetic silently discards exactly the tail behavior percentiles exist to capture; a correct fleet-wide percentile requires aggregating the underlying latency samples, or merging histograms, centrally before computing the percentile. Finally, p99 needs more total samples than p50 to be statistically meaningful; at low request volume, a reported p99 may just be the second-slowest request you happened to sample rather than a stable estimate, so a load test needs enough sustained volume before its tail percentiles are worth trusting.

What interviewers look for: the concrete "percentiles don't average" trap, which is a strong signal of real production experience with fleet-wide metrics, plus fluency connecting percentile choice back to what an SLO should actually be written against.

Common mistakes: reporting only the mean or p50 as "the" latency number; averaging per-instance percentiles into a fleet-wide figure.