Performance Profiling Interview Questions for .NET Developers

Q1 What's the difference between sampling and instrumentation profiling, and when would you choose each?#

Short answer: A sampling profiler periodically captures every thread's call stack at a fixed interval and infers where time goes statistically, at low, near-constant overhead; an instrumentation profiler injects timing code at method boundaries to get exact call counts and durations, at overhead that can be an order of magnitude higher and can itself distort the timing it is trying to measure.

Sampling is the right default, and it is what dotnet-trace's built-in CPU profile and PerfView's default collection mode both use: at roughly a hundred samples per second, a thread that spends 40% of wall-clock time in one method will show up in roughly 40% of samples, which is enough to find hot paths without materially slowing the process down. That makes it safe to run against a live service, including in production, for a bounded window. Instrumentation earns its overhead when you need something sampling cannot give you: exact call counts, coverage of methods so short-lived or rare that sampling would under-represent or miss them entirely, or precise timing of a specific, already-identified method rather than a broad search for hot spots. The trade-off is real: instrumenting every method call can slow a process down enough to change its own behavior, especially anything timing-sensitive like lock acquisition, which is why instrumentation profiling is typically a targeted, offline exercise rather than something you run against live traffic.

What interviewers look for: a clear statement of the overhead trade-off and a concrete rule for when each is appropriate, not just definitions. Mentioning that instrumentation can distort the very behavior it measures is a strong signal of hands-on experience.

Common mistakes: treating sampling as strictly less accurate rather than as the right tool for a different question; suggesting instrumentation profiling as a default for production diagnosis.

Q2 What is EventPipe, and how does it relate to dotnet-trace, ETW and your own custom events?#

Short answer: EventPipe is the cross-platform, always-on tracing subsystem built into the .NET runtime since .NET Core; it is what lets dotnet-trace, dotnet-counters and dotnet-monitor pull CLR and application events out of a running process on Linux and macOS as well as Windows, without depending on Windows-only ETW.

Before EventPipe, deep CLR tracing meant ETW on Windows and a separate mechanism on Linux, which fragmented tooling across platforms and made container-based diagnostics awkward. EventPipe unifies this: every .NET process exposes a diagnostics IPC channel, a named pipe on Windows or a Unix domain socket on Linux and macOS, that a client connects to and requests specific event providers and verbosity levels from. Microsoft-Windows-DotNETRuntime, the CLR's own provider, is what supplies GC, JIT, exception, contention and thread-pool events; dotnet-trace collect is essentially a client that connects over this channel, asks for a chosen set of providers, and writes what comes back to a .nettrace file. Because it works over IPC against an already-running process, no restart, no special launch flags and no Windows dependency are required, which is exactly what makes it practical to trace a container or a Kubernetes pod on demand. Application code participates in the same mechanism by defining its own EventSource-derived type: those custom events flow into the identical trace stream, interleaved with the runtime's own, which is how you correlate a business-level event, such as "order processing started," with the GC and thread-pool activity happening at the same moment.

What interviewers look for: knowing EventPipe as the actual mechanism, not just "dotnet-trace uses ETW," and understanding why cross-platform, in-process tracing over IPC is what makes container profiling possible at all.

Follow-up questions:

• How would you add a custom event to your own trace using EventSource?
• What's the difference between the diagnostics IPC channel and the older, Windows-only ETW pipeline?

Q3 Walk through collecting a CPU trace with dotnet-trace and reading the result as a flame graph.#

Short answer: Collect with the dotnet-sampled-thread-time profile, export to Speedscope format, and read the flame graph by width, not position: a wide frame means a large share of samples had that method on the stack, and the distinction between a frame's own width and its children's width tells you whether the time is spent in that method or in what it calls.

dotnet tool install --global dotnet-trace

dotnet-trace collect -p 4807 --profile dotnet-sampled-thread-time \
    --format speedscope -o cpu.speedscope.json --duration 00:00:00:30

Open the resulting file at speedscope.app, or open the raw .nettrace in Visual Studio or PerfView, both of which render the same data as an interactive flame graph. Unlike a typical time-series chart, the x-axis in a flame graph is not wall-clock time; frames are aggregated by call stack and sorted by size, so a wide frame anywhere in the graph means that call stack accounted for a large share of all samples, regardless of when during the trace it ran. The reading skill that matters most is self-time versus total-time: a frame that is wide but whose children fill almost all of that width is mostly waiting on what it calls, while a frame that is wide and has a thin sliver of its own uncovered width is where the CPU actually spent its time. Interviewers expect you to go straight to the widest leaf frames, not the widest frames overall, because the widest frame near the root is usually just Main or a request pipeline entry point.

What interviewers look for: the exact CLI invocation from memory, and the self-time-versus-total-time reading skill, which is the single most common thing candidates get wrong when asked to interpret a sample flame graph on the spot.

Common mistakes: treating the x-axis as chronological time; stopping at the first wide frame instead of drilling down to the widest leaf.

Q4 When would you reach for PerfView instead of dotnet-trace or the Visual Studio profiler?#

Short answer: dotnet-trace and the Visual Studio profiler cover the everyday collect-and-look workflow with a friendlier interface; PerfView is the tool for the deepest, most complete view of everything EventPipe and ETW can capture at once, and for analysis views, such as grouping and folding stacks by module, that the lighter tools do not offer.

PerfView is open source, free, and is the same tool the .NET runtime team itself uses to diagnose performance issues in the runtime, which shows in how much of the CLR's internal behavior its views expose: full GC internals generation by generation, JIT compilation events, first-chance exceptions, and heap snapshots that can be diffed directly inside the tool. Its grouping and folding features let you collapse "everything under this module" or isolate "just my code," which turns an intimidating thousand-frame stack into something readable, a capability the simpler tools do not match. On Windows, PerfView can also collect through native ETW directly, which picks up OS-level events, such as disk I/O and hard page faults, that a pure EventPipe trace does not include, which matters when the real question is whether a slow request is CPU-bound or I/O-bound at the operating system level. The trade-off is a steeper learning curve: PerfView's UI rewards investment but is far less approachable on day one than dotnet-trace collect followed by a Speedscope view.

What interviewers look for: recognition that PerfView's value is breadth and depth of what it can capture and analyze, not that it is simply "the advanced version" of the other tools.

Q5 How do you diagnose lock contention from counters or a trace?#

Short answer: Watch the Monitor Lock Contention Count counter for a rising rate during the slow window, then capture a trace and look at which call stacks are blocked waiting to enter a monitor; the fix is almost always to shrink the critical section or reduce how many unrelated operations share the same lock object.

// Coarse lock: every tenant serializes on the same object, even though their
// updates are completely independent of each other.
private readonly object _gate = new();
private readonly Dictionary<string, TenantState> _state = new();

public void Update(string tenantId, Action<TenantState> mutate)
{
    lock (_gate)
    {
        mutate(_state[tenantId]);
    }
}

// Sharded: unrelated tenants no longer contend for the same lock at all.
// TenantState itself must still handle any compound update atomically.
private readonly ConcurrentDictionary<string, TenantState> _state = new();

public void Update(string tenantId, Action<TenantState> mutate) =>
    mutate(_state.GetOrAdd(tenantId, static _ => new TenantState()));

A rising Monitor Lock Contention Count confirms threads are queuing for a lock, but it does not say which one, so the next step is a trace covering the slow window; the CLR runtime provider's contention events, visible in PerfView's blocked-time views or in the Visual Studio concurrency profiler, show exactly which call stacks were waiting and for how long. Once you know which lock is contended, the fix is rarely "add more hardware," it is reducing what the lock protects: narrowing the critical section to the minimum work that actually needs to be atomic, splitting one coarse lock into several finer-grained ones keyed by the data they protect, or replacing a lock entirely with a lock-free structure such as ConcurrentDictionary<TKey, TValue> when the access pattern fits.

What interviewers look for: the counter-then-trace sequence, and a concrete remediation, not just "reduce lock contention" as an abstract goal.

Common mistakes: reaching for a lock-free rewrite before confirming contention is actually the bottleneck; sharding a lock without checking whether the protected state itself needs coordination across shards.

Q6 How do you detect and diagnose thread-pool starvation?#

Short answer: Watch ThreadPool Queue Length and ThreadPool Thread Count together: a growing queue while CPU usage stays low is starvation, and the almost-always root cause is code blocking a pool thread on asynchronous work instead of awaiting it.

// Starves the pool: the calling thread blocks waiting for I/O to complete,
// on every request, so the pool has to keep creating more threads just to
// keep processing the backlog of threads that are themselves stuck waiting.
public OrderSummary GetSummary(int orderId) =>
    _repository.GetSummaryAsync(orderId).GetAwaiter().GetResult();

// Frees the thread immediately; the pool only ever holds genuinely runnable work.
public Task<OrderSummary> GetSummaryAsync(int orderId) =>
    _repository.GetSummaryAsync(orderId);

The signature in counters is distinctive: CPU usage looks idle or moderate, because the threads are not computing, they are blocked, yet latency climbs and the queue length counter keeps growing. The thread pool grows cautiously by design, so once a backlog of blocked threads starts queuing new work behind it, the pool cannot add capacity fast enough to catch up on its own, and the backlog compounds rather than self-heals. The fix is to find and remove the blocking call, replacing .Result, .Wait() or GetAwaiter().GetResult() with await all the way up the call chain. When work genuinely is long-running and synchronous, such as a CPU-bound batch job, the answer is not to leave it on the shared pool at all, but to run it on a dedicated thread or a separately sized queue, so it cannot starve the pool that also serves ordinary request handling.

What interviewers look for: the specific counter pair and the "CPU idle, latency high" signature, plus knowing that the fix targets the blocking call itself, not just adding more threads via configuration.

Follow-up questions:

• Why does raising ThreadPool.SetMinThreads only mask this problem rather than fix it?
• How would you find the specific blocking call in a service with hundreds of endpoints?

Q7 How do you profile safely in a production container without hurting the service you're diagnosing?#

Short answer: Default to sampling, not instrumentation, bound every capture with an explicit --duration, avoid heavyweight providers such as full allocation sampling for anything but short, targeted windows, and treat the resulting trace file as sensitive data because it can contain values from live requests.

A short, sampling-based dotnet-trace collect typically costs a low, roughly constant single-digit percentage of CPU for its duration, which is safe against live traffic when it is bounded and not left running indefinitely. Heavier providers, such as the gc-verbose profile's allocation sampling, are appropriate for a focused, minutes-long capture during an active incident, not as a standing collection running continuously against production. Trace files also need somewhere to go: a busy service can fill a small container's writable filesystem quickly, so either write to a mounted volume with headroom or stream the capture off the container as soon as it completes, and clean up afterward. Finally, remember that a trace or a heap dump can contain data from real requests, sitting in stack arguments or the object graph itself, so handle the resulting artifact under the same data-handling rules that apply to production data generally, not as a throwaway debug file.

What interviewers look for: concrete, practiced safeguards, bounded duration, sampling over instrumentation, and data-handling awareness, rather than a general assurance that "it's safe because it's EventPipe."

Common mistakes: leaving a trace collecting indefinitely instead of a fixed duration; forgetting that a captured trace or dump can carry sensitive production data.

Q8 How would you profile a Kubernetes pod you can't get a shell into?#

Short answer: Run dotnet-monitor as a sidecar container in the same pod so it shares the app's process namespace, then reach its HTTP API through a kubectl port-forward to request a trace, dump or gcdump on demand, with no shell and no redeploy of the application image.

kubectl port-forward pod/orders-api-7d9f8 52323:52323

GET /trace?pid=1&profile=Cpu&durationSeconds=30 HTTP/1.1
Host: localhost:52323

dotnet-monitor ships both as a global tool and as a container image, and deploying it as a sidecar means it can see the application container's process without needing a shell inside that container at all, which matters for minimal or distroless images that do not ship a shell or diagnostic tools by design. Its HTTP API accepts a process identifier, a trace profile such as Cpu, and a bounded duration, and streams the resulting .nettrace file straight back over the connection. It can also be configured with automated collection rules that fire on a metric threshold, such as elevated ThreadPool Queue Length, so an artifact is already captured by the time anyone opens an incident. A newer, complementary option is kubectl debug, which can attach an ephemeral diagnostic container into a running pod's process namespace on demand, for cases where you need arbitrary tools rather than what dotnet-monitor's API exposes.

What interviewers look for: a concrete, no-shell-required answer naming dotnet-monitor as a sidecar and its HTTP-based collection model, which shows real container-diagnostics experience rather than a generic "I'd exec into the pod" answer that does not work against locked-down images.

Q9 How do you build a "performance culture" on a team so problems are caught before they reach production?#

Short answer: Tie performance to explicit, numeric budgets agreed with stakeholders, wire benchmarks and load tests into CI so regressions fail a build instead of reaching a customer, make the cost of a regression visible on a shared dashboard, and spread profiling skill across the team instead of concentrating it in one person.

A vague goal like "make it fast" cannot be enforced in review or in CI; a stated budget, such as p99 under a fixed threshold at a given request rate, can. Once that budget exists, benchmarks and load tests that check it belong in the pipeline, not in a senior engineer's personal habit of running them occasionally before a release. Making the current numbers visible on a dashboard the whole team sees, not just the person who gets paged, turns a regression into something the team notices together instead of something discovered during an incident. Performance incidents deserve the same blameless postmortem treatment as availability incidents, focused on the process gap that let a regression through, a missing benchmark, a missing alert, rather than on who wrote the offending line. Finally, treat profiling as a skill to teach deliberately: pairing sessions and short internal write-ups on real incidents build the muscle across the team, so finding a hot path or a starved thread pool is not a single point of failure resting on one person's experience.

What interviewers look for: a systemic answer that goes beyond "write more tests," touching budgets, CI enforcement, visibility and knowledge-sharing, which is what distinguishes a lead-level answer from a purely technical one.

Common mistakes: describing only tooling (benchmarks, dashboards) without mentioning the human side (review habits, blameless postmortems, knowledge-sharing) that keeps the tooling effective over time.

Q10 p99 latency spiked right after a deploy and CPU looks idle. Walk through finding the cause.#

Short answer: Idle CPU with high latency almost always means threads are waiting, not computing, so check ThreadPool Queue Length and Monitor Lock Contention Count first, and correlate whatever you find against exactly what changed in the deploy, since the timing match usually narrows the search faster than a fresh profiling session would.

dotnet-counters monitor --counters System.Runtime -p 4807

Start with the cheap, non-disruptive check: if ThreadPool Queue Length is elevated and climbing while CPU sits low, you are looking at starvation, and the deploy's diff is the fastest place to find a new blocking call, since the timing lines up exactly. If the thread pool looks healthy but Monitor Lock Contention Count is elevated, the same diff review applies, looking for a newly introduced or newly hot lock. If neither counter shows anything unusual, pivot away from the process itself and toward its dependencies: a downstream HTTP call or database query that got slower, whether from a new N+1 query pattern, a missing index, or a dependency that itself degraded around the same time, produces exactly this signature, idle CPU and high latency, because the request threads are asynchronously waiting on I/O rather than blocking a pool thread. Distributed tracing or dependency-duration metrics, not local CLR counters, are what confirm that branch. In both branches, the deploy correlation is doing most of the work: knowing precisely what changed turns "profile the whole service" into "check whether this specific change added a blocking call or a slower query," which is a much faster path to root cause under incident pressure.

What interviewers look for: the counters-first triage order, and explicit use of the deploy timing as a lead rather than treating the investigation as if it started from zero.

Follow-up questions:

• How would your approach change if the spike had no correlated deploy at all?
• What would make you suspect a downstream dependency over an in-process cause before looking at any tooling?