Observability and Distributed Tracing Interview Questions

Q1 What's the real difference between traces, metrics and logs, and how do you decide what to instrument with each in a .NET service?#

Short answer: Metrics are cheap, pre-aggregated numeric time series that tell you something is wrong across a fleet; traces are the causal, per-request graph that tell you where in one specific request it went wrong; logs are discrete, human-readable events that tell you why, down to a single line of code. You need all three because none of them can substitute for the question the others answer.

The dividing line in practice is cardinality. A metric like an HTTP request-duration histogram is cheap precisely because its labels — route template, status code, maybe a tenant tier — are drawn from a small, known set; a time-series database can pre-aggregate across millions of requests into percentiles without ever materializing a row per request. The moment you try to add a user_id or order_id label to a metric, you've turned a bounded time series into an effectively unbounded one, and most metrics backends either reject it, bill you heavily for it, or fall over. That unbounded, per-instance data is exactly what traces and logs are for: a trace attaches unlimited attributes to one specific request's span tree, and a log line can say exactly which order failed and why, because neither is trying to pre-aggregate across the whole fleet.

In a .NET service this maps onto three separate APIs that OpenTelemetry unifies: System.Diagnostics.Metrics (Meter, Counter<T>, Histogram<T>, ObservableGauge<T>) for metrics, System.Diagnostics.Activity/ActivitySource for traces, and ILogger for logs. The practical rule of thumb: instrument business and infrastructure counters and durations as metrics for dashboards and alerting, instrument any operation that crosses a service, database or queue boundary as a span so it shows up in a trace, and log at decision points and failure paths with enough structured context to reconstruct what happened without a debugger attached. Over-logging inside a hot loop is the single most common way teams blow their logging budget and drown out the signal they actually need.

What interviewers look for: that you reach for cardinality as the organizing principle instead of reciting "the three pillars" as trivia, and that you can name the concrete .NET types behind each signal.

Common mistakes: trying to use logs as a substitute for metrics — grepping and counting log lines to compute a rate — which doesn't scale and is far too slow to query under incident pressure.

Q2 Walk through how you'd wire OpenTelemetry into an ASP.NET Core service. What do the packages and the builder calls actually do?#

Short answer: You add the core OpenTelemetry SDK package plus instrumentation packages for the libraries you want auto-instrumented (ASP.NET Core, HttpClient, SQL, gRPC) and an exporter package, then call AddOpenTelemetry() on the service collection and configure tracing and metrics with WithTracing(...) and WithMetrics(...); .NET Aspire wraps the same setup behind a single call in its generated service-defaults project.

The instrumentation packages (OpenTelemetry.Instrumentation.AspNetCore, .Http, .SqlClient, .GrpcNetClient, and others) don't invent new telemetry — they subscribe to the ActivitySources and Meters that ASP.NET Core, HttpClient, Microsoft.Data.SqlClient and gRPC already emit internally, and translate them into OpenTelemetry's data model. That's the important design point: instrumentation is decoupled from the exporter. You can point the same pipeline at the console exporter while developing locally and an OTLP exporter (OpenTelemetry.Exporter.OpenTelemetryProtocol) against a collector in staging and production, without touching a single line of business code.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("orders-api"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSqlClientInstrumentation()
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddOtlpExporter());

var app = builder.Build();

ConfigureResource sets the service name and attributes that every span and metric carries so a backend can group telemetry by service; without it, every instance of every service shows up as an anonymous blob. In a project built on .NET Aspire, this entire block is generated once in a shared ServiceDefaults project and pulled into each service with a single builder.AddServiceDefaults() call, which is why interviewers increasingly expect you to know both the raw SDK shape and the fact that Aspire exists specifically to stop every team from hand-rolling it service by service.

What interviewers look for: fluency with the actual builder API, not just "we use OpenTelemetry" — and awareness that instrumentation packages wrap existing ActivitySource/Meter emission rather than being a separate telemetry system bolted on top.

Follow-up questions:

• How would you add a custom span around a business operation that isn't auto-instrumented?
• What's the practical difference between pushing to an OTLP collector and exposing a Prometheus scrape endpoint?

Q3 Explain ActivitySource and Activity in .NET. Why did Microsoft build distributed tracing into the base class library instead of leaving it entirely to a tracing vendor's SDK?#

Short answer: ActivitySource is the type a library or application uses to create Activity instances (spans, in OpenTelemetry terms); it lives in System.Diagnostics, ships as part of the runtime, and has zero dependency on OpenTelemetry or any other backend, so a library can emit rich tracing data without ever taking a dependency on a specific vendor — an ActivityListener attached later decides what to sample, tag and export.

This split solves a chicken-and-egg problem that plagued .NET tracing for years: a library author — say, the maintainer of a database driver — wants to emit tracing data but doesn't want to force every consumer onto one specific tracing vendor's SDK, and doesn't want to ship five instrumentation variants for five vendors. By putting ActivitySource/Activity in the base class library, ASP.NET Core, HttpClient, SqlClient and gRPC can all create activities unconditionally — the cost is negligible when nothing is listening — and whichever observability library the application actually references, OpenTelemetry being the dominant one today, subscribes via ActivityListener and decides what happens next.

private static readonly ActivitySource Source = new("OrdersApi.Checkout");

public async Task<OrderResult> CheckoutAsync(Cart cart, CancellationToken ct)
{
    using var activity = Source.StartActivity("checkout.process", ActivityKind.Internal);
    activity?.SetTag("cart.item_count", cart.Items.Count);
    activity?.SetTag("cart.total", cart.Total);

    var result = await _paymentClient.ChargeAsync(cart, ct);
    activity?.SetTag("payment.status", result.Status);
    return result;
}

StartActivity returns null — not a no-op instance — when nothing is sampling that source, which is why every call is null-conditional (activity?.); it's a deliberate fast path so instrumenting your own business logic this way costs essentially nothing when no exporter is attached. Whether that activity turns into an exported span is decided later, by the ActivityListener's sampling callback — the exact hook OpenTelemetry's samplers plug into.

What interviewers look for: understanding that the runtime provides the instrumentation API and OpenTelemetry provides the pipeline — sampling, processing, export — on top of it; that decoupling is the actual architectural insight, not just "Activity is like a span."

Common mistakes: assuming you need an OpenTelemetry package reference just to add custom spans to your own code — ActivitySource and Activity are already part of the .NET runtime, with no extra dependency required.

Q4 What is the W3C Trace Context standard, and how does context actually propagate across an HTTP call between two .NET services?#

Short answer: W3C Trace Context is a standardized HTTP header format — traceparent, plus an optional tracestate — for carrying a trace ID, the calling span's ID and sampling flags across a network boundary, and it has been the default ID format for Activity in .NET since .NET 5, replacing the older, .NET-only "hierarchical" ID format that nothing outside the .NET ecosystem understood.

Every Activity, once started, has a 16-byte TraceId shared by every span in the same logical request, an 8-byte SpanId unique to that activity, and a ParentSpanId pointing at whichever activity caused it; the format is controlled by Activity.DefaultIdFormat, and W3C trace context has been the default since .NET 5. When an instrumented HttpClient makes an outbound call, the instrumentation serializes the current activity's trace ID, span ID and flags into a traceparent header on the outgoing request; the receiving service's ASP.NET Core instrumentation reads that header before the request pipeline starts, and starts its own activity as a child of the incoming context instead of a new, disconnected trace.

POST /api/orders/checkout HTTP/1.1
Host: orders-api.internal
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
tracestate: vendorname=opaquevalue

The traceparent value breaks down as version (00), the 32-hex-character trace ID, the 16-hex-character parent (span) ID, and 2-hex-character trace flags, whose low bit signals whether the caller wants this trace sampled. tracestate is a separate, vendor-extensible header that carries additional state without every intermediary needing to understand it. Because this is a W3C standard rather than a .NET-specific convention, a trace can cross from a .NET service into a Node.js, Java or Go service and back and still form one continuous trace in your backend — which is the entire point of standardizing it instead of leaving every framework to invent its own propagation format.

What interviewers look for: the concrete header name and structure, not just "trace context propagates via headers" — and the understanding that standardization, not the propagation mechanism itself, is what made cross-language distributed tracing practical.

Follow-up questions:

• How would you detect that an intermediary proxy is stripping trace headers and silently breaking traces?
• How does trace context propagate across a message queue instead of a direct HTTP call?

Q5 Your system handles tens of thousands of requests per second, and exporting a full trace for every one of them isn't realistic. Explain head-based versus tail-based sampling and when you'd use each.#

Short answer: Head-based sampling decides whether to record a trace at its very start — usually a cheap probabilistic decision like "sample 5% of requests" — made before anyone knows how the request turns out; tail-based sampling waits until a trace is complete and then decides whether to keep it, which lets you always keep the traces that actually matter, such as errors or high latency, at the cost of buffering and coordinating that decision across every service involved.

Head-based sampling is simple and cheap: only the root service flips a weighted coin when the trace starts, and the decision then propagates down via the sampled flag in the traceparent header so every downstream service honors it rather than each one deciding independently. The problem is exactly what you'd expect from random sampling: if your error rate is 0.1% and your sampling rate is 5%, the overwhelming majority of sampled traces are boring, successful requests, and there's a real chance you never capture a trace for the specific failure you're trying to debug.

Tail-based sampling fixes that by deferring the decision: every span for a trace is buffered — typically by an OpenTelemetry Collector configured with a tail-sampling processor — until the trace looks complete, and then a policy decides whether to export it: keep everything with an error status, keep everything above a latency threshold, and randomly sample only a small slice of the boring, fast, successful traffic. The cost is real — the collector has to hold spans in memory until a trace closes out, and because a single trace's spans can arrive at different collector instances behind a load balancer, tail sampling usually requires routing all spans for one trace ID to the same collector instance, which adds operational complexity head-based sampling never has to deal with.

What interviewers look for: an articulated trade-off — statistical coverage and simplicity versus guaranteed capture of interesting traces at the cost of memory and routing complexity — rather than just naming both terms.

Common mistakes: assuming sampling is decided independently per service; an inconsistent sampling decision across services is exactly what produces broken, partial traces that stop halfway through a call graph.

Q6 How do you design structured logging so it's actually useful at scale, and what's specifically wrong with string-interpolated log messages?#

Short answer: Structured logging means every log call emits a message template plus a set of named, typed properties as separate fields — not a fully-formed string — so a log backend can index, filter and aggregate on those fields instead of running regular expressions over free text; string interpolation throws that structure away and also defeats ILogger's built-in log-level short-circuiting, since the string gets built whether or not the log level is even enabled.

ILogger's message-template overloads exist for this reason: logger.LogWarning("Order {OrderId} failed with status {Status}", orderId, status) keeps OrderId and Status as first-class, queryable fields in any structured sink — Application Insights, Seq, an OTLP log exporter, Elasticsearch — while still rendering a readable message for console output. Interpolating the string yourself collapses that into one opaque blob of text, so "find every failed order for customer X" turns from an indexed field query into a full-text scan across your entire log volume, which is slow, expensive and fragile the moment someone edits the message wording.

public partial class CheckoutService
{
    private readonly ILogger<CheckoutService> _logger;

    [LoggerMessage(Level = LogLevel.Warning,
        Message = "Order {OrderId} failed with status {Status}")]
    partial void LogOrderFailed(string orderId, string status);

    public async Task ProcessAsync(Order order, CancellationToken ct)
    {
        var result = await _paymentGateway.ChargeAsync(order, ct);
        if (!result.Succeeded)
        {
            LogOrderFailed(order.Id, result.Status);
        }
    }
}

The [LoggerMessage] source generator goes further than the template overloads: it generates a strongly-typed logging method at compile time that checks whether the level is enabled before doing any formatting or boxing, so a disabled call in a hot path costs essentially nothing. At scale, the other half of the discipline is correlation and cardinality: every log line should carry the current trace ID so you can pivot from a dashboard straight to a request's logs, and properties that are effectively unique per call need to be chosen deliberately, because an indexed field with unbounded cardinality is exactly as expensive as an unbounded metric label.

What interviewers look for: the mechanical reason interpolation is wrong — losing structure and defeating level checks — not just "structured logging is a best practice," plus the trace-ID correlation detail that actually makes logs useful during an incident.

Q7 Define SLI, SLO and error budget. How do you choose good SLIs for an API, and what should happen operationally once the error budget is spent?#

Short answer: An SLI (service level indicator) is a specific, measured signal of user-perceived behavior — the proportion of requests served under some latency threshold, or the proportion returning a non-5xx status; an SLO (service level objective) is a target for that SLI over a window, such as "99.9% of requests succeed over 30 days"; the error budget is 100% minus the SLO, converted into an allowance of permitted badness that's meant to be spent, not hoarded.

Good SLIs measure what the user actually experiences, not what's easiest to instrument. Server-side CPU or queue depth are useful operational metrics but poor SLIs, because a user doesn't care that CPU hit 80 percent — they care whether their request came back correctly and fast. The strongest SLIs are usually a request-success ratio measured at the edge, not inside one internal service, and a latency threshold expressed as a percentile, because percentiles capture tail behavior an average silently hides: a service can have a perfectly healthy average latency while one percent of users wait several seconds.

The error budget turns an SLO from an aspiration into an operating agreement between product and engineering: while comfortably within budget, that's the signal it's safe to ship faster or take more risk with a rollout. Once the budget is exhausted for the period, the agreed consequence — agreed before the incident, never negotiated during it — is typically a freeze on further feature risk until reliability work brings the service back under budget. The failure mode teams fall into without this agreement is arguing about whether the SLO even matters in the middle of an incident, which a pre-agreed budget and its consequences exist to prevent.

What interviewers look for: the distinction between an operationally convenient metric and a genuine user-facing SLI, and whether you treat the error budget as a governance tool with real consequences rather than a vanity dashboard number.

Common mistakes: setting an SLO at 100 percent, which is both unachievable and removes any room to take risk safely, or measuring an SLI from inside the system instead of from the client's vantage point, which hides exactly the failures — DNS, load balancer, network — that matter most to users.

Q8 Why should alerts fire on symptoms rather than causes, and what does that look like in practice for the same underlying problem?#

Short answer: A symptom-based alert fires on user-visible impact — elevated error rate, a breached latency SLO, a queue actually backing up past a threshold that matters — while a cause-based alert fires on an internal signal that might be a problem, like CPU crossing 80 percent; symptom alerts page you only when something is genuinely broken for a user, while cause alerts page you constantly for things that self-heal or never mattered, which is how teams end up ignoring their own paging system.

Take a downstream database connection pool exhausting itself under load. A cause-based version of this alert fires the instant pool utilization crosses some percentage, which happens routinely during normal traffic spikes and autoscaling catch-up, producing a page that requires a human to go check whether it actually mattered. A symptom-based version instead fires when latency or error rate on the endpoints that depend on that pool actually breaches the SLO for a sustained window, which only happens when pool exhaustion is severe enough to be a real, user-visible problem — and it stays silent through the transient spikes that resolve on their own. The pool-utilization metric doesn't disappear; it still belongs on a dashboard, and it's exactly what you look at after the symptom alert pages you, to find the cause. It just isn't what wakes someone up at 3 a.m.

This is also why burn-rate alerting is the standard pattern for SLO-based paging rather than a flat threshold: a fast burn rate — consuming a month's error budget in an hour — should page immediately, while a slow, steady burn over the full window can be a lower-urgency ticket instead of a page, because the two situations call for genuinely different response times. Getting this distinction wrong in either direction is expensive: alerting purely on causes trains engineers to ignore pages, and alerting only on the slowest possible symptom signal means you don't find out you're burning budget until it's too late to do anything but apologize.

What interviewers look for: a concrete example distinguishing the two, not just definitions, plus familiarity with burn-rate alerting as the mechanism that makes symptom-based SLO alerting actionable at different urgencies.

Q9 A downstream dependency starts timing out intermittently. Walk through how you'd use traces, metrics and logs together to find the root cause.#

Short answer: Start at the metric that told you something's wrong — elevated p99 latency or error rate on a dashboard — pivot to traces to find which downstream call is actually slow and whether it's isolated or cascading, then pivot from one specific slow trace's ID into that service's logs to see the exact exception or resource state at that moment: metrics tell you it's happening, traces tell you where, logs tell you why.

The dashboard alert typically starts you at an aggregate: p99 latency on an endpoint crossed the SLO threshold five minutes ago. That number alone doesn't tell you whether the slowness is in your own service, a specific downstream call, or something environmental like garbage-collection pauses — so the next step is opening a handful of actual traces from that time window, filtered to the slow ones, and looking at the span breakdown. If every slow trace shows the same downstream span — a call to a payment provider, or a specific SQL query — taking most of the wall-clock time, you've localized the problem to one dependency instead of guessing across the whole call graph.

From there, you take the trace ID of one specific slow or failed request and pivot directly into logs filtered to that trace ID, across every service that touched it — only possible because trace ID was correlated into every log line to begin with. That's where you find detail a span's duration can't tell you: a connection-pool exhaustion exception, a retry-after value from the dependency, a timeout configured lower than the dependency's real p99. The pattern generalizes: metrics for when and how much, traces for where in the call graph, logs for the specific reason at that exact point — and the trace ID is the thread that lets you move between all three without re-deriving context by hand.

What interviewers look for: a coherent, ordered investigation — metric to trace to log — rather than three disconnected facts about each signal, and explicit use of trace-ID correlation as the mechanism that ties the investigation together.

Common mistakes: starting the investigation in raw logs, grepping for errors around the right timestamp, without first localizing which service and dependency is responsible — slow, and often misleading, under high log volume.

Q10 As a system grows to dozens of services, how do you keep observability cardinality and cost under control without losing the ability to debug incidents?#

Short answer: Treat every tag or attribute attached to a metric as a cost decision limited to bounded, known-set values; keep traces and logs, which can carry high-cardinality data safely, as the place for anything per-request; apply tail or adaptive sampling so you're not exporting 100 percent of uninteresting traffic; and use tiered retention so expensive, high-detail data ages out fast while cheap, aggregated data is kept far longer.

The most common way teams blow their observability budget isn't traffic volume — it's an engineer adding a high-cardinality attribute to a metric, such as a user or session identifier on a counter, usually to make one debugging session easier, and forgetting to remove it afterward. That single change can multiply a time-series database's storage and query cost by orders of magnitude, because the backend now tracks a separate series per unique value instead of a handful. The fix is process as much as tooling: attributes on metrics get reviewed the way a schema migration would be, and anything needing per-instance granularity goes on a span or log line instead, where high cardinality is the expected, supported case.

On the tracing side, cost control is mostly a sampling and retention problem: sample aggressively on the high-volume, boring path, keep everything that's an error or breaches a latency threshold via tail sampling, and set a retention window on raw trace data that's long enough to debug last week's incident but not so long you're storing months of successful traces nobody will ever query. Logs follow the same shape — informational and above kept for weeks, debug-level detail kept for days or enabled only on demand — while the aggregated metrics derived from all of it, like error rate and SLO burn, get retained far longer, because a year-over-year reliability trend is cheap to store and genuinely valuable, unlike a year of raw spans.

What interviewers look for: recognition that cardinality, not raw request volume, is the actual cost driver, and a concrete retention and sampling strategy rather than "just sample less," which trades away exactly the incident-debugging capability observability exists to provide.