Production .NET services fail in ways a debugger cannot see: a pod that slowly runs out of memory, a request path that occasionally stalls for seconds while CPU sits idle, or a spike in CPU that only happens under real traffic. This guide is for engineers who own services in production and need to diagnose them without attaching a debugger or shipping a new build. You will learn the built-in .NET diagnostics toolkit built on EventPipe: dotnet-counters, dotnet-trace, dotnet-dump, dotnet-gcdump and dotnet-monitor, how PerfView and the Visual Studio diagnostic tools fit in, how to diagnose CPU, memory, thread-pool starvation and lock contention specifically, and how to collect diagnostics safely inside containers and Kubernetes.

What Is the .NET Diagnostics Toolkit?#

Every tool in this guide is a client of the same underlying mechanism: EventPipe, the cross-platform eventing system built into the .NET runtime since .NET Core 3.0. EventPipe replaces the old, Windows-only dependency on ETW for cross-platform scenarios: any .NET process exposes a diagnostic IPC channel (a Unix domain socket on Linux and macOS, a named pipe on Windows) that diagnostic clients connect to, without the target process needing to be started with any special flag. dotnet-counters, dotnet-trace, dotnet-dump and dotnet-gcdump are all thin command-line clients over this channel, and dotnet-monitor wraps the same clients behind an HTTP API meant to run continuously next to your app.

How EventPipe Diagnostics Work#

When a .NET process starts, it creates its diagnostic IPC socket, by default under the directory named by the TMPDIR (Unix) or a well-known temp location (Windows), named after its process ID. A diagnostic tool connects to that socket and sends a command: "start streaming these event providers," "write a GC dump," or "write a full process dump." Because the target process cooperates in generating the dump itself, tools like dotnet-dump and dotnet-gcdump typically do not need the same elevated ptrace-style access that attaching an external native debugger would need; they need to reach the socket file and, for a process dump, the same user context. This detail matters most in containers, covered later in this guide.

Getting Started: Installing and Connecting to a Process#

Every tool in this family installs the same way, as a .NET global tool, and every tool has a ps subcommand that lists the .NET processes it can see:

Bash
dotnet tool install --global dotnet-counters
dotnet tool install --global dotnet-trace
dotnet tool install --global dotnet-dump
dotnet tool install --global dotnet-gcdump

dotnet-counters ps

On .NET 10 SDKs, dnx can run a tool once without a permanent global install, which is convenient for CI jobs or a one-off investigation on a machine you do not own:

Bash
dnx dotnet-counters monitor --process-id 4821 --counters System.Runtime

dotnet-counters: Live Health Monitoring#

dotnet-counters is the first tool to reach for. It samples EventCounters and, on .NET 9 and later, meter-based metrics, and prints them live or writes them to a file, all with negligible overhead. The monitor command refreshes a live table; collect writes CSV or JSON for later analysis.

Bash
dotnet-counters monitor --process-id 4821 --counters System.Runtime --refresh-interval 1

dotnet-counters collect --process-id 4821 --counters System.Runtime --format json -o counters.json

The classic System.Runtime EventCounters, available since .NET Core 3.0, cover most first-look questions: cpu-usage, working-set, gc-heap-size, gen-0-gc-count, gen-1-gc-count, gen-2-gc-count, alloc-rate, exception-count, threadpool-thread-count, threadpool-queue-length, monitor-lock-contention-count and time-in-gc. Since .NET 9, the runtime also publishes an equivalent set of meter-based instruments with OpenTelemetry-style dotted names, such as dotnet.gc.collections, dotnet.process.cpu.time, dotnet.thread_pool.thread.count and dotnet.thread_pool.queue.length, which dotnet-counters can display the same way and which flow naturally into OpenTelemetry if you already export metrics that way.

dotnet-trace: CPU and Event Traces#

dotnet-trace records a time-ordered stream of runtime events, most commonly sampled call stacks for CPU analysis, without a native profiler and with much lower overhead than a full ETW session. collect is the main command; --profile selects a pre-built set of providers, and --providers lets you enable specific ones by name, keyword and level.

Bash
dotnet-trace collect --process-id 4821 --profile cpu-sampling --duration 00:00:00:30

dotnet-trace collect --process-id 4821 \
  --providers Microsoft-Windows-DotNETRuntime:0x4000:5 \
  -o contention.nettrace

Built-in profiles include cpu-sampling (thread stacks at roughly 100 Hz, for hot-path analysis), gc-verbose (detailed GC cycles with allocation sampling, higher overhead), gc-collect (GC pause timing only, very low overhead) and database (ADO.NET and EF Core command execution). The provider string in the second example enables the CLR's Contention keyword (0x4000) at an informational level, which is the targeted way to capture lock-contention events instead of a broad, high-overhead trace. dotnet-trace convert turns a .nettrace file into Speedscope or Chromium format so you can open it at speedscope.app or in Chrome's chrome://tracing without installing anything else.

dotnet-dump: Process Snapshots and SOS#

dotnet-dump collect writes a memory dump of a running process; dotnet-dump analyze opens an interactive shell over that dump using the same SOS debugging extension that Visual Studio and WinDbg use. Dumps are the heaviest tool in this list, both in size and in the moment of collection, because the runtime briefly suspends the process to produce a consistent snapshot.

Bash
dotnet-dump collect --process-id 4821 --type Full -o app_crash.dmp

dotnet-dump analyze app_crash.dmp
Text
> dumpheap -stat
> dumpheap -mt 00007ffab3c12345
> gcroot 00007ffab4a01230
> threads
> clrstack
> syncblk

The --type option controls how much is captured: Full includes the entire address space, Heap keeps only the managed heap and GC data, Mini is closest to a native crash dump, and Triage strips identifying data for sharing outside your organization. Inside analyze, dumpheap -stat gives an object-type histogram of the managed heap, gcroot traces what is keeping a specific object alive, clrstack and threads show managed call stacks, and syncblk lists monitor lock owners and waiters, which is the fastest way to confirm a deadlock or contention from a dump alone.

dotnet-gcdump: Heap Snapshots Without a Full Dump#

dotnet-gcdump triggers a garbage collection over EventPipe and reconstructs the object graph from the resulting events, instead of writing out the whole process. The result is far smaller and faster to collect than a full dump, and safer to pull repeatedly from a live, busy service.

Bash
dotnet-gcdump collect --process-id 4821 --timeout 60 -o snapshot1.gcdump

On Windows, .gcdump files open in PerfView or Visual Studio's managed memory analyzer, both of which can diff two snapshots to show exactly which types grew between them. As of this writing there is no supported way to open a .gcdump file directly on Linux or macOS, so a common pattern on Linux hosts is to collect it there and copy it to a Windows machine, or a Windows container, for analysis.

dotnet-monitor: Always-On Collection for Production#

dotnet-monitor wraps the tools above behind an HTTP API so you do not have to shell into a production host to use them. It runs as a global tool or, more commonly in production, as a Docker image deployed alongside your app. By default it binds its main API to https://localhost:52323 and, if metrics are enabled, a separate metrics endpoint on http://localhost:52325. API key authentication is enabled by default, and disabling it is explicitly called out as unsafe in production; --temp-apikey generates a short-lived key for local testing.

The feature that matters most for unattended production use is collection rules: conditions such as sustained high CPU, growing GC heap size or a spike in exceptions can automatically trigger a trace or dump, without a human watching a dashboard at the right moment. Collected artifacts are either downloaded over HTTP or, in restricted environments, pushed to an egress provider, such as blob storage, so nothing depends on the container's local, ephemeral disk.

PerfView and Visual Studio Diagnostic Tools#

PerfView is a free, Windows-based analysis tool from the .NET performance team that opens .nettrace and .etl files collected elsewhere, including from Linux via dotnet-trace. Its CPU stacks view, grouped and folded by module or type, is often faster for finding a genuinely hot call path than reading raw stacks, and its GC heap views can open and diff .gcdump snapshots.

Visual Studio has two complementary entry points. The Diagnostic Tools window appears automatically while debugging (F5) and shows live CPU, memory and events without any separate collection step. The standalone Performance Profiler (Debug > Performance Profiler, or Alt+F2) runs without a debugger attached, offers CPU Usage, .NET Object Allocation Tracking, Database and Events Viewer tools, and can also open .nettrace files that dotnet-trace collected on a Linux server, which is the most common way teams bring a Kubernetes production trace back to a developer machine for analysis.

Diagnosing Specific Problems#

High CPU concentrated in a few methods: confirm with dotnet-counters (cpu-usage or dotnet.process.cpu.time near 100%), then dotnet-trace collect --profile cpu-sampling for 15 to 30 seconds under load, and open the result in PerfView or Visual Studio to find the hottest call stacks.

Growing memory: watch gc-heap-size (or dotnet.gc.last_collection.memory.committed_size) over time with dotnet-counters; a size that never returns to baseline after a full GC is the signature of a leak, not just normal allocation. Take two dotnet-gcdump snapshots minutes apart under steady load and diff them, or go straight to dotnet-dump with dumpheap -stat and gcroot when you need to know exactly what is holding objects alive. Finding and Fixing Memory Leaks walks through this investigation end to end.

Thread-pool starvation: the signature is threadpool-thread-count (or dotnet.thread_pool.thread.count) climbing to two or three times the processor count while threadpool-queue-length (dotnet.thread_pool.queue.length) stays high and CPU usage is well under 100%, meaning threads are blocked rather than working. For an ongoing, reproducible case, dotnet-stack report --process-id 4821 prints current thread stacks immediately and often shows the culprit, such as a call sitting inside Task.GetResultCore or ManualResetEventSlim.Wait, both signs of blocking on asynchronous code. For an intermittent case, dotnet-trace captures the WaitHandleWait event, added in .NET 9, which fires whenever a thread blocks on a wait handle, giving you a time-ordered record even when you cannot catch the problem live.

Lock contention: monitor-lock-contention-count (or the .NET 9+ meter equivalent) rising under load points at contended lock statements or Monitor calls. Confirm with a dotnet-trace collection using the Contention keyword shown above, or, from an existing dump, syncblk to see exactly which thread holds a monitor and which threads are waiting on it.

Collecting Diagnostics in Containers and Kubernetes Safely#

Containers change two things about this toolkit: how a diagnostic client reaches the target process, and how safe it is to collect a large artifact on a resource-constrained pod.

  • Reaching the diagnostic socket. EventPipe's IPC socket lives under the container's temp directory, so a diagnostic tool running in a separate sidecar container needs to see it. Set shareProcessNamespace: true on the pod so containers can see each other's processes, and mount a shared, writable volume for the temp directory if your runtime or tooling does not honor DOTNET_DiagnosticPorts-style socket configuration by default.
  • Avoid extra debugger privileges. Because the target process writes its own dump on request over the diagnostic channel, dotnet-dump and dotnet-gcdump generally do not need the SYS_PTRACE capability that attaching a native debugger like gdb would, which keeps the pod's security context tighter.
  • Prefer the lightest tool for the job. A full dotnet-dump collect --type Full briefly suspends the process and can produce a multi-gigabyte file on a pod that may have little free disk or memory headroom; try dotnet-gcdump or a Mini/Triage dump first, and reserve a full dump for cases that genuinely need it.
  • Run dotnet-monitor as a sidecar, not as a public endpoint. Bind it to localhost or a cluster-internal network policy only, keep API key or Azure AD authentication on, and configure an egress provider so collected artifacts land in durable storage instead of the pod's ephemeral, restart-losing filesystem.
  • Treat every artifact as sensitive. Dumps and heap snapshots can contain connection strings, tokens and customer data that happened to be in memory; restrict who can trigger collection, encrypt artifacts at rest, and delete them once the investigation is done.

Running .NET on Kubernetes and Containerizing .NET Applications cover the surrounding pod and image design these safeguards build on.

Best Practices#

  • Start with dotnet-counters before anything heavier. It has the lowest overhead of any tool here and usually tells you which of the other tools to reach for next.
  • Prefer targeted providers over broad ones. --profile cpu-sampling or a specific keyword like Contention costs far less than enabling every CLR event.
  • Collect gcdumps before full dumps when the question is about the managed heap; escalate to a full dump only when you need native state or non-GC-rooted data too.
  • Automate collection with dotnet-monitor rules instead of waiting for someone to notice a problem and remote into a production host.
  • Copy artifacts off the host or pod immediately, since containers and cloud VMs can be recycled without warning, taking local files with them.
  • Correlate with the rest of your observability stack. Counters and traces are most useful lined up against the request logs and traces you already collect with OpenTelemetry in .NET.

Common Pitfalls#

  • Enabling gc-verbose or a full provider set on a busy production service for an extended period, adding overhead heavy enough to change the behavior you are trying to observe.
  • Collecting a full dump on a memory-constrained pod and triggering an out-of-memory kill during the collection itself.
  • Forgetting that a dump is a point-in-time snapshot. A single dump cannot show a leak growing; you need at least two snapshots, minutes apart, to see a trend.
  • Running diagnostic tools as a different user or in a separate PID namespace from the target process, so they cannot see its diagnostic socket at all.
  • Leaving dotnet-monitor authentication disabled or exposing its port publicly, turning a diagnostics endpoint into a way to exfiltrate memory contents.
  • Chasing CPU tools for a starvation problem, or thread-pool tools for a CPU problem. Read the counters first; high CPU and thread-pool starvation look similar in application logs but need different tools.

Choosing the Right Diagnostic Tool#

SymptomStart withEscalate to
Unknown, general health checkdotnet-counters monitordotnet-trace --profile cpu-sampling
High CPU in specific codedotnet-trace --profile cpu-samplingPerfView or Visual Studio CPU Usage view
Growing memory, suspected leakdotnet-counters (gc-heap-size trend)dotnet-gcdump, then dotnet-dump + gcroot
Requests stall, CPU idledotnet-counters (thread-pool counters)dotnet-stack report, dotnet-trace (WaitHandleWait)
Frequent lock waits under loaddotnet-counters (monitor-lock-contention-count)dotnet-trace with the Contention keyword, or syncblk
Unattended production monitoringdotnet-monitor collection rulesManual dotnet-dump/dotnet-trace for deep dives

Frequently Asked Questions#

Do I need to restart my application to use these diagnostic tools?#

No. Every tool in this toolkit connects to a running process over the diagnostic IPC channel that EventPipe exposes automatically; nothing needs to be enabled at startup, and there is no special build configuration required, unlike some third-party profilers that need an agent injected at launch.

What is the performance overhead of leaving dotnet-counters or dotnet-monitor running all the time?#

dotnet-counters sampling and dotnet-monitor's idle state add negligible overhead, similar to normal EventCounters and metrics collection you would run anyway. The cost comes from what you trigger, not from having the tool present: a cpu-sampling trace or a gcdump briefly adds overhead while it runs, and a full process dump causes a short pause while the snapshot is written.

How do I open a .nettrace file collected on a Linux server?#

Copy it to a Windows machine and open it in PerfView or Visual Studio's Performance Profiler, both of which read .nettrace files regardless of where they were collected. Alternatively, use dotnet-trace convert to produce a Speedscope file and view it in a browser at speedscope.app without installing anything.

Can I collect a dump from a process running inside a distroless or minimal container image?#

You need the dotnet-dump tool itself available to the target, either by running it from a debug sidecar container that shares the pod's process namespace, or by using dotnet-monitor as a sidecar that talks to the target over its diagnostic port. Neither approach requires installing anything inside the minimal application image itself.

Which tool should I reach for first when I do not know what is wrong?#

dotnet-counters monitor against the System.Runtime counters. In under a minute it tells you whether CPU, memory, thread-pool queue length or lock contention looks abnormal, which points you at the right specialized tool instead of guessing.

Summary#

  • Every tool here is a client of EventPipe, the runtime's built-in, cross-platform diagnostic channel.
  • dotnet-counters is the fast, low-overhead first look; dotnet-trace captures detailed event and CPU data; dotnet-dump and dotnet-gcdump capture heap and process state for offline analysis.
  • dotnet-monitor turns these into an HTTP API you can automate and gate behind authentication in production.
  • PerfView and the Visual Studio diagnostic tools are where you open and analyze the files these command-line tools collect.
  • In containers, share the process namespace or diagnostic socket, avoid unnecessarily heavy dumps, and treat every collected artifact as sensitive data.

Further Reading#