From a666306e40724817986416fd2bf5205b89057184 Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Thu, 7 Aug 2025 16:51:43 -0400 Subject: [PATCH 01/10] [Diagnostics] Cleanup --- docs/core/diagnostics/dotnet-trace.md | 38 +++++++++++-------- .../eventsource-collect-and-view-traces.md | 2 +- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index b2f6512906ca9..a6f6e17b860dd 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -43,7 +43,7 @@ The `dotnet-trace` tool: * Is a cross-platform .NET Core tool. * Enables the collection of .NET Core traces of a running process without a native profiler. * Is built on [`EventPipe`](./eventpipe.md) of the .NET Core runtime. -* Delivers the same experience on Windows, Linux, or macOS. +* On Linux, provides additional integration with kernel user_events for native tracing tool compatibility. ## Options @@ -55,10 +55,6 @@ The `dotnet-trace` tool: Displays the version of the dotnet-trace utility. -- **`--duration`** - - How long to run the trace. `--duration 00:00:00:05` will run it for 5 seconds. - ## Commands | Command | @@ -76,13 +72,23 @@ Collects a diagnostic trace from a running process or launches a child process a ### Synopsis ```dotnetcli -dotnet-trace collect [--buffersize ] [--clreventlevel ] [--clrevents ] +dotnet-trace collect + [--buffersize ] + [--clreventlevel ] + [--clrevents ] [--dsrouter ] - [--format ] [-h|--help] [--duration dd:hh:mm:ss] - [-n, --name ] [--diagnostic-port] [-o|--output ] [-p|--process-id ] - [--profile ] [--providers ] + [--format ] + [-h|--help] + [--duration dd:hh:mm:ss] + [-n, --name ] + [--diagnostic-port] + [-o|--output ] + [-p|--process-id ] + [--profile ] + [--providers ] [-- ] (for target applications running .NET 5 or later) - [--show-child-io] [--resume-runtime] + [--show-child-io] + [--resume-runtime] [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] @@ -158,7 +164,7 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- - **`--dsrouter {ios|ios-sim|android|android-emu}** - Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information. + Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information. - **`--format {Chromium|NetTrace|Speedscope}`** @@ -204,11 +210,11 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available: - | Profile | Description | - |---------|-------------| - |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.| - |`gc-verbose`|Tracks GC collections and samples object allocations.| - |`gc-collect`|Tracks GC collections only at very low overhead.| + | Profile | Description | + |---------|-------------| + |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.| + |`gc-verbose`|Tracks GC collections and samples object allocations.| + |`gc-collect`|Tracks GC collections only at very low overhead.| - **`--providers `** diff --git a/docs/core/diagnostics/eventsource-collect-and-view-traces.md b/docs/core/diagnostics/eventsource-collect-and-view-traces.md index 14e7dcbc6975a..fd6b2a32d9dc6 100644 --- a/docs/core/diagnostics/eventsource-collect-and-view-traces.md +++ b/docs/core/diagnostics/eventsource-collect-and-view-traces.md @@ -214,7 +214,7 @@ into other formats, such as Chromium or [Speedscope](https://www.speedscope.app/ Trace completed. ``` - dotnet-trace uses the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in + dotnet-trace uses a comma-delimited variant of the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in the `--providers` argument. For more options on how to take traces using dotnet-trace, see the [dotnet-trace docs](./dotnet-trace.md#collect-a-trace-with-dotnet-trace). From a2e4f74e421bad25777e98de5608157becf1f2f4 Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Thu, 7 Aug 2025 16:52:00 -0400 Subject: [PATCH 02/10] [Diagnostisc][dotnet-trace] Add collect-linux verb --- docs/core/diagnostics/dotnet-trace.md | 129 ++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index a6f6e17b860dd..bba276360ff3e 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -60,6 +60,7 @@ The `dotnet-trace` tool: | Command | |-----------------------------------------------------------| | [dotnet-trace collect](#dotnet-trace-collect) | +| [dotnet-trace collect-linux](#dotnet-trace-collect-linux) | | [dotnet-trace convert](#dotnet-trace-convert) | | [dotnet-trace ps](#dotnet-trace-ps) | | [dotnet-trace list-profiles](#dotnet-trace-list-profiles) | @@ -265,6 +266,134 @@ dotnet-trace collect > - When specifying a stopping event through the `--stopping-event-*` options, as the EventStream is being parsed asynchronously, there will be some events that pass through between the time a trace event matching the specified stopping event options is parsed and the EventPipeSession is stopped. +## dotnet-trace collect-linux + +Collects diagnostic traces from .NET applications using Linux user_events as a transport layer. This command provides the same functionality as [`dotnet-trace collect`](#dotnet-trace-collect) but routes .NET runtime events through the Linux kernel's user_events subsystem before writing them to `.nettrace` files. + +This transport approach enables automatic unification of user-space .NET events with kernel-space system events, since both are captured in the same kernel tracing infrastructure. Linux tools like `perf` and `ftrace` can monitor events in real-time while maintaining full compatibility with existing .NET profiling workflows. + +### Prerequisites + +- Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+) +- Appropriate permissions to access `/sys/kernel/tracing/user_events_data` +- .NET 10+ + +### Synopsis + +```dotnetcli +dotnet-trace collect-linux + [--buffersize ] + [--clreventlevel ] + [--clrevents ] + [--format ] + [-h|--help] + [--duration dd:hh:mm:ss] + [-n, --name ] + [--diagnostic-port] + [-o|--output ] + [-p|--process-id ] + [--profile ] + [--providers ] + [-- ] (for target applications running .NET 10 or later) + [--show-child-io] + [--resume-runtime] + [--stopping-event-provider-name ] + [--stopping-event-event-name ] + [--stopping-event-payload-filter ] + [--tracepoint-configs ] + [--kernel-events ] +``` + +### Options + +`dotnet-trace collect-linux` supports all the same options as [`dotnet-trace collect`](#dotnet-trace-collect), excluding `--dsrouter`, and additionally offers: + +- **`--tracepoint-configs ` (required)** + + Defines the explicit mapping between EventPipe providers and kernel tracepoints. Each provider in `--providers` must have a corresponding entry in `--tracepoint-configs` + + **Format:** `ProviderName::` + + Where: + - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) + - `DefaultTracepointName`: Default tracepoint name for this provider (can be empty to require explicit assignment) + - `TracepointSets`: Semi-colon delimited `TracepointName=` + - `EventIds`: Plus-delimited event IDs to route to that tracepoint + + > [!NOTE] + > All tracepoint names are automatically prefixed with the provider name to avoid collisions. For example, `gc_events` for the `Microsoft-Windows-DotNETRuntime` provider becomes `Microsoft_Windows_DotNETRuntime_gc_events`. + + **Examples:** + ``` + # Scenario: All events from provider go to a default tracepoint + --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" + # All enabled events from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_dotnet_runtime + + # Scenario: Split events by categories + --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3;jit_events=10+11+12" + # EventIDs 1, 2, and 3 will be written to Microsoft_Windows_DotNETRuntime_gc_events + # EventIDs 10, 11, and 12 will be written to Microsoft_Windows_DotNETRuntime_jit_events + + # Multiple providers (comma-separated) + --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3,MyCustomProvider:custom_events" + # EventIds 1, 2, and 3 from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_gc_events + # All enabled events from MyCustomProvider will be written to MyCustomProvider_custom_events + ``` + +- **`--kernel-events ` (optional)** + + A comma-separated list of kernel event categories to include in the trace. These events are automatically grouped into kernel-named tracepoints. Available categories include: + + | Category | Description | Linux Tracepoints | + |----------|-------------|-------------------| + | `syscalls` | System call entry/exit events | `syscalls:sys_enter_*`, `syscalls:sys_exit_*` | + | `sched` | Process scheduling events | `sched:sched_switch`, `sched:sched_wakeup` | + | `net` | Network-related events | `net:netif_rx`, `net:net_dev_xmit` | + | `fs` | Filesystem I/O events | `ext4:*`, `vfs:*` | + | `mm` | Memory management events | `kmem:*`, `vmscan:*` | + + These events correspond to Linux kernel tracepoints documented in the [Linux kernel tracing documentation](https://www.kernel.org/doc/html/latest/trace/index.html). For more details on available tracepoints, see [ftrace](https://www.kernel.org/doc/html/latest/trace/ftrace.html) and [tracepoints](https://www.kernel.org/doc/html/latest/trace/tracepoints.html). + + Example: `--kernel-events syscalls,sched,net` + +### Linux Integration + +**Tracepoint Configuration Requirements:** + +- **Mandatory Mapping**: Every provider must be explicitly mapped to at least a default tracepoint and/or exclusive tracepoint sets via `--tracepoint-configs` +- **Tracepoint Isolation**: Each tracepoint can only receive events from one provider +- **Event Routing**: Different event IDs within a provider can be routed to different tracepoints for granular control +- **Automatic Prefixing**: All tracepoint names are prefixed with the provider name to avoid collisions + +**Kernel Integration Points:** + +The kernel tracepoints can be accessed through standard Linux tracing interfaces: + +- **ftrace**: `/sys/kernel/tracing/events/user_events/` +- **perf**: Use `perf list user_events*` to see available events +- **System monitoring tools**: Any tool that can consume Linux tracepoints + +### Examples + +```dotnetcli +# All runtime events to one tracepoint +dotnet-trace collect-linux --process-id 1234 \ + --providers Microsoft-Windows-DotNETRuntime:0x8000:5 \ + --kernel-events syscalls,sched \ + --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" + +# Split runtime events by category +dotnet-trace collect-linux --process-id 1234 \ + --providers Microsoft-Windows-DotNETRuntime:0x8001:5 \ + --kernel-events syscalls,sched,net,fs \ + --tracepoint-configs "Microsoft-Windows-DotNETRuntime::exception_events=80;gc_events=1+2" + +# Multiple providers +dotnet-trace collect-linux --process-id 1234 \ + --providers "Microsoft-Windows-DotNETRuntime:0x8001:5,MyCustomProvider:0xFFFFFFFF:5" \ + --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime,MyCustomProvider:custom_events" +``` + ## dotnet-trace convert Converts `nettrace` traces to alternate formats for use with alternate trace analysis tools. From e1a3ec705989ce75b903018b57d9096373cc18f2 Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Thu, 7 Aug 2025 17:13:53 -0400 Subject: [PATCH 03/10] [Diagnostics][dotnet-trace] Add event filter option --- docs/core/diagnostics/dotnet-trace.md | 33 +++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index bba276360ff3e..616f3d7e01754 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -93,6 +93,7 @@ dotnet-trace collect [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] + [--event-filters ] ``` ### Options @@ -256,6 +257,34 @@ dotnet-trace collect A string, parsed as [payload_field_name]:[payload_field_value] pairs separated by commas, that will stop the trace upon hitting an event containing all specified payload pairs. Requires `--stopping-event-provider-name` and `--stopping-event-event-name` to be set. for example, `--stopping-event-provider-name Microsoft-Windows-DotNETRuntime --stopping-event-event-name Method/JittingStarted --stopping-event-payload-filter MethodNameSpace:Program,MethodName:OnButtonClick` to stop the trace upon the first `Method/JittingStarted` event for the method `OnButtonClick` in the `Program` namespace emitted by the `Microsoft-Windows-DotNETRuntime` event provider. +- **`--event-filters `** + + Defines an additional optional filter for each provider's events. When no `--event-filters` is specified for a provider, all events allowed by the provider's keywords and level configuration are collected. Event filters provide additional granular control beyond the keyword/level filtering. + + **Format:** `ProviderName::` + + Where: + - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) + - `Enable` : Boolean value indicating whether EventIds will be enabled or disabled, defaults to false + - `EventIds`: Plus-delimited event IDs to enable or disable, defaults to empty. + + **Examples:** + ``` + # Scenario: Disable specific events from Microsoft-Windows-DotNETRuntime + --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3+4+5+6+7+8+9" + + # Scenario: Enable specific events from a provider + --event-filters "Microsoft-Windows-DotNETRuntime:true:80+129+130+250" + # Only events 80, 129, 130, and 250 will be collected from this provider (others are filtered out) + + # Scenario: Multiple providers with mixed filtering - some providers have no filters + --providers "Microsoft-Windows-DotNETRuntime:0xFFFFFFFF:5,System.Threading.Tasks.TplEventSource:0xFFFFFFFF:5,MyCustomProvider:0xFFFFFFFF:5" + --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3,System.Threading.Tasks.TplEventSource:true:7+8+9" + # Microsoft-Windows-DotNETRuntime: All events EXCEPT 1,2,3 are collected + # System.Threading.Tasks.TplEventSource: ONLY events 7,8,9 are collected + # MyCustomProvider: ALL events are collected (no filter specified - follows provider keywords/level) + ``` + > [!NOTE] > - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace. @@ -300,6 +329,7 @@ dotnet-trace collect-linux [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] + [--event-filters ] [--tracepoint-configs ] [--kernel-events ] ``` @@ -323,6 +353,9 @@ dotnet-trace collect-linux > [!NOTE] > All tracepoint names are automatically prefixed with the provider name to avoid collisions. For example, `gc_events` for the `Microsoft-Windows-DotNETRuntime` provider becomes `Microsoft_Windows_DotNETRuntime_gc_events`. + > [!TIP] + > Use `--event-filters` to disable specific events before they are routed to tracepoints. Event filtering happens before tracepoint routing - only events that pass the filter will be sent to their assigned tracepoints. + **Examples:** ``` # Scenario: All events from provider go to a default tracepoint From 6c9fdb6cc0dd668905fcaacf0bf11e8159a89cfc Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Thu, 14 Aug 2025 12:18:10 -0400 Subject: [PATCH 04/10] Revert "[Diagnostics][dotnet-trace] Add event filter option" This reverts commit e1a3ec705989ce75b903018b57d9096373cc18f2. --- docs/core/diagnostics/dotnet-trace.md | 33 --------------------------- 1 file changed, 33 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index 616f3d7e01754..bba276360ff3e 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -93,7 +93,6 @@ dotnet-trace collect [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] - [--event-filters ] ``` ### Options @@ -257,34 +256,6 @@ dotnet-trace collect A string, parsed as [payload_field_name]:[payload_field_value] pairs separated by commas, that will stop the trace upon hitting an event containing all specified payload pairs. Requires `--stopping-event-provider-name` and `--stopping-event-event-name` to be set. for example, `--stopping-event-provider-name Microsoft-Windows-DotNETRuntime --stopping-event-event-name Method/JittingStarted --stopping-event-payload-filter MethodNameSpace:Program,MethodName:OnButtonClick` to stop the trace upon the first `Method/JittingStarted` event for the method `OnButtonClick` in the `Program` namespace emitted by the `Microsoft-Windows-DotNETRuntime` event provider. -- **`--event-filters `** - - Defines an additional optional filter for each provider's events. When no `--event-filters` is specified for a provider, all events allowed by the provider's keywords and level configuration are collected. Event filters provide additional granular control beyond the keyword/level filtering. - - **Format:** `ProviderName::` - - Where: - - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) - - `Enable` : Boolean value indicating whether EventIds will be enabled or disabled, defaults to false - - `EventIds`: Plus-delimited event IDs to enable or disable, defaults to empty. - - **Examples:** - ``` - # Scenario: Disable specific events from Microsoft-Windows-DotNETRuntime - --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3+4+5+6+7+8+9" - - # Scenario: Enable specific events from a provider - --event-filters "Microsoft-Windows-DotNETRuntime:true:80+129+130+250" - # Only events 80, 129, 130, and 250 will be collected from this provider (others are filtered out) - - # Scenario: Multiple providers with mixed filtering - some providers have no filters - --providers "Microsoft-Windows-DotNETRuntime:0xFFFFFFFF:5,System.Threading.Tasks.TplEventSource:0xFFFFFFFF:5,MyCustomProvider:0xFFFFFFFF:5" - --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3,System.Threading.Tasks.TplEventSource:true:7+8+9" - # Microsoft-Windows-DotNETRuntime: All events EXCEPT 1,2,3 are collected - # System.Threading.Tasks.TplEventSource: ONLY events 7,8,9 are collected - # MyCustomProvider: ALL events are collected (no filter specified - follows provider keywords/level) - ``` - > [!NOTE] > - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace. @@ -329,7 +300,6 @@ dotnet-trace collect-linux [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] - [--event-filters ] [--tracepoint-configs ] [--kernel-events ] ``` @@ -353,9 +323,6 @@ dotnet-trace collect-linux > [!NOTE] > All tracepoint names are automatically prefixed with the provider name to avoid collisions. For example, `gc_events` for the `Microsoft-Windows-DotNETRuntime` provider becomes `Microsoft_Windows_DotNETRuntime_gc_events`. - > [!TIP] - > Use `--event-filters` to disable specific events before they are routed to tracepoints. Event filtering happens before tracepoint routing - only events that pass the filter will be sent to their assigned tracepoints. - **Examples:** ``` # Scenario: All events from provider go to a default tracepoint From 16053b41837cbe9ea6d7b35d4a3d2f3f34a7c17c Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Fri, 15 Aug 2025 15:18:39 -0400 Subject: [PATCH 05/10] Address Feedback --- docs/core/diagnostics/dotnet-trace.md | 235 +++++++++++------- .../eventsource-collect-and-view-traces.md | 2 +- 2 files changed, 148 insertions(+), 89 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index bba276360ff3e..96a46cfbba03d 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -43,7 +43,8 @@ The `dotnet-trace` tool: * Is a cross-platform .NET Core tool. * Enables the collection of .NET Core traces of a running process without a native profiler. * Is built on [`EventPipe`](./eventpipe.md) of the .NET Core runtime. -* On Linux, provides additional integration with kernel user_events for native tracing tool compatibility. +* Delivers the same experience on Windows, Linux, and macOS. +* Offers an additional command on Linux to collect Linux perf events. ## Options @@ -126,7 +127,7 @@ dotnet-trace collect | ------------ | ------------------- | | `gc` | `0x1` | | `gchandle` | `0x2` | - | `fusion` | `0x4` | + | `assemblyloader` | `0x4` | | `loader` | `0x8` | | `jit` | `0x10` | | `ngen` | `0x20` | @@ -145,7 +146,7 @@ dotnet-trace collect | `gcheapdump` | `0x100000` | | `gcsampledobjectallocationhigh` | `0x200000` | | `gcheapsurvivalandmovement` | `0x400000` | - | `gcheapcollect` | `0x800000` | + | `managedheapcollect` | `0x800000` | | `gcheapandtypenames` | `0x1000000` | | `gcsampledobjectallocationlow` | `0x2000000` | | `perftrack` | `0x20000000` | @@ -159,7 +160,10 @@ dotnet-trace collect | `compilationdiagnostic` | `0x2000000000` | | `methoddiagnostic` | `0x4000000000` | | `typediagnostic` | `0x8000000000` | + | `jitinstrumentationdata` | `0x10000000000` | + | `profiler` | `0x20000000000` | | `waithandle` | `0x40000000000` | + | `allocationsampling` | `0x80000000000` | You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md). @@ -213,7 +217,9 @@ dotnet-trace collect | Profile | Description | |---------|-------------| - |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.| + |`dotnet-default`|Combination of `dotnet-thread-time` and `dotnet-clr-default`. This is the default profile used when no other provider/event/profile options are specified.| + |`dotnet-thread-time`|Useful for tracking .NET thread-time information. This is the default profile used when no other provider/event/profile options are specified.| + |`dotnet-clr-default`|Useful for tracking a multitude of .NET events.| |`gc-verbose`|Tracks GC collections and samples object allocations.| |`gc-collect`|Tracks GC collections only at very low overhead.| @@ -268,131 +274,184 @@ dotnet-trace collect ## dotnet-trace collect-linux -Collects diagnostic traces from .NET applications using Linux user_events as a transport layer. This command provides the same functionality as [`dotnet-trace collect`](#dotnet-trace-collect) but routes .NET runtime events through the Linux kernel's user_events subsystem before writing them to `.nettrace` files. +Collects diagnostic traces using perf_events, a Linux OS technology. `collect-linux` requires admin privileges to capture kernel- and user-mode events, and by default, captures events from all processes. -This transport approach enables automatic unification of user-space .NET events with kernel-space system events, since both are captured in the same kernel tracing infrastructure. Linux tools like `perf` and `ftrace` can monitor events in real-time while maintaining full compatibility with existing .NET profiling workflows. +This Linux-only command includes the same .NET events as [`dotnet-trace collect`](#dotnet-trace-collect), and it uses the kernel’s user_events mechanism to emit .NET events as perf events, enabling unification of user-space .NET events with kernel-space system events. ### Prerequisites - Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+) -- Appropriate permissions to access `/sys/kernel/tracing/user_events_data` +- Root permissions - .NET 10+ ### Synopsis ```dotnetcli dotnet-trace collect-linux - [--buffersize ] + [-h|--help] + + # Provider/Event Specification + [--providers ] [--clreventlevel ] [--clrevents ] + [--perf-event-tracepoints ] + [--profile ] + + # Trace Collection [--format ] - [-h|--help] + [-o|--output ] [--duration dd:hh:mm:ss] + + # .NET Process Target (Optional) [-n, --name ] - [--diagnostic-port] - [-o|--output ] [-p|--process-id ] - [--profile ] - [--providers ] - [-- ] (for target applications running .NET 10 or later) + [-- ] [--show-child-io] - [--resume-runtime] - [--stopping-event-provider-name ] - [--stopping-event-event-name ] - [--stopping-event-payload-filter ] - [--tracepoint-configs ] - [--kernel-events ] ``` ### Options -`dotnet-trace collect-linux` supports all the same options as [`dotnet-trace collect`](#dotnet-trace-collect), excluding `--dsrouter`, and additionally offers: +#### Provider/Event Specification Options -- **`--tracepoint-configs ` (required)** +- **`--providers `** - Defines the explicit mapping between EventPipe providers and kernel tracepoints. Each provider in `--providers` must have a corresponding entry in `--tracepoint-configs` + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profile. - **Format:** `ProviderName::` + This list of providers is in the form: - Where: - - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) - - `DefaultTracepointName`: Default tracepoint name for this provider (can be empty to require explicit assignment) - - `TracepointSets`: Semi-colon delimited `TracepointName=` - - `EventIds`: Plus-delimited event IDs to route to that tracepoint + - `Provider[,Provider]` + - `Provider` is in the form: `KnownProviderName[:Flags[:Level][:KeyValueArgs]]`. + - `KeyValueArgs` is in the form: `[key1=value1][;key2=value2]`. - > [!NOTE] - > All tracepoint names are automatically prefixed with the provider name to avoid collisions. For example, `gc_events` for the `Microsoft-Windows-DotNETRuntime` provider becomes `Microsoft_Windows_DotNETRuntime_gc_events`. + To learn more about some of the well-known providers in .NET, refer to [Well-known Event Providers](./well-known-event-providers.md). - **Examples:** - ``` - # Scenario: All events from provider go to a default tracepoint - --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" - # All enabled events from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_dotnet_runtime - - # Scenario: Split events by categories - --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3;jit_events=10+11+12" - # EventIDs 1, 2, and 3 will be written to Microsoft_Windows_DotNETRuntime_gc_events - # EventIDs 10, 11, and 12 will be written to Microsoft_Windows_DotNETRuntime_jit_events - - # Multiple providers (comma-separated) - --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3,MyCustomProvider:custom_events" - # EventIds 1, 2, and 3 from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_gc_events - # All enabled events from MyCustomProvider will be written to MyCustomProvider_custom_events - ``` +- **`--clreventlevel `** + + Verbosity of CLR events to be emitted. + The following table shows the available event levels. + + | String value | Numeric value | + | --------------- | :-----------: | + | `logalways` | `0` | + | `critical` | `1` | + | `error` | `2` | + | `warning` | `3` | + | `informational` | `4` | + | `verbose` | `5` | + +- **`--clrevents `** + + A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect-linux --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect-linux --clrevents gc+gchandle --clreventlevel informational`. The table below shows the list of available keywords: + + | Keyword String Alias | Keyword Hex Value | + | ------------ | ------------------- | + | `gc` | `0x1` | + | `gchandle` | `0x2` | + | `assemblyloader` | `0x4` | + | `loader` | `0x8` | + | `jit` | `0x10` | + | `ngen` | `0x20` | + | `startenumeration` | `0x40` | + | `endenumeration` | `0x80` | + | `security` | `0x400` | + | `appdomainresourcemanagement` | `0x800` | + | `jittracing` | `0x1000` | + | `interop` | `0x2000` | + | `contention` | `0x4000` | + | `exception` | `0x8000` | + | `threading` | `0x10000` | + | `jittedmethodiltonativemap` | `0x20000` | + | `overrideandsuppressngenevents` | `0x40000` | + | `type` | `0x80000` | + | `gcheapdump` | `0x100000` | + | `gcsampledobjectallocationhigh` | `0x200000` | + | `gcheapsurvivalandmovement` | `0x400000` | + | `managedheapcollect` | `0x800000` | + | `gcheapandtypenames` | `0x1000000` | + | `gcsampledobjectallocationlow` | `0x2000000` | + | `perftrack` | `0x20000000` | + | `stack` | `0x40000000` | + | `threadtransfer` | `0x80000000` | + | `debugger` | `0x100000000` | + | `monitoring` | `0x200000000` | + | `codesymbols` | `0x400000000` | + | `eventsource` | `0x800000000` | + | `compilation` | `0x1000000000` | + | `compilationdiagnostic` | `0x2000000000` | + | `methoddiagnostic` | `0x4000000000` | + | `typediagnostic` | `0x8000000000` | + | `jitinstrumentationdata` | `0x10000000000` | + | `profiler` | `0x20000000000` | + | `waithandle` | `0x40000000000` | + | `allocationsampling` | `0x80000000000` | + + You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md). + +- **`--perf-event-tracepoints `** + + A comma-separated list of kernel perf event tracepoints to include in the trace. Available kernel events can be found under tracefs, which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available tracepoints or through the `events/` subdirectory for categorized events. + + Example: `--perf-event-tracepoints syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup` + +- **`--profile `** + + A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available: + + | Profile | Description | + |---------|-------------| + |`dotnet-default`|Combination of `dotnet-thread-time` and `dotnet-clr-default`. This is the default profile used when no other provider/event/profile options are specified.| + |`dotnet-thread-time`|Useful for tracking .NET thread-time information. This is the default profile used when no other provider/event/profile options are specified.| + |`dotnet-clr-default`|Useful for tracking a multitude of .NET events.| + |`gc-verbose`|Tracks GC collections and samples object allocations.| + |`gc-collect`|Tracks GC collections only at very low overhead.| + |`cpu-sampling`|Useful for tracking CPU usage.| -- **`--kernel-events ` (optional)** +#### Trace Collection Options - A comma-separated list of kernel event categories to include in the trace. These events are automatically grouped into kernel-named tracepoints. Available categories include: +- **`--format {Chromium|NetTrace|Speedscope}`** - | Category | Description | Linux Tracepoints | - |----------|-------------|-------------------| - | `syscalls` | System call entry/exit events | `syscalls:sys_enter_*`, `syscalls:sys_exit_*` | - | `sched` | Process scheduling events | `sched:sched_switch`, `sched:sched_wakeup` | - | `net` | Network-related events | `net:netif_rx`, `net:net_dev_xmit` | - | `fs` | Filesystem I/O events | `ext4:*`, `vfs:*` | - | `mm` | Memory management events | `kmem:*`, `vmscan:*` | + Sets the output format for the trace file conversion. The default is `NetTrace`. - These events correspond to Linux kernel tracepoints documented in the [Linux kernel tracing documentation](https://www.kernel.org/doc/html/latest/trace/index.html). For more details on available tracepoints, see [ftrace](https://www.kernel.org/doc/html/latest/trace/ftrace.html) and [tracepoints](https://www.kernel.org/doc/html/latest/trace/tracepoints.html). +- **`-o|--output `** - Example: `--kernel-events syscalls,sched,net` + The output path for the collected trace data. If not specified it defaults to `__.nettrace`, for example, `myapp_20210315_111514.nettrace``. -### Linux Integration +- **`--duration `** -**Tracepoint Configuration Requirements:** + The time for the trace to run. Use the `dd:hh:mm:ss` format. For example `00:00:00:05` will run it for 5 seconds. -- **Mandatory Mapping**: Every provider must be explicitly mapped to at least a default tracepoint and/or exclusive tracepoint sets via `--tracepoint-configs` -- **Tracepoint Isolation**: Each tracepoint can only receive events from one provider -- **Event Routing**: Different event IDs within a provider can be routed to different tracepoints for granular control -- **Automatic Prefixing**: All tracepoint names are prefixed with the provider name to avoid collisions +#### .NET Process Target Options -**Kernel Integration Points:** +> ![NOTE] +> The default behavior for collect-linux is to collect .NET events and perf event tracepoints from ALL discoverable processes. -The kernel tracepoints can be accessed through standard Linux tracing interfaces: +- **`-n, --name `** -- **ftrace**: `/sys/kernel/tracing/events/user_events/` -- **perf**: Use `perf list user_events*` to see available events -- **System monitoring tools**: Any tool that can consume Linux tracepoints + The name of the process to collect the trace from. -### Examples + > [!NOTE] + > By default, `/tmp` will be searched to discover .NET processes diagnostics ports. Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. -```dotnetcli -# All runtime events to one tracepoint -dotnet-trace collect-linux --process-id 1234 \ - --providers Microsoft-Windows-DotNETRuntime:0x8000:5 \ - --kernel-events syscalls,sched \ - --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" - -# Split runtime events by category -dotnet-trace collect-linux --process-id 1234 \ - --providers Microsoft-Windows-DotNETRuntime:0x8001:5 \ - --kernel-events syscalls,sched,net,fs \ - --tracepoint-configs "Microsoft-Windows-DotNETRuntime::exception_events=80;gc_events=1+2" - -# Multiple providers -dotnet-trace collect-linux --process-id 1234 \ - --providers "Microsoft-Windows-DotNETRuntime:0x8001:5,MyCustomProvider:0xFFFFFFFF:5" \ - --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime,MyCustomProvider:custom_events" -``` +- **`-p|--process-id `** + + The process ID to collect the trace from. + + > [!NOTE] + > By default, `/tmp` will be searched to discover .NET processes diagnostics ports. Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. + +- **`-- `** + + After the collection configuration parameters, the user can append `--` followed by a command to start a .NET application. This may be helpful when diagnosing issues that happen early in the process, such as startup performance issue or assembly loader and binder errors. + +- **`--show-child-io`** + + Only applicable when `-- ` is used. Shows the input and output streams of a launched child process in the current console. + +> [!NOTE] + +> - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace. + +> - To collect a trace using `dotnet-trace collect-linux`, it needs to be run with root permissions (`CAP_PERFMON`/`CAP_SYS_ADMIN`). Otherwise, the tool will fail to collect events. ## dotnet-trace convert diff --git a/docs/core/diagnostics/eventsource-collect-and-view-traces.md b/docs/core/diagnostics/eventsource-collect-and-view-traces.md index fd6b2a32d9dc6..14e7dcbc6975a 100644 --- a/docs/core/diagnostics/eventsource-collect-and-view-traces.md +++ b/docs/core/diagnostics/eventsource-collect-and-view-traces.md @@ -214,7 +214,7 @@ into other formats, such as Chromium or [Speedscope](https://www.speedscope.app/ Trace completed. ``` - dotnet-trace uses a comma-delimited variant of the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in + dotnet-trace uses the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in the `--providers` argument. For more options on how to take traces using dotnet-trace, see the [dotnet-trace docs](./dotnet-trace.md#collect-a-trace-with-dotnet-trace). From 26adb7b7a70902ddcedd79b7b484cfc513fd438e Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Fri, 29 Aug 2025 18:34:36 -0400 Subject: [PATCH 06/10] [Diagnostics][dotnet-trace] Revamp profiles --- docs/core/diagnostics/dotnet-trace.md | 65 ++++++++++++++++++--------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index 96a46cfbba03d..41cd6a6015a23 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -86,7 +86,7 @@ dotnet-trace collect [--diagnostic-port] [-o|--output ] [-p|--process-id ] - [--profile ] + [--profiles ] [--providers ] [-- ] (for target applications running .NET 5 or later) [--show-child-io] @@ -211,21 +211,24 @@ dotnet-trace collect > [!NOTE] > On Linux and macOS, using this option requires the target application and `dotnet-trace` to share the same `TMPDIR` environment variable. Otherwise, the command will time out. -- **`--profile `** +- **`--profiles `** - A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available: + A comma-separated list of named, pre-defined set of provider configurations for common tracing scenarios. The union of profiles and providers will be in effect. + + Default behavior (when `--profiles`, `--providers`, and `--clrevents` are omitted): dotnet-trace enables a useful, low-overhead composition: `dotnet-common` + `dotnet-thread-time`. + + Available profiles: | Profile | Description | |---------|-------------| - |`dotnet-default`|Combination of `dotnet-thread-time` and `dotnet-clr-default`. This is the default profile used when no other provider/event/profile options are specified.| - |`dotnet-thread-time`|Useful for tracking .NET thread-time information. This is the default profile used when no other provider/event/profile options are specified.| - |`dotnet-clr-default`|Useful for tracking a multitude of .NET events.| + |`dotnet-common`|Lightweight .NET runtime diagnostics (GC, JIT, loader/assembly loader, threading, contention). Designed to stay low overhead.| + |`dotnet-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.| |`gc-verbose`|Tracks GC collections and samples object allocations.| |`gc-collect`|Tracks GC collections only at very low overhead.| - **`--providers `** - A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profile. + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profiles `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profiles. This list of providers is in the form: @@ -278,6 +281,19 @@ Collects diagnostic traces using perf_events, a Linux OS technology. `collect-li This Linux-only command includes the same .NET events as [`dotnet-trace collect`](#dotnet-trace-collect), and it uses the kernel’s user_events mechanism to emit .NET events as perf events, enabling unification of user-space .NET events with kernel-space system events. +### Default collection behavior + +When `--profiles`, `--clrevents`, and `--providers` aren’t specified, `collect-linux` enables the default `profile` providing the comprehensive composition: + +- `dotnet-common` — lightweight .NET runtime diagnostics. +- `dotnet-thread-time` — runtime thread stack sampling. +- `kernel-cpu` — kernel CPU sampling (perf-based) via `Universal.Events/cpu`. +- `kernel-cswitch` — kernel context switches via `Universal.Events/cswitch` for on/off-CPU analysis. + +By default, a machine-wide trace will be collected. .NET Processes are discovered through their diagnostics ports, which are located under the `TMPDIR` environment variable when set and otherwise under `/tmp`. `dotnet-trace` will need to share the same `TMPDIR` value as .NET Processes of interest, otherwise events will not be collected from those processes. + +If collecting events from all .NET Processes is undesired, `-n, --name ` or `-p|--process-id ` can be used to specify a particular process. **NOTE**: Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. + ### Prerequisites - Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+) @@ -295,7 +311,7 @@ dotnet-trace collect-linux [--clreventlevel ] [--clrevents ] [--perf-event-tracepoints ] - [--profile ] + [--profiles ] # Trace Collection [--format ] @@ -315,7 +331,7 @@ dotnet-trace collect-linux - **`--providers `** - A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profile. + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profiles `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the specified profiles. This list of providers is in the form: @@ -393,18 +409,24 @@ dotnet-trace collect-linux Example: `--perf-event-tracepoints syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup` -- **`--profile `** +- **`--profiles `** + + A named, pre-defined set of provider configurations for common tracing scenarios. You can specify multiple profiles as a comma-separated list. When multiple profiles are specified, the providers and settings are combined (union), and duplicates are ignored. - A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available: + Available profiles: | Profile | Description | |---------|-------------| - |`dotnet-default`|Combination of `dotnet-thread-time` and `dotnet-clr-default`. This is the default profile used when no other provider/event/profile options are specified.| - |`dotnet-thread-time`|Useful for tracking .NET thread-time information. This is the default profile used when no other provider/event/profile options are specified.| - |`dotnet-clr-default`|Useful for tracking a multitude of .NET events.| + |`dotnet-common`|Lightweight .NET runtime diagnostics (GC, JIT, loader/assembly loader, threading, contention). Designed to stay low overhead.| + |`dotnet-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.| + |`kernel-cpu`|Kernel CPU sampling (perf-based), emitted as `Universal.Events/cpu`, for precise on-CPU attribution.| + |`kernel-cswitch`|Kernel thread context switches, emitted as `Universal.Events/cswitch`, for on/off-CPU and scheduler analysis.| + |`kernel-precise-thread-time`|Combines `kernel-cpu` + `kernel-cswitch` for a precise thread-time view using kernel events only.| |`gc-verbose`|Tracks GC collections and samples object allocations.| |`gc-collect`|Tracks GC collections only at very low overhead.| - |`cpu-sampling`|Useful for tracking CPU usage.| + + > [!NOTE] + > The former default `cpu-sampling` profile is now named as `dotnet-thread-time`. #### Trace Collection Options @@ -422,22 +444,21 @@ dotnet-trace collect-linux #### .NET Process Target Options -> ![NOTE] -> The default behavior for collect-linux is to collect .NET events and perf event tracepoints from ALL discoverable processes. +See [Default collection behavior](#default-collection-behavior) - **`-n, --name `** The name of the process to collect the trace from. > [!NOTE] - > By default, `/tmp` will be searched to discover .NET processes diagnostics ports. Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. + > Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. - **`-p|--process-id `** The process ID to collect the trace from. > [!NOTE] - > By default, `/tmp` will be searched to discover .NET processes diagnostics ports. Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. + > Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. - **`-- `** @@ -600,11 +621,11 @@ dotnet-trace collect -- hello.exe arg1 arg2 The preceding command generates output similar to the following: ```output -No profile or providers specified, defaulting to trace profile 'cpu-sampling' +No profile or providers specified. Using default composition: dotnet-common + dotnet-thread-time Provider Name Keywords Level Enabled By -Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profile -Microsoft-Windows-DotNETRuntime 0x00000014C14FCCBD Informational(4) --profile +Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profiles +Microsoft-Windows-DotNETRuntime 0x00000014C14FCCBD Informational(4) --profiles Process : E:\temp\gcperfsim\bin\Debug\net5.0\gcperfsim.exe Output File : E:\temp\gcperfsim\trace.nettrace From 0d09d90866beae1b1a214cfb2c156fe6312993d0 Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Tue, 9 Sep 2025 17:14:58 -0400 Subject: [PATCH 07/10] Address profiles feedback --- docs/core/diagnostics/dotnet-trace.md | 64 +++++++++------------------ 1 file changed, 22 insertions(+), 42 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index 41cd6a6015a23..bac6c72a13ace 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -86,7 +86,7 @@ dotnet-trace collect [--diagnostic-port] [-o|--output ] [-p|--process-id ] - [--profiles ] + [--profile ] [--providers ] [-- ] (for target applications running .NET 5 or later) [--show-child-io] @@ -211,24 +211,27 @@ dotnet-trace collect > [!NOTE] > On Linux and macOS, using this option requires the target application and `dotnet-trace` to share the same `TMPDIR` environment variable. Otherwise, the command will time out. -- **`--profiles `** +- **`--profile `** A comma-separated list of named, pre-defined set of provider configurations for common tracing scenarios. The union of profiles and providers will be in effect. - Default behavior (when `--profiles`, `--providers`, and `--clrevents` are omitted): dotnet-trace enables a useful, low-overhead composition: `dotnet-common` + `dotnet-thread-time`. + Default behavior (when `--profile`, `--providers`, and `--clrevents` are omitted): dotnet-trace enables a useful, low-overhead composition: `dotnet-common` + `dotnet-thread-time`. Available profiles: | Profile | Description | |---------|-------------| - |`dotnet-common`|Lightweight .NET runtime diagnostics (GC, JIT, loader/assembly loader, threading, contention). Designed to stay low overhead.| - |`dotnet-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.| + |`dotnet-common`|Lightweight .NET runtime diagnostics designed to stay low overhead. Includes:
  • GC
  • AssemblyLoader
  • Loader
  • Jit
  • Exception
  • Threading
  • JittedMethodILToNativeMap
  • Compilation
Equivalent to `--providers "Microsoft-Windows-DotNETRuntime:0x100003801D:4"`.| + |`dotnet-sampled-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.| |`gc-verbose`|Tracks GC collections and samples object allocations.| |`gc-collect`|Tracks GC collections only at very low overhead.| + > [!NOTE] + > The former default `cpu-sampling` profile is now `--profile dotnet-sampled-thread-time` + `--providers "Microsoft-Windows-DotNETRuntime:0x14C14FCCBD:4"`. + - **`--providers `** - A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profiles `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profiles. + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profiles. This list of providers is in the form: @@ -283,16 +286,14 @@ This Linux-only command includes the same .NET events as [`dotnet-trace collect` ### Default collection behavior -When `--profiles`, `--clrevents`, and `--providers` aren’t specified, `collect-linux` enables the default `profile` providing the comprehensive composition: +When `--profile`, `--clrevents`, and `--providers` aren’t specified, `collect-linux` enables the default `profile` providing the comprehensive composition: - `dotnet-common` — lightweight .NET runtime diagnostics. -- `dotnet-thread-time` — runtime thread stack sampling. - `kernel-cpu` — kernel CPU sampling (perf-based) via `Universal.Events/cpu`. -- `kernel-cswitch` — kernel context switches via `Universal.Events/cswitch` for on/off-CPU analysis. -By default, a machine-wide trace will be collected. .NET Processes are discovered through their diagnostics ports, which are located under the `TMPDIR` environment variable when set and otherwise under `/tmp`. `dotnet-trace` will need to share the same `TMPDIR` value as .NET Processes of interest, otherwise events will not be collected from those processes. +By default, a machine-wide trace will be collected. .NET Processes are discovered through their diagnostics ports, which are located under the `TMPDIR` environment variable when set and otherwise under `/tmp`. -If collecting events from all .NET Processes is undesired, `-n, --name ` or `-p|--process-id ` can be used to specify a particular process. **NOTE**: Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. +If collecting events from all .NET Processes is undesired, `-n, --name ` or `-p|--process-id ` can be used to specify a particular process. ### Prerequisites @@ -310,8 +311,8 @@ dotnet-trace collect-linux [--providers ] [--clreventlevel ] [--clrevents ] - [--perf-event-tracepoints ] - [--profiles ] + [--perf-events ] + [--profile ] # Trace Collection [--format ] @@ -331,7 +332,7 @@ dotnet-trace collect-linux - **`--providers `** - A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profiles `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the specified profiles. + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the specified profiles. This list of providers is in the form: @@ -403,13 +404,13 @@ dotnet-trace collect-linux You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md). -- **`--perf-event-tracepoints `** +- **`--perf-events `** - A comma-separated list of kernel perf event tracepoints to include in the trace. Available kernel events can be found under tracefs, which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available tracepoints or through the `events/` subdirectory for categorized events. + A comma-separated list of kernel perf events to include in the trace. Available kernel events can be found under tracefs, which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available events or through the `events/` subdirectory for categorized events. - Example: `--perf-event-tracepoints syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup` + Example: `--perf-events syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup` -- **`--profiles `** +- **`--profile `** A named, pre-defined set of provider configurations for common tracing scenarios. You can specify multiple profiles as a comma-separated list. When multiple profiles are specified, the providers and settings are combined (union), and duplicates are ignored. @@ -417,17 +418,12 @@ dotnet-trace collect-linux | Profile | Description | |---------|-------------| - |`dotnet-common`|Lightweight .NET runtime diagnostics (GC, JIT, loader/assembly loader, threading, contention). Designed to stay low overhead.| - |`dotnet-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.| + |`dotnet-common`|Lightweight .NET runtime diagnostics designed to stay low overhead. Includes:
  • GC
  • AssemblyLoader
  • Loader
  • Jit
  • Exception
  • Threading
  • JittedMethodILToNativeMap
  • Compilation
Equivalent to `--providers "Microsoft-Windows-DotNETRuntime:0x100003801D:4"`.| |`kernel-cpu`|Kernel CPU sampling (perf-based), emitted as `Universal.Events/cpu`, for precise on-CPU attribution.| |`kernel-cswitch`|Kernel thread context switches, emitted as `Universal.Events/cswitch`, for on/off-CPU and scheduler analysis.| - |`kernel-precise-thread-time`|Combines `kernel-cpu` + `kernel-cswitch` for a precise thread-time view using kernel events only.| |`gc-verbose`|Tracks GC collections and samples object allocations.| |`gc-collect`|Tracks GC collections only at very low overhead.| - > [!NOTE] - > The former default `cpu-sampling` profile is now named as `dotnet-thread-time`. - #### Trace Collection Options - **`--format {Chromium|NetTrace|Speedscope}`** @@ -450,28 +446,12 @@ See [Default collection behavior](#default-collection-behavior) The name of the process to collect the trace from. - > [!NOTE] - > Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. - - **`-p|--process-id `** The process ID to collect the trace from. - > [!NOTE] - > Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. - -- **`-- `** - - After the collection configuration parameters, the user can append `--` followed by a command to start a .NET application. This may be helpful when diagnosing issues that happen early in the process, such as startup performance issue or assembly loader and binder errors. - -- **`--show-child-io`** - - Only applicable when `-- ` is used. Shows the input and output streams of a launched child process in the current console. - > [!NOTE] -> - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace. - > - To collect a trace using `dotnet-trace collect-linux`, it needs to be run with root permissions (`CAP_PERFMON`/`CAP_SYS_ADMIN`). Otherwise, the tool will fail to collect events. ## dotnet-trace convert @@ -624,8 +604,8 @@ The preceding command generates output similar to the following: No profile or providers specified. Using default composition: dotnet-common + dotnet-thread-time Provider Name Keywords Level Enabled By -Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profiles -Microsoft-Windows-DotNETRuntime 0x00000014C14FCCBD Informational(4) --profiles +Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profile +Microsoft-Windows-DotNETRuntime 0x00000014C14FCCBD Informational(4) --profile Process : E:\temp\gcperfsim\bin\Debug\net5.0\gcperfsim.exe Output File : E:\temp\gcperfsim\trace.nettrace From 8118b867be16c50909ef75d3893110712d487c5c Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Mon, 22 Sep 2025 14:37:33 -0400 Subject: [PATCH 08/10] Refine descriptions and limitations --- docs/core/diagnostics/dotnet-trace.md | 41 +++++++++++++-------------- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index bac6c72a13ace..817c3d58ce634 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -107,7 +107,7 @@ dotnet-trace collect - **`--clreventlevel `** - Verbosity of CLR events to be emitted. + Verbosity of CLR events to be emitted. This option only applies when `--clrevents` is specified and not overridden by `--profile` or `--providers`. The following table shows the available event levels. | String value | Numeric value | @@ -121,7 +121,7 @@ dotnet-trace collect - **`--clrevents `** - A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect --clrevents gc+gchandle --clreventlevel informational`. The table below shows the list of available keywords: + A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect --clrevents gc+gchandle --clreventlevel informational`. If the CLR runtime provider `Microsoft-Windows-DotNETRuntime` is also enabled through `--providers` or `--profile`, this option will be ignored. The table below shows the list of available keywords: | Keyword String Alias | Keyword Hex Value | | ------------ | ------------------- | @@ -213,9 +213,9 @@ dotnet-trace collect - **`--profile `** - A comma-separated list of named, pre-defined set of provider configurations for common tracing scenarios. The union of profiles and providers will be in effect. + A comma-separated list of named, pre-defined set of provider configurations for common tracing scenarios. Providers configured through `--providers` will override the profile's configuration. Similarly, if any profile configures the CLR runtime provider, it will override any configurations prescribed through `--clrevents`. - Default behavior (when `--profile`, `--providers`, and `--clrevents` are omitted): dotnet-trace enables a useful, low-overhead composition: `dotnet-common` + `dotnet-thread-time`. + Default behavior (when `--profile`, `--providers`, and `--clrevents` are omitted): dotnet-trace enables a useful, low-overhead composition: `dotnet-common` + `dotnet-sampled-thread-time`. Available profiles: @@ -225,13 +225,14 @@ dotnet-trace collect |`dotnet-sampled-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.| |`gc-verbose`|Tracks GC collections and samples object allocations.| |`gc-collect`|Tracks GC collections only at very low overhead.| + |`database`|Captures ADO.NET and Entity Framework database commands.| > [!NOTE] > The former default `cpu-sampling` profile is now `--profile dotnet-sampled-thread-time` + `--providers "Microsoft-Windows-DotNETRuntime:0x14C14FCCBD:4"`. - **`--providers `** - A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profiles. + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from `--profile` and `--clrevents`. This list of providers is in the form: @@ -286,10 +287,10 @@ This Linux-only command includes the same .NET events as [`dotnet-trace collect` ### Default collection behavior -When `--profile`, `--clrevents`, and `--providers` aren’t specified, `collect-linux` enables the default `profile` providing the comprehensive composition: +When `--providers`, `--profile`, `--clrevents`, and `--perf-events` aren’t specified, `collect-linux` enables the default `profile` providing the comprehensive composition: - `dotnet-common` — lightweight .NET runtime diagnostics. -- `kernel-cpu` — kernel CPU sampling (perf-based) via `Universal.Events/cpu`. +- `cpu-sampling` — kernel CPU sampling (perf-based) via `Universal.Events/cpu`. By default, a machine-wide trace will be collected. .NET Processes are discovered through their diagnostics ports, which are located under the `TMPDIR` environment variable when set and otherwise under `/tmp`. @@ -315,15 +316,12 @@ dotnet-trace collect-linux [--profile ] # Trace Collection - [--format ] [-o|--output ] [--duration dd:hh:mm:ss] # .NET Process Target (Optional) [-n, --name ] [-p|--process-id ] - [-- ] - [--show-child-io] ``` ### Options @@ -332,7 +330,7 @@ dotnet-trace collect-linux - **`--providers `** - A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the specified profiles. + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from `--profile` and `--clrevents`. This list of providers is in the form: @@ -344,7 +342,7 @@ dotnet-trace collect-linux - **`--clreventlevel `** - Verbosity of CLR events to be emitted. + Verbosity of CLR events to be emitted. This option only applies when `--clrevents` is specified and not overridden by `--profile` or `--providers`. The following table shows the available event levels. | String value | Numeric value | @@ -358,7 +356,7 @@ dotnet-trace collect-linux - **`--clrevents `** - A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect-linux --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect-linux --clrevents gc+gchandle --clreventlevel informational`. The table below shows the list of available keywords: + A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect-linux --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect-linux --clrevents gc+gchandle --clreventlevel informational`. If the CLR runtime provider `Microsoft-Windows-DotNETRuntime` is also enabled through `--providers` or `--profile`, this option will be ignored. The table below shows the list of available keywords: | Keyword String Alias | Keyword Hex Value | | ------------ | ------------------- | @@ -406,33 +404,32 @@ dotnet-trace collect-linux - **`--perf-events `** - A comma-separated list of kernel perf events to include in the trace. Available kernel events can be found under tracefs, which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available events or through the `events/` subdirectory for categorized events. + A comma-separated list of perf events to include in the trace. Available events can be found under tracefs, which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available events or through the `events/` subdirectory for categorized events. Example: `--perf-events syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup` - **`--profile `** - A named, pre-defined set of provider configurations for common tracing scenarios. You can specify multiple profiles as a comma-separated list. When multiple profiles are specified, the providers and settings are combined (union), and duplicates are ignored. + A comma-separated list of named, pre-defined set of provider configurations for common tracing scenarios. Providers configured through `--providers` will override the profile's configuration. Similarly, if any profile configures the CLR runtime provider, it will override any configurations prescribed through `--clrevents`. + + Default behavior (when `--profile`, `--providers`, `--clrevents`, and `--perf-events` are omitted): dotnet-trace enables a useful, low-overhead composition: `dotnet-common` + `cpu-sampling`. Available profiles: | Profile | Description | |---------|-------------| |`dotnet-common`|Lightweight .NET runtime diagnostics designed to stay low overhead. Includes:
  • GC
  • AssemblyLoader
  • Loader
  • Jit
  • Exception
  • Threading
  • JittedMethodILToNativeMap
  • Compilation
Equivalent to `--providers "Microsoft-Windows-DotNETRuntime:0x100003801D:4"`.| - |`kernel-cpu`|Kernel CPU sampling (perf-based), emitted as `Universal.Events/cpu`, for precise on-CPU attribution.| - |`kernel-cswitch`|Kernel thread context switches, emitted as `Universal.Events/cswitch`, for on/off-CPU and scheduler analysis.| + |`cpu-sampling`|Kernel CPU sampling (perf-based), emitted as `Universal.Events/cpu`, for precise on-CPU attribution.| + |`thread-time`|Kernel thread context switches, emitted as `Universal.Events/cswitch`, for on/off-CPU and scheduler analysis.| |`gc-verbose`|Tracks GC collections and samples object allocations.| |`gc-collect`|Tracks GC collections only at very low overhead.| + |`database`|Captures ADO.NET and Entity Framework database commands.| #### Trace Collection Options -- **`--format {Chromium|NetTrace|Speedscope}`** - - Sets the output format for the trace file conversion. The default is `NetTrace`. - - **`-o|--output `** - The output path for the collected trace data. If not specified it defaults to `__.nettrace`, for example, `myapp_20210315_111514.nettrace``. + The output path for the collected trace data. If not specified, it defaults to `trace__.nettrace` for the default machine-wide trace and to `__.nettrace` for a process-specific trace (`--name` or `--process-id`) - **`--duration `** From 72027e3bbb03e5a7b498c7f073d67f44ed748271 Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Tue, 7 Oct 2025 11:12:31 -0400 Subject: [PATCH 09/10] [DotnetTrace][CollectLinux] Cut target process options --- docs/core/diagnostics/dotnet-trace.md | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index 817c3d58ce634..daa853f605206 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -292,9 +292,7 @@ When `--providers`, `--profile`, `--clrevents`, and `--perf-events` aren’t spe - `dotnet-common` — lightweight .NET runtime diagnostics. - `cpu-sampling` — kernel CPU sampling (perf-based) via `Universal.Events/cpu`. -By default, a machine-wide trace will be collected. .NET Processes are discovered through their diagnostics ports, which are located under the `TMPDIR` environment variable when set and otherwise under `/tmp`. - -If collecting events from all .NET Processes is undesired, `-n, --name ` or `-p|--process-id ` can be used to specify a particular process. +A machine-wide trace will be collected. .NET Processes are discovered through their diagnostics ports, which are located under the `TMPDIR` environment variable when set and otherwise under `/tmp`. ### Prerequisites @@ -318,10 +316,6 @@ dotnet-trace collect-linux # Trace Collection [-o|--output ] [--duration dd:hh:mm:ss] - - # .NET Process Target (Optional) - [-n, --name ] - [-p|--process-id ] ``` ### Options @@ -435,18 +429,6 @@ dotnet-trace collect-linux The time for the trace to run. Use the `dd:hh:mm:ss` format. For example `00:00:00:05` will run it for 5 seconds. -#### .NET Process Target Options - -See [Default collection behavior](#default-collection-behavior) - -- **`-n, --name `** - - The name of the process to collect the trace from. - -- **`-p|--process-id `** - - The process ID to collect the trace from. - > [!NOTE] > - To collect a trace using `dotnet-trace collect-linux`, it needs to be run with root permissions (`CAP_PERFMON`/`CAP_SYS_ADMIN`). Otherwise, the tool will fail to collect events. From 54d662b1d16f54a4f50307edb2ce11debae8b024 Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Tue, 7 Oct 2025 12:31:11 -0400 Subject: [PATCH 10/10] Add collect-linux example --- docs/core/diagnostics/dotnet-trace.md | 49 ++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 9 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index daa853f605206..78ec337ea2423 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -541,7 +541,7 @@ Output the parameters of each method in full. If not specified, parameters will ## Collect a trace with dotnet-trace -To collect traces using `dotnet-trace`: +To collect traces using `dotnet-trace collect`: - Get the process identifier (PID) of the .NET Core application to collect traces from. @@ -558,14 +558,22 @@ To collect traces using `dotnet-trace`: The preceding command generates output similar to the following: ```output - Press to exit... - Connecting to process: /dotnet.exe - Collecting to file: /trace.nettrace - Session Id: - Recording trace 721.025 (KB) + No profile or providers specified, defaulting to trace profiles 'dotnet-common' + 'dotnet-sampled-thread-time'. + + Provider Name Keywords Level Enabled By + Microsoft-Windows-DotNETRuntime 0x000000100003801D Informational(4) --profile + Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profile + + Process : + Output File : _20251007_154557.nettrace + [00:00:00:02] Recording trace 178.172 (KB) + Press or to exit... + Stopping the trace. This may take several minutes depending on the application being traced. + + Trace completed. ``` -- Stop collection by pressing the `` key. `dotnet-trace` will finish logging events to the *trace.nettrace* file. +- Stop collection by pressing the `` key. `dotnet-trace` will finish logging events to the `.nettrace` file. ## Launch a child application and collect a trace from its startup using dotnet-trace @@ -580,11 +588,11 @@ dotnet-trace collect -- hello.exe arg1 arg2 The preceding command generates output similar to the following: ```output -No profile or providers specified. Using default composition: dotnet-common + dotnet-thread-time +No profile or providers specified, defaulting to trace profiles 'dotnet-common' + 'dotnet-sampled-thread-time'. Provider Name Keywords Level Enabled By +Microsoft-Windows-DotNETRuntime 0x000000100003801D Informational(4) --profile Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profile -Microsoft-Windows-DotNETRuntime 0x00000014C14FCCBD Informational(4) --profile Process : E:\temp\gcperfsim\bin\Debug\net5.0\gcperfsim.exe Output File : E:\temp\gcperfsim\trace.nettrace @@ -640,6 +648,29 @@ However, when you want to gain a finer control over the lifetime of the app bein > [!IMPORTANT] > Launching your app with `dotnet run` can be problematic because the dotnet CLI may spawn many child processes that are not your app and they can connect to `dotnet-trace` before your app, leaving your app to be suspended at run time. It is recommended you directly use a self-contained version of the app or use `dotnet exec` to launch the application. +## (Linux-only) Collect a trace with Linux perf events using dotnet-trace + +To collect traces using `dotnet-trace collect-linux`: + + ```output + $ sudo dotnet-trace collect-linux + No providers, profiles, ClrEvents, or PerfEvents were specified, defaulting to trace profiles 'dotnet-common' + 'cpu-sampling'. + + Provider Name Keywords Level Enabled By + Microsoft-Windows-DotNETRuntime 0x000000100003801D Informational(4) --profile + + Linux Events Enabled By + cpu-sampling --profile + + [00:00:00:04] Recording trace. + Press or to exit... + + Recording stopped. + Resolving symbols. + Finished recording trace. + Trace written to trace_20251007_160232.nettrace + ``` + ## View the trace captured from dotnet-trace On Windows, you can view *.nettrace* files in [Visual Studio](/visualstudio/profiling/beginners-guide-to-performance-profiling?#step-2-analyze-cpu-usage-data) or [PerfView](https://github.com/microsoft/perfview) for analysis.