Announcing TraceProcessor Preview 0.1.0
Process ETW traces in .NET.
Background
Event Tracing for Windows (ETW) is a powerful trace collection system built-in to the Windows operating system. Windows has deep integration with ETW, including data on system behavior all the way down to the kernel for events like context switches, memory allocation, process create and exit, and many more. The system-wide data available from ETW makes it a good fit for end-to-end performance analysis or other questions that require looking at the interaction between by many components throughout the system.
Unlike text logging, ETW provides structured events designed for automated data processing. Microsoft has built powerful tools on top of these structured events, including Windows Performance Analyzer (WPA), which provides a graphical interface for visualizing and exploring the trace data captured in a ETW trace file (.etl).
Inside Microsoft, we heavily use ETW traces to measure the performance of new builds of Windows. Given the volume of data produced the Windows engineering system, automated analysis is essential. For our automated trace analysis, we heavily use C# and .NET, so we created a package that provides a .NET API for accessing many kinds of ETW trace data. This technology is also used inside Windows Performance Analyzer to power several of its tables. We are happy to release a preview of this package that we use to analyze Windows so that you can use it to analyze your own applications and systems as well.
Getting Started
This package is available from NuGet with the following package ID:
Microsoft.Windows.EventTracing.Processing.All
After installing this package, the following console application lists all the process command lines in a trace:
using Microsoft.Windows.EventTracing;
using Microsoft.Windows.EventTracing.Processes;
using System;
class Program
{
static void Main(string[] args)
{
if (args.Length != 1)
{
Console.Error.WriteLine("Usage: <trace.etl>");
return;
}
using (ITraceProcessor trace = TraceProcessor.Create(args[0]))
{
IPendingResult<IProcessDataSource> pendingProcessData = trace.UseProcesses();
trace.Process();
IProcessDataSource processData = pendingProcessData.Result;
foreach (IProcess process in processData.Processes)
{
Console.WriteLine(process.CommandLine);
}
}
}
}
The core interface is ITraceProcessor, and using this interface follows a pattern of first, tell the processor what data you want to use from a trace; second, process the trace; and finally, access the results. Telling the processor what kinds of data you want up front means you do not need to spend time processing large volumes of all possible kinds of trace data. Instead, TraceProcessor just does the work needed to provide the specific kinds of data you request.
Data Sources
Process information is just one of many kinds of data that can be stored in an ETW trace. Note that which data is in an .etl file depends on what providers were enabled when the trace was captured. The following list shows the kinds of trace data available from TraceProcessor:
Code |
Description |
Related WPA Items |
| trace.UseClassicEvents() | Provides classic ETW events from a trace, which do not include schema information. | Generic Events table (when Event Type is Classic or WPP) |
| trace.UseConnectedStandbyData() | Provides data from a trace about the system entering and exiting connected standby. | CS Summary table |
| trace.UseCpuIdleStates() | Provides data from a trace about CPU C-states. | CPU Idle States table (when Type is Actual) |
| trace.UseCpuSamplingData() | Provides data from a trace about CPU usage based on periodic sampling of the instruction pointer. | CPU Usage (Sampled) table |
| trace.UseCpuSchedulingData() | Provides data from a trace about CPU thread scheduling, including context switches and ready thread events. | CPU Usage (Precise) table |
| trace.UseDevicePowerData() | Provides data from a trace about device D-states. | Device DState table |
| trace.UseDirectXData() | Provides data from a trace about DirectX activity. | GPU Utilization table |
| traceUseDiskIOData() | Provides data from a trace about Disk I/O activity. | Disk Usage table |
| trace.UseEnergyEstimationData() | Provides data from a trace about estimated per-process energy usage from Energy Estimation Engine. | Energy Estimation Engine Summary (by Process) table |
| trace.UseEnergyMeterData() | Provides data from a trace about measured energy usage from Energy Meter Interface (EMI). | Energy Estimation Engine (by Emi) table |
| trace.UseFileIOData() | Provides data from a trace about File I/O activity. | File I/O table |
| trace.UseGenericEvents() | Provides manifested and TraceLogging events from a trace. | Generic Events table (when Event Type is Manifested or TraceLogging) |
| trace.UseHandles() | Provides partial data from a trace about active kernel handles. | Handles table |
| trace.UseHardFaults() | Provides data from a trace about hard page faults. | Hard Faults table |
| trace.UseHeapSnapshots() | Provides data from a trace about process heap usage. | Heap Snapshot table |
| trace.UseHypercalls() | Provides data about Hyper-V hypercalls that occured during a trace. | |
| trace.UseImageSections() | Provides data from a trace about the sections of an image. | Section Name column of the CPU Usage (Sampled) table |
| trace.UseInterruptHandlingData() | Provides data from a trace about Interrupt Service Routine (ISR) and Deferred Procedure Call (DPC) activity. | DPC/ISR table |
| trace.UseMarks() | Provides the marks (labeled timestamps) from a trace. | Marks table |
| trace.UseMemoryUtilizationData() | Provides data from a trace about total system memory utilization. | Memory Utilization table |
| trace.UseMetadata() | Provides trace metadata available without further processing. | System Configuration, Traces and General |
| trace.UsePlatformIdleStates() | Provides data from a trace about the target and actual platform idle states of a system. | Platform Idle State table |
| trace.UsePoolAllocations() | Provides data from a trace about kernel pool memory usage. | Pool Summary table |
| trace.UsePowerConfigurationData() | Provides data from a trace about system power configuration. | System Configuration, Power Settings |
| trace.UsePowerDependencyCoordinatorData() | Provides data from a trace about active power dependency coordinator phases. | Notification Phase Summary table |
| trace.UseProcesses() | Provides data about processes active during a trace as well as their images and PDBs. | Processes table Images table Symbols Hub |
| trace.UseProcessorCounters() | Provides data from a trace about processor performance counter values from Processor Counter Monitor (PCM). | |
| trace.UseProcessorFrequencyData() | Provides data from a trace about the frequency at which processors ran. | Processor Frequency table (when Type is Actual) |
| trace.UseProcessorProfileData() | Provides data from a trace about the active processor power profile. | Processor Profiles table |
| trace.UseProcessorParkingData() | Provides data from a trace about which processors were parked or unparked. | Processor Parking State table |
| trace.UseProcessorParkingLimits() | Provides data from a trace about the maximum allowed number of unparked processors. | Core Parking Cap State table |
| trace.UseProcessorQualityOfServiceData() | Provides data from a trace about the quality of service level for each processor. | Processor Qos Class table |
| trace.UseProcessorThrottlingData() | Provides data from a trace about processor maximum frequency throttling. | Processor Constraints table |
| trace.UseReadyBootData() | Provides data from a trace about boot prefetching activity from Ready Boot. | Ready Boot Events table |
| trace.UseReferenceSetData() | Provides data from a trace about pages of virtual memory used by each process. | Reference Set table |
| trace.UseRegionsOfInterest() | Provides named regions of interest intervals from a trace as specified in an xml configuration file. | Regions of Interest table |
| trace.UseRegistryData() | Provides data about registry activity during a trace. | Registry table |
| trace.UseResidentSetData() | Provides data from a trace about the pages of virtual memory for each process that were resident in physical memory. | Resident Set table |
| trace.UseRundownData() | Provides data from a trace about intervals during which trace rundown data collection occurred. | Shaded regions in the graph timeline |
| trace.UseScheduledTasks() | Provides data about scheduled tasks that ran during a trace. | Scheduled Tasks table |
| trace.UseServices() | Provides data about services that were active or had their state captured during a trace. | Services table System Configuration, Services |
| trace.UseStacks() | Provides data about stacks recorded during a trace. | Stacks table |
| trace.UseStackEvents() | Provides data about events associated with stacks recorded during a trace. | Stacks table |
| trace.UseStackTags() | Provides a mapper that groups stacks from a trace into stack tags as specified in an XML configuration file. | Columns such as Stack Tag and Stack (Frame Tags) |
| trace.UseSymbols() | Provides the ability to load symbols for a trace. | Configure Symbol Paths Load Symbols |
| trace.UseSyscalls() | Provides data about syscalls that occurred during a trace. | Syscalls table |
| trace.UseSystemMetadata() | Provides general, system-wide metadata from a trace. | System Configuration |
| trace.UseSystemPowerSourceData() | Provides data from a trace about the active system power source (AC vs DC). | System Power Source table |
| trace.UseSystemSleepData() | Provides data from a trace about overall system power state. | Power Transition table |
| trace.UseTargetCpuIdleStates() | Provides data from a trace about target CPU C-states. | CPU Idle States table (when Type is Target) |
| trace.UseTargetProcessorFrequencyData() | Provides data from a trace about target processor frequencies. | Processor Frequency table (when Type is Target) |
| trace.UseThreads() | Provides data about threads active during a trace. | Thread Lifetimes table |
| trace.UseTraceStatistics() | Provides statistics about the events in a trace. | System Configuration, Trace Statistics |
| trace.UseUtcData() | Provides data from a trace about Microsoft telemetry activity using Universal Telemetry Client (UTC). | Utc table |
| trace.UseWindowInFocus() | Provides data from a trace about changes to the active UI window in focus. | Window in Focus table |
| trace.UseWindowsTracePreprocessorEvents() | Provides Windows software trace preprocessor (WPP) events from a trace. | WPP Trace table Generic Events table (when Event Type is WPP) |
| trace.UseWinINetData() | Provides data from a trace about internet activity via Windows Internet (WinINet). | Download Details table |
| trace.UseWorkingSetData() | Provides data from a trace about pages of virtual memory that were in the working set for each process or kernel category. | Virtual Memory Snapshots table |
Extensibility
Many kinds of trace data have built-in support TraceProcessor, but if you have your other providers that you would like to analyze (including your own custom providers), that data is also available from the trace live while processing occurs.
For example, here is a simple way to get the list of providers IDs in a trace:
HashSet<Guid> providerIds = new HashSet<Guid>();
trace.Use((e) => providerIds.Add(e.ProviderId));
trace.Process();
foreach (Guid providerId in providerIds)
{
Console.WriteLine(providerId);
}
The following example shows a simplified custom data source:
CustomDataSource customDataSource = new CustomDataSource();
trace.Use(customDataSource);
trace.Process();
Console.WriteLine(customDataSource.Count);
…
class CustomDataSource : IFilteredEventConsumer
{
public IReadOnlyList<Guid> ProviderIds { get; } = new Guid[] { new Guid("your provider ID") };
public int Count { get; private set; }
public void Process(EventContext eventContext)
{
++Count;
}
}
Feedback
The documentation for TraceProcessor is currently limited to partial IntelliSense support, though we would like to add more reference documentation as well as samples. (Documentation is a work-in-progress).
This release is a preview to gauge interest, so if this package is useful to you, please let us know. We would also appreciate feedback on what areas we invest in further, such as:
- Expanding IntelliSense documentation
- Adding TraceProcessor API reference documentation to docs.microsoft.com
- Adding samples
- Removing the requirement for msis in the bin directory
- Adding additional data to TraceProcessor that is currently present in WPA
- Stabilizing on a GA/v1 release
For questions using this package, you can post on StackOverflow with the tag .net-traceprocessing, and feedback can also be sent via email to traceprocessing@microsoft.com.
The post Announcing TraceProcessor Preview 0.1.0 appeared first on Windows Blog.
Source: Announcing TraceProcessor Preview 0.1.0
