Announcing TraceProcessor Preview 0.1.0

Process ETW traces in .NET.

Background

Event Tracing for Windows (ETW) is a powerful trace collection system built-in to the Windows operating system. Windows has deep integration with ETW, including data on system behavior all the way down to the kernel for events like context switches, memory allocation, process create and exit, and many more. The system-wide data available from ETW makes it a good fit for end-to-end performance analysis or other questions that require looking at the interaction between by many components throughout the system.

Unlike text logging, ETW provides structured events designed for automated data processing. Microsoft has built powerful tools on top of these structured events, including Windows Performance Analyzer (WPA), which provides a graphical interface for visualizing and exploring the trace data captured in a ETW trace file (.etl).

Inside Microsoft, we heavily use ETW traces to measure the performance of new builds of Windows. Given the volume of data produced the Windows engineering system, automated analysis is essential. For our automated trace analysis, we heavily use C# and .NET, so we created a package that provides a .NET API for accessing many kinds of ETW trace data. This technology is also used inside Windows Performance Analyzer to power several of its tables. We are happy to release a preview of this package that we use to analyze Windows so that you can use it to analyze your own applications and systems as well.

Getting Started

This package is available from NuGet with the following package ID:

Microsoft.Windows.EventTracing.Processing.All

After installing this package, the following console application lists all the process command lines in a trace:

using Microsoft.Windows.EventTracing;
using Microsoft.Windows.EventTracing.Processes;
using System;

class Program
{
    static void Main(string[] args)
    {
        if (args.Length != 1)
        {
            Console.Error.WriteLine("Usage: <trace.etl>");
            return;
        }

        using (ITraceProcessor trace = TraceProcessor.Create(args[0]))
        {
            IPendingResult<IProcessDataSource> pendingProcessData = trace.UseProcesses();
            trace.Process();
            IProcessDataSource processData = pendingProcessData.Result;

            foreach (IProcess process in processData.Processes)
            {
                Console.WriteLine(process.CommandLine);
            }
        }
    }
}

The core interface is ITraceProcessor, and using this interface follows a pattern of first, tell the processor what data you want to use from a trace; second, process the trace; and finally, access the results. Telling the processor what kinds of data you want up front means you do not need to spend time processing large volumes of all possible kinds of trace data. Instead, TraceProcessor just does the work needed to provide the specific kinds of data you request.

Data Sources

Process information is just one of many kinds of data that can be stored in an ETW trace. Note that which data is in an .etl file depends on what providers were enabled when the trace was captured. The following list shows the kinds of trace data available from TraceProcessor:

Code	Description	Related WPA Items
trace.UseClassicEvents()	Provides classic ETW events from a trace, which do not include schema information.	Generic Events table (when Event Type is Classic or WPP)
trace.UseConnectedStandbyData()	Provides data from a trace about the system entering and exiting connected standby.	CS Summary table
trace.UseCpuIdleStates()	Provides data from a trace about CPU C-states.	CPU Idle States table (when Type is Actual)
trace.UseCpuSamplingData()	Provides data from a trace about CPU usage based on periodic sampling of the instruction pointer.	CPU Usage (Sampled) table
trace.UseCpuSchedulingData()	Provides data from a trace about CPU thread scheduling, including context switches and ready thread events.	CPU Usage (Precise) table
trace.UseDevicePowerData()	Provides data from a trace about device D-states.	Device DState table
trace.UseDirectXData()	Provides data from a trace about DirectX activity.	GPU Utilization table
traceUseDiskIOData()	Provides data from a trace about Disk I/O activity.	Disk Usage table
trace.UseEnergyEstimationData()	Provides data from a trace about estimated per-process energy usage from Energy Estimation Engine.	Energy Estimation Engine Summary (by Process) table
trace.UseEnergyMeterData()	Provides data from a trace about measured energy usage from Energy Meter Interface (EMI).	Energy Estimation Engine (by Emi) table
trace.UseFileIOData()	Provides data from a trace about File I/O activity.	File I/O table
trace.UseGenericEvents()	Provides manifested and TraceLogging events from a trace.	Generic Events table (when Event Type is Manifested or TraceLogging)
trace.UseHandles()	Provides partial data from a trace about active kernel handles.	Handles table
trace.UseHardFaults()	Provides data from a trace about hard page faults.	Hard Faults table
trace.UseHeapSnapshots()	Provides data from a trace about process heap usage.	Heap Snapshot table
trace.UseHypercalls()	Provides data about Hyper-V hypercalls that occured during a trace.
trace.UseImageSections()	Provides data from a trace about the sections of an image.	Section Name column of the CPU Usage (Sampled) table
trace.UseInterruptHandlingData()	Provides data from a trace about Interrupt Service Routine (ISR) and Deferred Procedure Call (DPC) activity.	DPC/ISR table
trace.UseMarks()	Provides the marks (labeled timestamps) from a trace.	Marks table
trace.UseMemoryUtilizationData()	Provides data from a trace about total system memory utilization.	Memory Utilization table
trace.UseMetadata()	Provides trace metadata available without further processing.	System Configuration, Traces and General
trace.UsePlatformIdleStates()	Provides data from a trace about the target and actual platform idle states of a system.	Platform Idle State table
trace.UsePoolAllocations()	Provides data from a trace about kernel pool memory usage.	Pool Summary table
trace.UsePowerConfigurationData()	Provides data from a trace about system power configuration.	System Configuration, Power Settings
trace.UsePowerDependencyCoordinatorData()	Provides data from a trace about active power dependency coordinator phases.	Notification Phase Summary table
trace.UseProcesses()	Provides data about processes active during a trace as well as their images and PDBs.	Processes table Images table Symbols Hub
trace.UseProcessorCounters()	Provides data from a trace about processor performance counter values from Processor Counter Monitor (PCM).
trace.UseProcessorFrequencyData()	Provides data from a trace about the frequency at which processors ran.	Processor Frequency table (when Type is Actual)
trace.UseProcessorProfileData()	Provides data from a trace about the active processor power profile.	Processor Profiles table
trace.UseProcessorParkingData()	Provides data from a trace about which processors were parked or unparked.	Processor Parking State table
trace.UseProcessorParkingLimits()	Provides data from a trace about the maximum allowed number of unparked processors.	Core Parking Cap State table
trace.UseProcessorQualityOfServiceData()	Provides data from a trace about the quality of service level for each processor.	Processor Qos Class table
trace.UseProcessorThrottlingData()	Provides data from a trace about processor maximum frequency throttling.	Processor Constraints table
trace.UseReadyBootData()	Provides data from a trace about boot prefetching activity from Ready Boot.	Ready Boot Events table
trace.UseReferenceSetData()	Provides data from a trace about pages of virtual memory used by each process.	Reference Set table
trace.UseRegionsOfInterest()	Provides named regions of interest intervals from a trace as specified in an xml configuration file.	Regions of Interest table
trace.UseRegistryData()	Provides data about registry activity during a trace.	Registry table
trace.UseResidentSetData()	Provides data from a trace about the pages of virtual memory for each process that were resident in physical memory.	Resident Set table
trace.UseRundownData()	Provides data from a trace about intervals during which trace rundown data collection occurred.	Shaded regions in the graph timeline
trace.UseScheduledTasks()	Provides data about scheduled tasks that ran during a trace.	Scheduled Tasks table
trace.UseServices()	Provides data about services that were active or had their state captured during a trace.	Services table System Configuration, Services
trace.UseStacks()	Provides data about stacks recorded during a trace.	Stacks table
trace.UseStackEvents()	Provides data about events associated with stacks recorded during a trace.	Stacks table
trace.UseStackTags()	Provides a mapper that groups stacks from a trace into stack tags as specified in an XML configuration file.	Columns such as Stack Tag and Stack (Frame Tags)
trace.UseSymbols()	Provides the ability to load symbols for a trace.	Configure Symbol Paths Load Symbols
trace.UseSyscalls()	Provides data about syscalls that occurred during a trace.	Syscalls table
trace.UseSystemMetadata()	Provides general, system-wide metadata from a trace.	System Configuration
trace.UseSystemPowerSourceData()	Provides data from a trace about the active system power source (AC vs DC).	System Power Source table
trace.UseSystemSleepData()	Provides data from a trace about overall system power state.	Power Transition table
trace.UseTargetCpuIdleStates()	Provides data from a trace about target CPU C-states.	CPU Idle States table (when Type is Target)
trace.UseTargetProcessorFrequencyData()	Provides data from a trace about target processor frequencies.	Processor Frequency table (when Type is Target)
trace.UseThreads()	Provides data about threads active during a trace.	Thread Lifetimes table
trace.UseTraceStatistics()	Provides statistics about the events in a trace.	System Configuration, Trace Statistics
trace.UseUtcData()	Provides data from a trace about Microsoft telemetry activity using Universal Telemetry Client (UTC).	Utc table
trace.UseWindowInFocus()	Provides data from a trace about changes to the active UI window in focus.	Window in Focus table
trace.UseWindowsTracePreprocessorEvents()	Provides Windows software trace preprocessor (WPP) events from a trace.	WPP Trace table Generic Events table (when Event Type is WPP)
trace.UseWinINetData()	Provides data from a trace about internet activity via Windows Internet (WinINet).	Download Details table
trace.UseWorkingSetData()	Provides data from a trace about pages of virtual memory that were in the working set for each process or kernel category.	Virtual Memory Snapshots table

Extensibility

Many kinds of trace data have built-in support TraceProcessor, but if you have your other providers that you would like to analyze (including your own custom providers), that data is also available from the trace live while processing occurs.

For example, here is a simple way to get the list of providers IDs in a trace:

HashSet<Guid> providerIds = new HashSet<Guid>();
trace.Use((e) => providerIds.Add(e.ProviderId));
trace.Process();

foreach (Guid providerId in providerIds)
{
    Console.WriteLine(providerId);
}

The following example shows a simplified custom data source:

CustomDataSource customDataSource = new CustomDataSource();
trace.Use(customDataSource);
trace.Process();
Console.WriteLine(customDataSource.Count);

…

class CustomDataSource : IFilteredEventConsumer
{
    public IReadOnlyList<Guid> ProviderIds { get; } = new Guid[] { new Guid("your provider ID") };

    public int Count { get; private set; }

    public void Process(EventContext eventContext)
    {
        ++Count;
    }
}

Feedback

The documentation for TraceProcessor is currently limited to partial IntelliSense support, though we would like to add more reference documentation as well as samples. (Documentation is a work-in-progress).

This release is a preview to gauge interest, so if this package is useful to you, please let us know. We would also appreciate feedback on what areas we invest in further, such as:

• Expanding IntelliSense documentation
• Adding TraceProcessor API reference documentation to docs.microsoft.com
• Adding samples
• Removing the requirement for msis in the bin directory
• Adding additional data to TraceProcessor that is currently present in WPA
• Stabilizing on a GA/v1 release

For questions using this package, you can post on StackOverflow with the tag .net-traceprocessing, and feedback can also be sent via email to [email protected].

The post Announcing TraceProcessor Preview 0.1.0 appeared first on Windows Blog.

Source: Announcing TraceProcessor Preview 0.1.0