Use SAD to view RCWs and CCWs

Download SAD v1.2 x86|x64

The first tool extension for SAD is about COM.
When you need to call COM objects methods from .NET code, the runtime creates for you R(untime) C(allable) W(rappers) to hide all the complexity behind this kind of communication and lifetime management. On the opposite, when your native code calls methods from .NET components presented as COM objects, the runtime also creates kind of proxy objects named C(om) C(allable) W(rappers).

In both case, you might need to ensure that your code handle the lifetime of such wrappers the right way (maybe playing with ReleaseComObject sometimes). To sum up, you need to know which COM objects are still refusing to self-release behind a RCW or how instances of .NET types are still referenced by CCWs. This is exactly what the « COM Analysis » tool does in SAD.

When you click this menu, a new window appears that parses the output result of sos!syncblk -all.

At the end of the analysis, the identified RCWs and CCWs are listed. If you click on a RCW line, you get a dump of the first addresses in the vtable of the corresponding COM object. Since the version 4.0 of the runtime, you always see __ComObject instead of the generated proxy class with a name that could help you figure out which COM object has been instanciated. As you can see,

you still get the vtable details even for__ComObject.
Note that you should have setup the symbols for your COM objects if you want to see meaningful names instead of just the name of the DLL where the COM objects are implemented.

The CCW case is even simpler since the type name and the reference of the .NET instance wrapped by a CCW appears in the right handside of the line

References:

I hope this helps.

Posted in .NET, Development, Tools | Leave a comment

How to find kernel handles leaks

One of my customers was looking for a way to find out if an application was leaking kernel objects or not during specific worklows. The first tool to look at is the Swiss knife ProcessExplorer from SysInternals.

When you select a process, the lower pane lets you see which kernel objects has been created with a lot of informations such as their handle, type and details. So, how to use ProcessExplorer to identify a leak? Simply by stopping the automatic refresh

and pressing the F5 key to refresh the kernel objects list. When a new object is created since the last refresh, it appears in green

or  it appears in red when it has been closed

However, this green/red dance does not support more than one comparison ara a time and it is not possible to keep track of the different snapshots along the way. This is why I’ve written Kernel Handle Leak Shell. This tool share many features with LeakShell that helps you find out which instances of managed types stay referenced over time:

  • take snapshots of kernel objects consumption at will
  • show the kernel objects count evolution in a graphical way
  • define new reference and current snapshots to trigger a comparison

but also bring new ones such as :

  • select a process to study
  • see the New, Same and Deleted objects between two snapshots
  • sort the lists by handle, kind or details
  • see the handles for a given snapshot by double-clicking it in the list
 

The engine used by this tool to list the kernel objects of a given process is handle.exe, another executable provided by Mark Russinovich on SysInternals. I’m simply redirecting the output of this console application into a string before parsing it to build my own internal list of kernel objects.

Download KernelLeakShell 1.0

The only prerequisite is to download handle.exe from SysInternals on top of the .NET Framework 4.0 for the DataGrid and the WPF Toolkit (included into the downloadable .zip) for the chart.

References:

I hope this helps.

Posted in Development, Tools | 2 Comments

LeakShell 1.4 – Use the dumps Luke!

After the first release of LeakShell, here is an update that allows you to directly provide dump files .
This evolution is based on what I’ve presented to control a debugger engine and send commands to the sos.dll extension.

You’re still able to use LeakShell as you do your investigation within WinDBG but also post-mortem based on dumps of the application takens at different time while memory is growing: you just have to drag the .dmp files to the snapshot listview on the upper left side of the application.

As with S.A.D., you need to use the right version of LeakShell: the x86 version for 32 bit dumps or the x64 version for the 64 bit dumps.

Download LeakShell 1.4 x86 | x64

The only prerequisite is to install DebugDiag x86 | x64 on top of .NET Framework 4.0.

Posted in .NET, Development, Memory, Tools | 2 Comments

S.A.D. or S(imple) A(fter) D(ump)

When I’m investigating issues for customers, I usually use WinDBG to dig into some dump files. In the case of .NET applications, I’m taking advantage of the commands exported by the sos/sosex/psscor extensions to get a view of the application state.

It is common that I need to execute commands based on others command output. For example, when I need to get the detailed state of threads, I’m calling !sos.threadstate to decipher the values returned by !sos.threads. In addition to the debugging session window, I’m also showing the Command Browser window (CTRL+N) to calling other commands such as !threadstate while seeing a previous command like !threads result

This is great because the Command Browser keeps track of the commands I’ve sent but unfortunately, only here; not in the debugging session window. So I often have
to scroll into the debugging session to find the previous command result I’m interested in. I’ve not found a way to define a kind of favorites or shortcuts to parts of the big chunk of text where all the command results end up.

Even though the commands are stacked and can be accessed with up and down arrows, there is no auto-completion available; neither in the debugging session nor in the Command Browser. It would be great if the extensions commands would be easily accessible via auto-completion: I’m using Visual Studio so much that it becomes difficult when auto-completion goes away   :^)

After I’ve found how to control debugger sessions as shown in the two previous posts, building a WinDBG for dummies seemed a good exercise. And… here comes S(imple) A(fter) D(ump)!

The list of features is small and the main goal is to make it simple to dig into dump files:

  • Load x86/x64 dump files
    see https://codenasarre.wordpress.com/2011/06/14/how-to-control-a-debugger-engine/ for more details

  • Load sos extension by default
    see https://codenasarre.wordpress.com/2011/06/22/sending-an-sos/ for more details

  • Keep track of all executed commands with their corresponding result and being able to delete some

    The executed commands appears in a list above the command box. Use the DEL key on a selected command to remove it from the history.

  • Have auto-completion for commands (just from sos for the moment) to send to the debugger
    The command box at the bottom left of the main window allows you to send commands to the debugger and the known or already executed appears in a popup list which content is based on what has been typed.

    Characters entered are used as filters

  • Being able to add comments in any command result
    The textbox on the right handside displays the result of the currently selected command in the history listbox on the left handside.

    This text box let you add comments everywhere that are kept even when you enter new commands or pick another existing one.

  • Easily access previous commands results
    Type SPACE in the command box to popup the list of executed commands.
    Type the number that prefixes the command seen in the listbox above the command
    box to directly access the corresponding command or directly click on it.

  • Being able to watch several commands results at the same time
    Same as WinDBG but with more than one single Command Browser: just double-click
    on a command in the history listbox

The only prerequisite is to install DebugDiag x86 | x64 on top of .NET Framework 4.0. However, the symbol and binary folders are retrieved from the _NT_EXECUTABLE_IMAGE_PATH and _NT_SYMBOL_PATH environment variables. If these are not set… well… Sad won’t be happy.

Download Sad: x86 | x64

In order to make dump analysis even simpler, I get other ideas to enhance Sad such as:

  • allowing explicit loading of extensions (sosex, psscor and more)
  • provide general purpose investigation modules such as memory, threads or exception
  • generating .xps documents based on selected commands results + annotations

Feel free to list your own!

Posted in .NET, Development, Tools | 3 Comments

Sending an SOS

The previous post explained how to take advantage of DbgHost.exe, the debugging host provided by DebugDiag, to make a debugger engine load a dump file and execute commands like what you can do manually in WinDBG and the Immediate window of Visual Studio.

The next step is to load well known extensions that we are usually use to diagnose problems from a dump file. If you are working with .NET applications, SOS is your friend. This extension provides many commands that help you understand what is going on in your managed applications. This .dll is installed with each version of the CLR under C:\Windows\Microsoft.NET\Framework. This is nice except when the dump you are trying to analyze comes from a system with a different version of the CLR runtime. In that case, there is no problem when trying to load the extension with the usual command line .loadby sos mscorwks/clr.
However, problems start when calling methods from the extension. Here is the typical error you get in that case:
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks___.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.

You can also run the debugger command .cordll to control the debugger’s
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.

So… What is exactly the issue?
The sos.dll extensions relies on a private data access layer named mscordacwks.dll that must also be in the binary paths of the symbol store. As a reminder, this piece of information is given as parameter to DbgControl.OpenDump().
The usual way to set the path is through the following environment variable:
_NT_EXECUTABLE_IMAGE_PATH=SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

Note that the symbol path is set through this other environment variable:
_NT_SYMBOL_PATH=SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

So… When the debugging engine tries to find the mscordacwks.dll, it looks for what exists in the symbol store; first in the local one (c:\symbols in my example) and if not there, in the http-based one (http://msdl.microsoft.com/download/symbols in my example).
As mentionned in the error message, you can use .cordll -ve -u -l to force a reload through the symbol engine.
This looks nice on the paper but, unfortunately, the public http-based Microsoft symbol store does not provide access to the mscordacwks.dll…
So, even if you follow the explanations given by the error message and you try to use:
!sym noisy; .cordll -ve -u -l
the additionnal details are not really helpful:

CLRDLL: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.4959 f:0
doesn't match desired version 2.0.50727.3607 f:0

shows that the installed version does not match the version expected from the dump file.

Next, you get a list of the attempts to load the dll from all and every folders:

SYMSRV: c:\symbols\mscordacwks_x86_x86_2.0.50727.3607.dll\4ADD5446590000\mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscordacwks_x86_x86_2.0.50727.3607.dll/4ADD5446590000/mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: c:\symbols\mscordacwks_x86_x86_2.0.50727.3607.dll\4ADD5446590000\mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscordacwks_x86_x86_2.0.50727.3607.dll/4ADD5446590000/mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: c:\symbols\mscordacwks_x86_x86_2.0.50727.3607.dll\4ADD5446590000\mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscordacwks_x86_x86_2.0.50727.3607.dll/4ADD5446590000/mscordacwks_x86_x86_2.0.50727.3607.dll not found
CLRDLL: Unable to find mscordacwks_x86_x86_2.0.50727.3607.dll by mscorwks search
CLRDLL: Unable to find 'mscordacwks_x86_x86_2.0.50727.3607.dll' on the path
SYMSRV: c:\symbols\mscorwks.dll\4ADD5446590000\mscorwks.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscorwks.dll/4ADD5446590000/mscorwks.dll not found
DBGHELP: C:\Windows\system32\mscorwks.dll - file not found
SYMSRV: c:\symbols\mscorwks.dll\4ADD5446590000\mscorwks.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscorwks.dll/4ADD5446590000/mscorwks.dll not found
DBGHELP: C:\Windows\system32\mscorwks.dll - file not found
SYMSRV: c:\symbols\mscorwks.dll\4ADD5446590000\mscorwks.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscorwks.dll/4ADD5446590000/mscorwks.dll not found
CLRDLL: Unable to find mscorwks.dll by search
CLRDLL: ERROR: Unable to load DLL mscordacwks_x86_x86_2.0.50727.3607.dll, Win32 error 0n2
Failed to load data access DLL, 0x80004005

So… To sum up, you simply need to manually get mscordacwks.dll from the system on which the dump file was taken and store it into the right folder of the local symbol store with the right name as given in the error message. For, example, based on the error messages given as example, you should copy the mscordacwks.dll into the folder:
C:\symbols\mscordacwks_x86_x86_2.0.50727.3607.dll\4ADD5446590000
with the following name:
mscordacwks_x86_x86_2.0.50727.3607.dll
After several investigations, you end up with many mscordacwks folder in your local symbol store:

Once the sos extension (and more importantly its friend mscordacwks.dll) loaded, you can start executing your prefered commands; beginning with !eeversion to get the version of the CLR and the corresponding sos extension:

2.0.50727.4959 retail
Workstation mode
SOS Version: 2.0.50727.4959 retail build

The next post will present a new tool built on what has been introduced both here and in the previous post

because things should (not) be S(imple) A(fter) D(ump) instead of usually being just sad

Posted in .NET, Development | 1 Comment

How to control a debugger engine?

In the previous post, I’ve introduced a tool that helps finding managed memory leaks based on output obtained via WinDBG and the SOS extension. Even though the clipboard interception makes the process easier and faster, it is still boring to open dumps in WinDBG, then load sos before copying the output of !dumpheap -stat.
It would be great if all this could be automated: this post will quickly present one way to write code that controls the debugging engine instead of manually dealing with WinDBG.

Looking for an API

After a few research, I ended up into the native API provided by dbghelp.dll that I used a long time ago. It was presented by Matt Pietrek in MSDN Magazine. It reminded me the old IMAGEHLP days when Matt and John Robbins detailed the symbolsoriented API in MSJ (any souvenir of SUPERASSERT? :^)

In addition to MSDN Magazine, John detailed part of the dbghelp API in one of his books.

But most of the API was related to symbol management and things have changed a little since that time. When you take a look at the include files, DbgEng.h contains 16.000+ lines and DbgHelp.h has more than 4.500 lines. The former defines a long list of COM interfaces with IDebugClient.OpenDumpFile to open a dump file. The latter allows you to access the different streams (defined in MINIDUMP_STREAM_TYPE) of a dump file with MiniDumpReadDumpStream() once it has been mapped into memory as a memory mapped files.

Since I want to access to a dump file in managed code, load extensions and send commands to be parsed, these two APIs are way too complicated for what I have in mind. CodePlex also provides two wrappers and a PowerShell interface for the debugger engine API but I’ve found another solution.

As stated on the download page, “the Debug Diagnostic Tool (DebugDiag) is designed to assist in troubleshooting issues such as hangs, slow performance, memory leaks or fragmentation, and crashes in any user-mode process. The tool includes additional debugging scripts focused on Internet Information Services (IIS) applications, web data access components, COM+ and related Microsoft technologies”.
Even better, “the Debugger Host (DbgHost.exe) hosts the Windows Symbolic Debugger Engine (dbgeng.dll) to attach to processes and generate memory dumps. It also hosts the main analyzer module to analyze memory dumps” and exposes high level services as a COM server.

Note that beta 2 version of DebugDiag is also available for x86 and x64.

Using DbgHost in C#

Once DebugDiag installed, the DbgHost COM server is here to help you. Open your favorite Visual Studio and add a COM reference into your C# project

But what do we get exactly?
Well… First, the IDbgControl interface allows you to open a dump file

by giving its path in addition to three other parameters:

  • SymbolPath: you give the symbol store path so the engine will find the .pdb files for the loaded modules.
  • ImagePath: you give the symbol store path so the engine will find the loaded .dlls when needed.
  • pProgress: I have no idea what is its purpose…

If you are working with WinDBG, you already know how the symbol store is important. Otherwise… well… read Advanced .NET Debugging by Mario Hewardt and Patrick Dussud.
You will see in a forthcoming post how important the symbol store is when dealing with the sos.dll extension.

Once a dump file is loaded by the debugging engine, you access it through the IDbgObj interface returned by IDbgControl.OpenDumpFile().


The Execute method takes the command to run as a string and returns the result as a string too.
Simple!

First code sample

It is easy to write a couple of lines that load a dump file and get a few details about it. In addition to the two already described methods, the following code shows how to use other IDbgObj properties.

    string dumpPath;
    dumpPath = @"c:\dumps\WpfLeaky_x86_3.5.dmp";
 
    DbgControl dbgControl = new DbgControl();
    DbgObj dbgSession = dbgControl.OpenDump(dumpPath, @"c:\symbols"@"c:\symbols"null);
    Console.WriteLine("dump '{0}' loaded.", dbgSession.DumpType);
    Console.WriteLine("exe: {0}", dbgSession.ExecutableName);
    Console.WriteLine("OSVersion: {0}", dbgSession.OSVersion);
    Console.WriteLine("   Major:       {0}", dbgSession.OSVersionMajor);
    Console.WriteLine("   Minor:       {0}", dbgSession.OSVersionMinor);
    Console.WriteLine("   ServicePack: {0}", dbgSession.OSServicePack);
    Console.WriteLine("   Build:       {0}", dbgSession.OSBuild);
 
    string result;
    string command;
    // look for additionnal details about the dump file
    command = "vertarget";
    Console.WriteLine(string.Format("\r\n{0}\r\n--------------------", command));
    result = dbgSession.Execute(command);
    Console.WriteLine(result);

As you can see in the result, the OSVersion does not seem to be meaningful.

dump 'MINIFULLDUMP' loaded.
exe: C:\Personel\dev\ManagedLeaks\Current\WpfLeaky\bin\Debug\WpfLeaky.exe
OSVersion: Unexpected
Major: 6
Minor: 1
ServicePack:
Build: 6.1.7600.16385 (win7_rtm.090713-1255)

vertarget
--------------------
Windows 7 Version 7600 MP (8 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
kernel32.dll version: 6.1.7600.16385 (win7_rtm.090713-1255)
Machine Name:
Debug session time: Mon Apr 18 06:07:41.000 2011 (UTC + 2:00)
System Uptime: 2 days 21:29:04.007
Process Uptime: 0 days 0:00:26.000
Kernel time: 0 days 0:00:00.000
User time: 0 days 0:00:00.000

Interestingly enough, you are able to figure out if a dump is 32 bit or 64 bit by parsing the first line of the “vertarget” command result: “x86” for 32 bit and “x64” for 64 bit. It is very important because you should always use the 32 bit version of the debugging engine to work with a 32 bit dump. Same rule for a 64 bit dump: always use the 64 bit version of the engine in that case.
The solution I’ve chosen is to compile two versions of my application: one for x86 and the other for x64. Each one checks that the dump to load has the expected bitness.

The next post will discuss the issues related to using the sos.dll extension.

Other Reference

Posted in .NET, Development | 2 Comments

LeakShell or how to (almost) automatically find managed leaks

I’m assuming that the notion of memory leak in .NET is clear but if it is not, you should read Identify And Prevent Memory Leaks In Managed Code by James Kovacs and the recap of GC behaviors by Jeffrey Richter in Part 1 and Part2.

When a memory leak has been identified through TaskManager, PerfMon, ProcessExplorer, System.Diagnostics.Process.PrivateMemorySize64 or System.OutOfMemoryException, it is time to find out what instances are still referenced while they should not and by which object(s).

This post focuses on the first part of the investigation : what instances are stuck in gen2 part of the managed memory.

I’ve used one of the leaks examples given in Finding Memory Leaks in WPF-based applications to write an application where objects are still referenced even one might think they are not.


As you might guess, when a StickyEventsWindow gets closed, it stays in memory due to incorrect event registration.

Rico Mariani provides a simple and efficient way of Tracking down managed memory leaks (how to find a GC leak). The idea is to use sos.dll and a debugger to list the objects managed by the GC.

When you investigate a memory leak, you are either lucky enough to reproduce it in your developpement environment or on production machines far far away. In the former case, you can attach your debugger to the application. In the latter case, you’ll ask the administrator to capture dumps of the application. For more details about the 666 ways to take a minidump, take a look at How to Capture a Minidump: Let Me Count the Ways by John Robbins.

In both cases, you end up using sos commands. Let’s follow the usual steps:

  1. Load the sos.dll extension.loadby sos mscorwks (for .NET 2/3.0/3.5).loadby sos clr (otherwise)
  2. List the objects in generations, sorted by size!dumpheap -stattotal 0 objects

    Statistics:

    MT Count TotalSize Class Name

    6cc45b70 1 12 System.Collections.Generic.ObjectEqualityComparer`1[[System.Type, mscorlib]]

    6cc44d7c 1 12 System.Security.Permissions.ReflectionPermission

    6cc44c98 1 12 System.Security.Permissions.FileDialogPermissiont

    6cc38140 699 53124 System.Threading.ExecutionContext+ExecutionContextRunData

    005ca200 155 58744 Free

    6cbf6de0 792 123740 System.Object[]

    6cc3fb64 3242 129056 System.String

    Total 25368 objects

At that point, we have the state of the application in term of allocated objects. We need to wait for more objects to be created an take another !dumpheap -stat snapshot but with leaked objects.

The painful process is to compare the two snapshots. Here is one solution:

  1. copy and paste special one list into Excel to project the MT, Count TotalSize and Class Name into columns
  2. sort the columns by MTNote that it is possible to have instances of the same ClassName for two reasons: same type in different AppDomain and differents types from different assemblies. Only the Method Table address is the key to distinguish CLR types.
  3. save into a .txt file
  4. windiff two files to see the differences

However, the differences might end up with a lot of noise:

  1. not interested in count reduction
  2. often not interested in types from the BCL but by our own type only

I decided to build a tool to compute these steps for me and voila: LeakShell was born. It automatically intercepts the copy to clipboard in text format that contains “!dumpheap -stat”, sort it by MT and add it into a collection of snapshops that you can compare one against the other.

Once you’ve identified the types of the objects with a count that keeps on increasing, it is time to take a look at your code. Well… you can also keep on investigating with sos.gcroot but it is a nightmare even if you use John Robbins trick to jump from address to address with a mouse click.

Another more graphical solution is to use CLRProfiler. But… how to bridge between the debugger and CLR Profiler? Use sos.traverseheap Luke! as explained by the integrated help:

!help traverseheap

——————————————————————————-

!TraverseHeap [-xml]

!TraverseHeap writes out a file in a format understood by the CLR Profiler.

You can download the CLR Profiler from this link:

http://www.microsoft.com/downloads/details.aspx?FamilyId=86CE6052-D7F4-4AEB-B7A-94635BEEBDDA&displaylang=en

It creates a graphical display of the GC heap to help you analyze the state of

your application. If you pass the “-xml” flag, the file is instead written out

in an easy-to-understand xml format:

You can break into your process, load SOS, take a snapshot of your heap with

this function, then continue.

Read NET Best Practice No: 1:- Detecting High Memory consuming functions in .NET code by Shivprasad koirala for more details about how to use the CLR Profiler.

Please feel free to post your comments about how to improve LeakShell.

I can’t promise that I’ll implement everything but… who knows?:^)

Download LeakShell

Other References:

CLR Profiler download links:

Posted in .NET, Memory, Tools | 5 Comments