Sending an SOS

The previous post explained how to take advantage of DbgHost.exe, the debugging host provided by DebugDiag, to make a debugger engine load a dump file and execute commands like what you can do manually in WinDBG and the Immediate window of Visual Studio.

The next step is to load well known extensions that we are usually use to diagnose problems from a dump file. If you are working with .NET applications, SOS is your friend. This extension provides many commands that help you understand what is going on in your managed applications. This .dll is installed with each version of the CLR under C:\Windows\Microsoft.NET\Framework. This is nice except when the dump you are trying to analyze comes from a system with a different version of the CLR runtime. In that case, there is no problem when trying to load the extension with the usual command line .loadby sos mscorwks/clr.
However, problems start when calling methods from the extension. Here is the typical error you get in that case:
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks___.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.

You can also run the debugger command .cordll to control the debugger’s
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.

So… What is exactly the issue?
The sos.dll extensions relies on a private data access layer named mscordacwks.dll that must also be in the binary paths of the symbol store. As a reminder, this piece of information is given as parameter to DbgControl.OpenDump().
The usual way to set the path is through the following environment variable:
_NT_EXECUTABLE_IMAGE_PATH=SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

Note that the symbol path is set through this other environment variable:
_NT_SYMBOL_PATH=SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

So… When the debugging engine tries to find the mscordacwks.dll, it looks for what exists in the symbol store; first in the local one (c:\symbols in my example) and if not there, in the http-based one (http://msdl.microsoft.com/download/symbols in my example).
As mentionned in the error message, you can use .cordll -ve -u -l to force a reload through the symbol engine.
This looks nice on the paper but, unfortunately, the public http-based Microsoft symbol store does not provide access to the mscordacwks.dll…
So, even if you follow the explanations given by the error message and you try to use:
!sym noisy; .cordll -ve -u -l
the additionnal details are not really helpful:

CLRDLL: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.4959 f:0
doesn't match desired version 2.0.50727.3607 f:0

shows that the installed version does not match the version expected from the dump file.

Next, you get a list of the attempts to load the dll from all and every folders:

SYMSRV: c:\symbols\mscordacwks_x86_x86_2.0.50727.3607.dll\4ADD5446590000\mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscordacwks_x86_x86_2.0.50727.3607.dll/4ADD5446590000/mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: c:\symbols\mscordacwks_x86_x86_2.0.50727.3607.dll\4ADD5446590000\mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscordacwks_x86_x86_2.0.50727.3607.dll/4ADD5446590000/mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: c:\symbols\mscordacwks_x86_x86_2.0.50727.3607.dll\4ADD5446590000\mscordacwks_x86_x86_2.0.50727.3607.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscordacwks_x86_x86_2.0.50727.3607.dll/4ADD5446590000/mscordacwks_x86_x86_2.0.50727.3607.dll not found
CLRDLL: Unable to find mscordacwks_x86_x86_2.0.50727.3607.dll by mscorwks search
CLRDLL: Unable to find 'mscordacwks_x86_x86_2.0.50727.3607.dll' on the path
SYMSRV: c:\symbols\mscorwks.dll\4ADD5446590000\mscorwks.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscorwks.dll/4ADD5446590000/mscorwks.dll not found
DBGHELP: C:\Windows\system32\mscorwks.dll - file not found
SYMSRV: c:\symbols\mscorwks.dll\4ADD5446590000\mscorwks.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscorwks.dll/4ADD5446590000/mscorwks.dll not found
DBGHELP: C:\Windows\system32\mscorwks.dll - file not found
SYMSRV: c:\symbols\mscorwks.dll\4ADD5446590000\mscorwks.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mscorwks.dll/4ADD5446590000/mscorwks.dll not found
CLRDLL: Unable to find mscorwks.dll by search
CLRDLL: ERROR: Unable to load DLL mscordacwks_x86_x86_2.0.50727.3607.dll, Win32 error 0n2
Failed to load data access DLL, 0x80004005

So… To sum up, you simply need to manually get mscordacwks.dll from the system on which the dump file was taken and store it into the right folder of the local symbol store with the right name as given in the error message. For, example, based on the error messages given as example, you should copy the mscordacwks.dll into the folder:
C:\symbols\mscordacwks_x86_x86_2.0.50727.3607.dll\4ADD5446590000
with the following name:
mscordacwks_x86_x86_2.0.50727.3607.dll
After several investigations, you end up with many mscordacwks folder in your local symbol store:

Once the sos extension (and more importantly its friend mscordacwks.dll) loaded, you can start executing your prefered commands; beginning with !eeversion to get the version of the CLR and the corresponding sos extension:

2.0.50727.4959 retail
Workstation mode
SOS Version: 2.0.50727.4959 retail build

The next post will present a new tool built on what has been introduced both here and in the previous post

because things should (not) be S(imple) A(fter) D(ump) instead of usually being just sad

Advertisements
This entry was posted in .NET, Development. Bookmark the permalink.

One Response to Sending an SOS

  1. Pingback: S.A.D. or S(imple) A(fter) D(ump) | Code & Debug

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s