B# .NET Blog

Oh my god, what a painful experience this was. Today, I was playing around with the Virtual Server 2005 R2 API to create a virtual networks visualizer which should be pretty straightforward to do thanks to the COM API that comes with the product. And yes indeed, it's fairly simple as I expected. However, debugging this stuff in VS2005 is rather difficult.

Let's start with the Ole32.dll's CoInitializeSecurity call that's required to get the Virtual Server 2005 COM interop to work. Simply stated, this call has to happen in the process before any other call to the function otherwise you end up with a RPC_E_ TOO_LATE error code (a.k.a. 0x80010119). Basically, one just needs to turn impersonation on in order to talk to Virtual Server 2005 (on the same machine or in the domain) and to gain access to the functions inside. The required call looks as follows:

CoInitializeSecurity(IntPtr.Zero, -1, IntPtr.Zero, IntPtr.Zero, (uint)RpcAuthnLevel.PktPrivacy, (uint)RpcImpLevel.Impersonate, IntPtr.Zero, (uint)EoAuthnCap.DynamicCloaking, IntPtr.Zero);

However, the trouble is the vshost.exe process in Visual Studio 2005 that's used for debugging purposes (when you press F5, it's the *.vshost.exe executable that get's launched). As the matter in fact, this process does the following:

  IL_0000:  call       class [mscorlib]System.Threading.EventWaitHandle [Microsoft.VisualStudio.HostingProcess.Utilities.Sync]Microsoft.VisualStudio.HostingProcess.Synchronize::get_HostingProcessInitialized()
  IL_0005:  brfalse.s  IL_0024
  IL_0007:  call       class [mscorlib]System.Threading.EventWaitHandle [Microsoft.VisualStudio.HostingProcess.Utilities.Sync]Microsoft.VisualStudio.HostingProcess.Synchronize::get_HostingProcessInitialized()
  IL_000c:  callvirt   instance bool [mscorlib]System.Threading.EventWaitHandle::Set()
  IL_0011:  pop
  IL_0012:  call       class [mscorlib]System.Threading.ManualResetEvent [Microsoft.VisualStudio.HostingProcess.Utilities.Sync]Microsoft.VisualStudio.HostingProcess.Synchronize::get_StartRunningUsersAssembly()
  IL_0017:  brfalse.s  IL_0024
  IL_0019:  call       class [mscorlib]System.Threading.ManualResetEvent [Microsoft.VisualStudio.HostingProcess.Utilities.Sync]Microsoft.VisualStudio.HostingProcess.Synchronize::get_StartRunningUsersAssembly()
  IL_001e:  callvirt   instance bool [mscorlib]System.Threading.WaitHandle::WaitOne()
  IL_0023:  pop
  IL_0024:  ret

The solution is fairly straightforward and is to disable the Visual Studio hosting process using the project's properties, tab Debug. Finally I can start debugging my new 'invention'.