When dealing with .dlls and Visual Studio, there is a well-known problem of the Visual Studio debugger holding onto the .pdb file, even after the .dll has been unloaded by a call to FreeLibrary().
There are certain use-cases where .dlls are going to be unloaded, recompiled, and then loaded again (think Runtime-Compiled C++), so keeping a lock onto the .pdb file renders the linker unable to produce a new file, which is bad. I stumbled upon this problem in the past already and circumvented the issue by generating .dlls and .pdbs with unique names on each recompile.
However, for the tool I’m currently working on, I wanted to take another stab at “fixing” the root issue: deleting the old .pdb file, so the linker can produce a new one.
But how do we delete a file that’s locked by another process?
Things that won’t work
- For obvious reasons, DeleteFile() won’t work and returns an “Access denied” error.
- Examining file-related activity using ProcMon led me to SetFileInformationByHandle and a thing known as FILE_DISPOSITION_INFO, which I’ve never heard of before. I tried setting the disposition on the .pdb file in question, but that didn’t work – the file would not get deleted.
- I then started looking into undocumented functions and thought that maybe NtDeleteFile could do this kind of thing without throwing an “Access denied” error, but no dice.
No matter what you try, the file cannot be deleted because a file handle is still being held by another process, so let’s “fix” that.
Closing a handle in another process
In order to close any handle that is being held open by another process, we must first find the process ID of the process that’s holding the handle.
This can be done by using the Restart Manager (I know, judging by the name you wouldn’t think that thing could actually help us). Thankfully, Raymond Chen has some source code on his blog for solving this kind of problem.
With the PID in hand, we can now open the process using OpenProcess and PROCESS_DUP_HANDLE. Using the undocumented NtQuerySystemInformation, we can get a list of all handles in the system, filter the ones that don’t belong to our PID, filter everything that’s not a file, and use the (again, undocumented) NtDuplicateObject on the remaining handles. This allows us to query a handle’s information such as its name by using NtQueryObject.
Iterating over all handles and checking their name against the file name of the .pdb finally allows us to find the one handle we need to close – but how do we do that?
CloseHandle? Nope.
CloseRemoteHandle? CloseProcessHandle?
Of course not. After all, this is the Win32 API. You close a remote handle by calling DuplicateHandle passing the DUPLICATE_CLOSE_SOURCE flag. Makes sense.
One more puzzle
Anyway, after closing the handle, we free all the data related to our queries, free the handle to the remote process, and all should be good.
There are no more open handles to the .pdb file now, so let’s try again deleting the .pdb file.
DeleteFile()? Still doesn’t work.
Setting the file disposition? Same.
NtDeleteFile()? Same.
I was about to give up before I stumbled upon a mysterious FILE_FLAG_DELETE_ON_CLOSE flag which can be passed to CreateFile.
Now that there’s no process holding an open handle to the file anymore, we can get our own handle to it, and essentially trick the system.
All we have to do is open the file using CreateFile and FILE_FLAG_DELETE_ON_CLOSE, immediately closing it again using CloseHandle.
And guess what? Windows will happily delete the file for us!
Disclaimer
This is certainly what I would call… creative code.
Please bear in mind that pretty much none of the above is the recommended way of doing things according to Microsoft. You shouldn’t really touch handles that belong to another process. Neither should you use undocumented functions.
I haven’t been able to test this on all versions of Visual Studio, but it’s generic Win32 code, so it should work (famous last words).
By the way, some of those undocumented functions are actually declared in <Winternl.h> that ships with the Windows SDK. In case you were wondering, all the necessary functions are exported by ntdll.dll.
Molecular Musings 
I solved this problem the same way Casey Muratori did in Handmade Hero. I pass the -PDB parameter to the call to cl when I recompile the game DLL using %random% in the build batch file to generate a random file name. When VC++ loads the new DLL, it loads the new PDB file as well. That is probably safer and should have no hidden side effects. Forcing a file closed that another process has opened just makes me cringe, not to mention this seems like it might be undesirable functionality that might be patched in the future.
This is certainly a much nicer solution for cases such as runtime-compiled C++, and also what I use for C++ scripting, in conjunction with PDBALTPATH so I can load .dlls from temporary directories.
That’s why I wrote a disclaimer section :).
The question whether the file can actually be deleted came up on Twitter, and people wanted to know what workarounds others were using. The PDB, PDBALTPATH, and random/unique/hashed names workarounds is pretty much what everybody does, but I wanted to give people another (hacky) solution in case they’re willing to accept the advantages/disadvantages it brings.
I don’t advocate closing remote handles, but there may be cases where you are under the control of the environment, and can justify doing something like this.
As for MS actually fixing this? They introduced this bug in either VS2012 or VS2013 with the new debug engine, and it’s still broken in VS2015 & VS2017. There is a bug issued here, which is closed as “won’t fix”.
Thanks for replying. Just one thing to point out. When I said “this seems like it might be undesirable functionality that might be patched in the future” I didn’t mean the locked PDB file. I meant the ability to force a handle closed that another process has opened.
It is pretty ridiculous that we have to do anything at all though. Once the DLL is unloaded, the PDB file should be closed as well. As good as VC++ is overall, I’m surprised this is being ignored. Seems like they may have made poor design decisions in their debugger architecture. That’s for another article, I suppose. 🙂
Totally agree!
You would think that once the .dll and .pdb have been unloaded, and there’s no more code that can be run from the .dll, the debugger would no longer need the information stored in the .pdb.
From their point of view it’s not a bug, but a new feature. You should really file request at uservoice and get people to star it. It’s just bureaucracy of a big company.
Hi does still work with Windows 10?
Yes, it should.
when you say that you filter everything that’s not a file do you check the bObjectType field of SYSTEM_HANDLE_TABLE_ENTRY_INFO? That seems to change with every version of Windows.
Yes, that’s what I did in the original code.
You are right, this seems to change with different versions of Windows.
Hmm, I often run into Visual Studio (even in VS2019 with latest updates) holding open PDB files with my 120 project solution file. It drives me nuts. However rather than restarting Visual Studio and waiting ages while it loads the massive solution I simply rename the offending PDB files in the output directory. Under Win32 any files held open by a process (including DLL and EXE files held open by the loader like running programs) can always be renamed. The file remains open by the process and the system tracks the rename. I also use this technique sometimes when I want to replace a program or DLL while it is still executing – you obviously have to terminate and restart before the change takes effect. Renaming PDB’s held open by VS always seems to work correctly with the only downside being having to manually clean up those renamed files once VS is closed. Obviously it would be better if MS resolved the issue in the first place!!!