There are certain use-cases where .dlls are going to be unloaded, recompiled, and then loaded again (think Runtime-Compiled C++), so keeping a lock onto the .pdb file renders the linker unable to produce a new file, which is bad. I stumbled upon this problem in the past already and circumvented the issue by generating .dlls and .pdbs with unique names on each recompile.
However, for the tool I’m currently working on, I wanted to take another stab at “fixing” the root issue: deleting the old .pdb file, so the linker can produce a new one.
But how do we delete a file that’s locked by another process?
Things that won’t work
- For obvious reasons, DeleteFile() won’t work and returns an “Access denied” error.
- Examining file-related activity using ProcMon led me to SetFileInformationByHandle and a thing known as FILE_DISPOSITION_INFO, which I’ve never heard of before. I tried setting the disposition on the .pdb file in question, but that didn’t work – the file would not get deleted.
- I then started looking into undocumented functions and thought that maybe NtDeleteFile could do this kind of thing without throwing an “Access denied” error, but no dice.
No matter what you try, the file cannot be deleted because a file handle is still being held by another process, so let’s “fix” that.
Closing a handle in another process
In order to close any handle that is being held open by another process, we must first find the process ID of the process that’s holding the handle.
This can be done by using the Restart Manager (I know, judging by the name you wouldn’t think that thing could actually help us). Thankfully, Raymond Chen has some source code on his blog for solving this kind of problem.
With the PID in hand, we can now open the process using OpenProcess and PROCESS_DUP_HANDLE. Using the undocumented NtQuerySystemInformation, we can get a list of all handles in the system, filter the ones that don’t belong to our PID, filter everything that’s not a file, and use the (again, undocumented) NtDuplicateObject on the remaining handles. This allows us to query a handle’s information such as its name by using NtQueryObject.
Iterating over all handles and checking their name against the file name of the .pdb finally allows us to find the one handle we need to close – but how do we do that?
Of course not. After all, this is the Win32 API. You close a remote handle by calling DuplicateHandle passing the DUPLICATE_CLOSE_SOURCE flag. Makes sense.
One more puzzle
Anyway, after closing the handle, we free all the data related to our queries, free the handle to the remote process, and all should be good.
There are no more open handles to the .pdb file now, so let’s try again deleting the .pdb file.
DeleteFile()? Still doesn’t work.
Setting the file disposition? Same.
I was about to give up before I stumbled upon a mysterious FILE_FLAG_DELETE_ON_CLOSE flag which can be passed to CreateFile.
Now that there’s no process holding an open handle to the file anymore, we can get our own handle to it, and essentially trick the system.
All we have to do is open the file using CreateFile and FILE_FLAG_DELETE_ON_CLOSE, immediately closing it again using CloseHandle.
And guess what? Windows will happily delete the file for us!
This is certainly what I would call… creative code.
Please bear in mind that pretty much none of the above is the recommended way of doing things according to Microsoft. You shouldn’t really touch handles that belong to another process. Neither should you use undocumented functions.
I haven’t been able to test this on all versions of Visual Studio, but it’s generic Win32 code, so it should work (famous last words).
By the way, some of those undocumented functions are actually declared in <Winternl.h> that ships with the Windows SDK. In case you were wondering, all the necessary functions are exported by ntdll.dll.