TL;DR – upgrade your tools, including Visual Studio, windbg, and Windows Performance Toolkit, if you want to handle Chromium’s symbol files.

Details:

Death, taxes, and browser engines relentlessly growing – those are the three things that you can really be certain of. And so it was in early 2020 when I realized that Chromium’s inexorable growth meant that we were eventually going to produce PDB (Windows debug symbol) files that exceeded the PDB format’s 4 GiB limit.

I filed a Visual Studio bug in February 2020 requesting that the limit be raised, and three years and three days later we flipped the switch so that Chromium can produce larger PDBs. At that point the PDB for Chrome was at 95% of 4 GiB, and several test binaries had already crossed over the threshold, so it was just in time.

My understanding of the PDB format is that it is page based. The format allows two to the 20th (2^20, or 1,048,576) pages, and the default page size is 4 KiB. These two numbers multiplied together give the maximum size of 4 GiB. The maximum number of pages can’t be increased, but the page size can be. In fact, the PDB format always recognized the need for different page sizes, but essentially no tools had ever supported a page size other than 4 KiB.

“All” that had to be done to support larger PDBs was to update a few tools to support larger pages. The tools that needed updating included:

• Visual Studio debugger
• link.exe – Microsoft’s linker
• lld-link – the linker used by Chromium
• windbg and other debuggers (kd.exe, ntsd.exe, etc.)
• dbghelp.dll (used for loading symbols)
• pdbstr.exe (used for source indexing)
• symstore.exe (used for uploading to symbol servers)
• msdia140.dll (COM API for loading symbols)
• Windows Performance Analyzer (WPA, the ETW trace viewer)
• Probably other tools

Easy!

According to comments on the bug an internal fix was released at Microsoft in August, 2021. By December this fix had shipped in updates to Visual Studio 2019 and in the just released Visual Studio 2022, and lld-link also supported it. Once the linkers supported larger page sizes we added a use_large_pdbs build setting which would switch the PDB page size to 8 KiB. However this setting initially needed to be off by default due to a lack of complete tool support.

Ideally the other tool updates would have shipped in the fall of 2021 but… they didn’t. I reached out to the Windbg and WPA teams and – though I could be wrong – I got the impression that they didn’t realize that they needed to ship updates until I told them.

There are various release channels for windbg (and the many tools associated with it) and WPA so it’s hard to say when fixes to these tools first shipped. Large PDB support appeared in the Microsoft Store versions sometime in 2022, and in nuget, but it wasn’t until fall 2022 when the Windows 11 22H2 SDK shipped that it was practical to update all of the versions of all of these tools.

The other thing that held us back was Windows 7. Newer versions of dbghelp.dll had problems running on Windows 7. So, as long as we were running tests on Windows 7 we had to generate PDBs that worked with the old versions of dbghelp.dll. Version 109 of Chrome is the last version that supports Windows 7 so as soon as we had branched for that version I could start working on large PDB support on trunk.

While working on the fix I hit a number of mysterious failures – you can see this by noticing that my first public attempt to create a change to switch to large PDBs was on December 28th – more than a month before the change landed. The first problem is that one test was stubbornly failing to load the updated PDB files. I gradually narrowed that down to the test failing to load the new version of dbghelp.dll. I then wasted a lot of time figuring out which DLL import was causing the problems before finally realizing that… the new dbghelp.dll was not being deployed to the test machines. It’s difficult to load a DLL that isn’t there. Hilariously, we had not been deploying dbghelp.dll to our test machines for years, but the system version was good enough so it hadn’t mattered, so this bug was never previously noticed. I deployed some pretty cool “loader snaps” diagnostics (based on this tool) but in the end it was overkill for “DLL not present” being the reason.

The next mysterious failure only happened on our official builders, which greatly complicated testing. This failure was because the wrong version of msdia140.dll was being loaded. In an almost perfect parallel to the first issue we had never properly been deploying msdia140.dll. I still don’t know where it had been loaded from before, but copying the right version to our output directory resolved that.

Finally, 37 days and 38 patch sets after uploading the first version of my change I landed it. The new PDB size limit is 8 GiB but we should be able to increase the page size to double this again anytime we want to.

Changes like this can easily be disruptive – it’s hard to anticipate all of the hidden dependencies or places this might fail – so I was pleased when the change landed and I heard almost nothing back. A couple of people complained that various tools couldn’t load Chrome Canary symbols but in all cases an update of the necessary tools to the latest Windows 11 SDK solved the problem. Note that the Windows SDK doesn’t install the Debuggers and Windows Performance Toolkit (which contains WPA) by default so if you depend on those tools you need to be sure to select them when installing or else you will still have the old versions.

Most people don’t work on Chromium, but if you run Chrome you may at some point want to be able to profile or debug the official builds you are running. You can configure your tools to point at Chrome’s symbol server and then symbols and source will be downloaded on demand – but only if you are running large-PDB compatible tools. I know that some game studios are also hitting the 4 GiB limit, so those developers also need the latest tools.

I’ve updated UIforETW so that it will install the latest version of WPA when you run it. You can find the latest release (currently 1.58) here.

Twitter announcement is here.

Hacker news discussion is here.