[NuGet.org Bug]: Symbol Server: Can't find pdbs bundled in nupkg via the Symbol Server API

[bruno-garcia]

Impact

I'm unable to use NuGet.org

Describe the bug

Unless packages are published via snupkg, using nuget.org's Symbol Server API doesn't result in finding symbols.

See: getsentry/symbolicator#1437

Repro Steps

Use the documented alternative to symbol packages:

https://learn.microsoft.com/en-us/dotnet/standard/library-guidance/nuget#symbol-packages

In other words, instead of publishing a separate snupkg, copy the pdb to the output directory to let it get copied into the nupkg next to the dll.

Google uses this approach here: https://github.com/googleapis/google-cloud-dotnet/blob/2f193795dd1daaba3ad6dd1f0f6b0c20c6597e6a/tools/DotnetToolProperties.xml#L8

Expected Behavior

When a debugger (or a crash reporting service like Sentry tries to fetch the debug symbols for a DLL loaded from nuget.org, it finds them and downloads them.

Unfortunately that doesn't happen. When using snupkg, it works as expected.

Screenshots

PDBs exist in the nupkg: https://nuget.info/packages/Google.Cloud.Storage.V1/4.10.0

Additional Context and logs

We use the Microsoft Symbol Server format to fetch symbols (worked for the other symbols on the screenshot above);

2024-04-17T15:21:07.275945Z DEBUG symbolicator_service::download::http: Fetching debug file from https://symbols.nuget.org/download/symbols/Google.Cloud.Storage.V1.pdb/A18B94CDA70745BEB0A2C316C2AB5314ffffffff/Google.Cloud.Storage.V1.pdb
[…]
2024-04-17T15:21:07.413918Z DEBUG symbolicator_service::download: Unexpected client error status code from https://symbols.nuget.org/download/symbols/Google.Cloud.Storage.V1.pdb/A18B94CDA70745BEB0A2C316C2AB5314ffffffff/Google.Cloud.Storage.V1.pdb: 404 Not Found

It's trying to download from that URL with an additional header of SymbolChecksum: SHA256:cd948ba107a7be65b0a2c316c2ab5314f8339cb1c32051c3138287f307a274d6. Assuming the identifiers and the checksum are all correct, where on the server should the file be found? How does snupkg vs nupkg translate into paths?

Quotes from this comment

[bruno-garcia]

Suggested solution: when processing nupkg, copy out PDBs and make them available same way as when ingesting snupkg packages. Before snupkg were the preferred way to upload symbols, "everyone" was doing the embedded pdb in nupkg, including Sentry for many years.

[joelverhagen]

Thanks for the suggestion, @bruno-garcia. I think this proposal makes sense in many ways. The data shows that active packages favor .snupkg -- 15% use .snupkg and 12% use "PDB in .nupkg".

In other words, both have similar adoption but .snupkg is on the rise and PDB is on the slight decline but still quite active. But as you say "PDB in .nupkg" is a documented alternative so perhaps it should have support that's just as good, or nearly as good, as .nupkg.

Some kinks that we'd need to work out are:

1. How do we manage Windows (non-portable) PDBs in .nupkgs?
   • Today, a .snupkg is rejected if it has non-portable PDBs. Would we reject .nupkgs with Windows PDBs? Or silently not ingest them into the symbol server? Open questions.
2. If a .nupkg has both PDBs in it and a .nupkg, which one "wins"? Do we just ingest all of the PDBs or just one set? Do we reject a .snupkg for a package that has PDBs in it?
3. Today .snupkg is mutable in the sense that you can delete and replace the .snupkg on an existing version. Is this scenario important for .nupkg based PDBs which are immutable today?

Perhaps another option is to improve the debugger client to know about PDBs in the NuGet global package folder. This makes sense for cases when you're debugging a project that depends on these other DLLs (e.g. Google APIs) as package reference. But what if you're debugging an entry point without project context? For example windbg can read from a symbol server like NuGet.org's symbol server (theoretically). In cases like that the NuGet global packages folder may not be populated with the packages associated with the runtime dependencies you're looking at.

There is also the perspective that other package sources may not want to implement the .snupkg ingestion flow but do want to implement a symbol server. In such cases they may very well want to ingest PDBs from .nupkg (or that cursed old .symbols.nupkg artifacts). Should we enhance the metadata in the NuGet V3 protocol to allow package sources to declare how they treat PDBs in .nupkg and/or .snupkg? It's possible this question extends beyond NuGet.org.

[bruno-garcia]

Some kinks that we'd need to work out are:

1. How do we manage Windows (non-portable) PDBs in .nupkgs?

   • Today, a .snupkg is rejected if it has non-portable PDBs. Would we reject .nupkgs with Windows PDBs? Or silently not ingest them into the symbol server? Open questions.

PPDB has been default on all flavors of .NET for many years now, I imagine silently not ingesting would be OK. That said further analysis on % of pdbs affected before committing to this decision might be the safer route.

2. If a .nupkg has both PDBs in it and a .nupkg, which one "wins"? Do we just ingest all of the PDBs or just one set? Do we reject a .snupkg for a package that has PDBs in it?

I imagine this isn't a common scenario but it's a fair point. I imagine it doesn't matter which one wins because the way this could happen is if an app is configured to write pdb to the bin folder and packages it, while also creating snupkg. So if one overrides the other, or if the nupkg extraction doesn't happen because before doing so it checks of snupkg already exists, either way would be fine.

3. Today .snupkg is mutable in the sense that you can delete and replace the .snupkg on an existing version. Is this scenario important for .nupkg based PDBs which are immutable today?

tbqh I don't know why one would need to make symbols mutable. They come out of the compiler with the executable and if you can't change the executable, why do that for the symbols? Here again (prob due to my ignorance on the subject) I imagine it's fine if snupkg are mutable while the extracted ppdb are not. Or "push a snupkg to override it" if that's the behavior decided above.

Perhaps another option is to improve the debugger client to know about PDBs in the NuGet global package folder. This makes sense for cases when you're debugging a project that depends on these other DLLs (e.g. Google APIs) as package reference. But what if you're debugging an entry point without project context? For example windbg can read from a symbol server like NuGet.org's symbol server (theoretically). In cases like that the NuGet global packages folder may not be populated with the packages associated with the runtime dependencies you're looking at.

The use case I'm thinking of (which is the one that affected my ability to debug things since I had no line numbers) was a live/prod app. Sentry captures the stack trace, and on the server (sentry itself, not a .NET app) looks up symbols via symbol servers.

There is also the perspective that other package sources may not want to implement the .snupkg ingestion flow but do want to implement a symbol server. In such cases they may very well want to ingest PDBs from .nupkg (or that cursed old .symbols.nupkg artifacts). Should we enhance the metadata in the NuGet V3 protocol to allow package sources to declare how they treat PDBs in .nupkg and/or .snupkg? It's possible this question extends beyond NuGet.org.

I don't have strong opinions here, I imagine folks on the NuGet team will know best what's the ideal route here

[loic-sharma]

It would be interesting to consider how we could make .NET debuggable by default. Today, there are many gaps in the experience. See NuGet/Home#10793 for another relevant discussion.

Should we enhance the metadata in the NuGet V3 protocol to allow package sources to declare how they treat PDBs in .nupkg and/or .snupkg?

I don't think we need additional metadata in the V3 protocol for this. In theory it is completely valid for a third-party NuGet server to ingest portable PDBs in a .nupkg:

1. Windows PDBs are inherently insecure. Symbol servers shouldn't expose these for .NET DLLs.
2. From my understanding, the symbol server protocol is basically a content-addressable store & there is only up to one valid portable PDB per DLL (the DLL contains the checksum of its portable PDB). If this is correct, the worst case scenario for ingesting all portable PDBs is that you have wasted space for portable PDBs that will never be downloaded.