No source context even though source bundles available

[bruno-garcia]

While adding new Sentry features to Sonarr I noticed I can't get source context to show up even though I uploaded PDBs and source bundles:

It's only broken using the Sentry CLI Source Bundles feature.
If I try to embed sources in the .NET PDB, it actually works.

Event in Sentry: https://bruno-garcia.sentry.io/issues/4647134873/?imageCodeId=65594f7b12000&imageDebugId=b951abf3-1d87-42a9-a886-65880caa884c-e185dcc0&project=4506249065332736&query=is%3Aunresolved&referrer=issue-stream&statsPeriod=14d&stream_index=0

PDB (renamed to txt so GH lets me upload it):
5cd89838-5d43-4156-ae7d-cdceb9c721a0-eee65b5b.pdb.txt

DLL (renamed to txt so GH lets me upload it):
Sonarr.dll.txt

Raw JSON

{"event_id":"6d022274fc1746ed9713d076c97ea06f","project":4506249065332736,"release":"10.0.0.34259-Release-dev","dist":null,"platform":"csharp","message":"EPIC FAIL!","datetime":"2023-11-19T00:03:18+00:00","tags":[["app.device","177a8117"],["branch","Release-dev"],["culture","en-CA"],["environment","testing"],["handled","no"],["level","fatal"],["logger","ConsoleApp"],["mechanism","Logger.Fatal"],["os.name","Darwin"],["runtime",".NET 6.0.25"],["runtime.name",".NET"],["release","10.0.0.34259-Release-dev"],["user","id:177a8117"]],"_metrics":{"bytes.ingested.event":6219,"bytes.stored.event":9930},"breadcrumbs":{"values":[{"timestamp":1700352198.955,"type":"default","category":"ConsoleApp","level":"fatal","message":"EPIC FAIL!"}]},"contexts":{"Current Culture":{"calendar":"GregorianCalendar","display_name":"English (Canada)","name":"en-CA","type":"Current Culture"},"Dynamic Code":{"Compiled":true,"Supported":true,"type":"Dynamic Code"},"Memory Info":{"allocated_bytes":4188016,"compacted":true,"concurrent":false,"finalization_pending_count":15,"fragmented_bytes":1193504,"heap_size_bytes":2440280,"high_memory_load_threshold_bytes":30923764531,"index":1,"memory_load_bytes":20615843020,"pause_durations":[2.693,0],"pause_time_percentage":0.32,"pinned_objects_count":3,"promoted_bytes":730424,"total_available_memory_bytes":34359738368,"total_committed_bytes":2670592,"type":"Memory Info"},"ThreadPool Info":{"available_completion_port_threads":1000,"available_worker_threads":32767,"max_completion_port_threads":1000,"max_worker_threads":32767,"min_completion_port_threads":10,"min_worker_threads":10,"type":"ThreadPool Info"},"app":{"app_start_time":"2023-11-19T00:03:18.259005+00:00","device_app_hash":"177a8117","app_name":"Sonarr","app_version":"10.0.0.34259","app_build":"10.0.0.34259-Release-dev","type":"app"},"device":{"boot_time":"2023-11-13T19:18:34.4450099+00:00","timezone":"Canada/Eastern","timezone_display_name":"(UTC-05:00) Eastern Time (Toronto)","type":"device"},"os":{"name":"Darwin","kernel_version":"23.0.0","raw_description":"Darwin 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:43 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6000","type":"os"},"runtime":{"name":".NET","version":"6.0.25","raw_description":".NET 6.0.25","identifier":"osx.14-x64","type":"runtime"},"trace":{"trace_id":"8fd1e02cc68e40d6966aa7401ac23572","span_id":"3ce1ad27c7f8af84","status":"unknown","type":"trace"}},"culprit":"ConsoleApp.cs in void ConsoleApp.AddFrame()","debug_meta":{"images":[{"code_id":"6559508d12000","code_file":"/Users/bruno/git/Sonarr/_output/net6.0/Sonarr.dll","debug_id":"5cd89838-5d43-4156-ae7d-cdceb9c721a0-eee65b5b","debug_file":"/Users/bruno/git/Sonarr/_temp/obj/Sonarr.Console/Release/net6.0/Sonarr.pdb","debug_checksum":"SHA256:3898d85c435d5661ee7dcdceb9c721a05b5be66e565f5ea2ee1b756308797b32","image_addr":"0x0","candidates":[{"download":{"status":"notfound"},"source":"sentry:android","source_name":"Android"},{"download":{"status":"notfound"},"source":"sentry:ios","source_name":"Apple"},{"download":{"status":"notfound"},"source":"sentry:microsoft","source_name":"Microsoft"},{"download":{"status":"notfound"},"source":"sentry:nuget","source_name":"NuGet.org"},{"debug":{"status":"ok"},"download":{"features":{"has_debug_info":true,"has_sources":true,"has_symbols":false,"has_unwind_info":false},"status":"ok"},"location":"sentry://project_debug_file/494559760","source":"sentry:project","source_name":"Sentry"}],"debug_status":"found","features":{"has_debug_info":true,"has_sources":true,"has_symbols":false,"has_unwind_info":false},"type":"pe_dotnet"}]},"environment":"testing","exception":{"values":[{"type":"System.Exception","value":"Crash with symbols uploaded","module":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","stacktrace":{"frames":[{"function":"void ConsoleApp.Main(string[] args)","package":"Sonarr, Version=10.0.0.34270, Culture=neutral, PublicKeyToken=null","filename":"ConsoleApp.cs","abs_path":"./Sonarr.Console/ConsoleApp.cs","lineno":45,"colno":17,"in_app":true,"data":{"symbolicator_status":"symbolicated"},"instruction_addr":"0x30","addr_mode":"rel:0","function_id":"0x1"},{"function":"void ConsoleApp.AddFrame()","package":"Sonarr, Version=10.0.0.34270, Culture=neutral, PublicKeyToken=null","filename":"ConsoleApp.cs","abs_path":"./Sonarr.Console/ConsoleApp.cs","lineno":99,"colno":13,"in_app":true,"data":{"symbolicator_status":"symbolicated"},"instruction_addr":"0xa","addr_mode":"rel:0","function_id":"0x2"}]},"raw_stacktrace":{"frames":[{"function":"void ConsoleApp.Main(string[] args)","package":"Sonarr, Version=10.0.0.34270, Culture=neutral, PublicKeyToken=null","filename":"./Sonarr.Console/ConsoleApp.cs","abs_path":"./Sonarr.Console/ConsoleApp.cs","lineno":45,"colno":17,"in_app":true,"instruction_addr":"0x30","addr_mode":"rel:0","function_id":"0x1"},{"function":"void ConsoleApp.AddFrame()","package":"Sonarr, Version=10.0.0.34270, Culture=neutral, PublicKeyToken=null","filename":"./Sonarr.Console/ConsoleApp.cs","abs_path":"./Sonarr.Console/ConsoleApp.cs","lineno":99,"colno":13,"in_app":true,"instruction_addr":"0xa","addr_mode":"rel:0","function_id":"0x2"}]},"thread_id":1,"mechanism":{"type":"Logger.Fatal","description":"Logger.Fatal was called","handled":false}}]},"fingerprint":["Fatal","ConsoleApp","EPIC FAIL!","System.Exception","Void AddFrame()"],"grouping_config":{"enhancements":"eJybzDRxc15qeXFJZU6qlZGBkbGugaGuoeEEAHJMCAM","id":"newstyle:2023-01-11"},"hashes":["4059f401df74a0b2dffb999001217ea2"],"ingest_path":[{"version":"23.11.0","public_key":"XE7QiyuNlja9PZ7I9qJlwQotzecWrUIN91BAO7Q5R38"}],"key_id":"3491023","level":"fatal","location":"ConsoleApp.cs","logentry":{"formatted":"EPIC FAIL!"},"logger":"ConsoleApp","metadata":{"display_title_with_tree_label":false,"filename":"ConsoleApp.cs","function":"void ConsoleApp.AddFrame()","in_app_frame_mix":"in-app-only","type":"System.Exception","value":"Crash with symbols uploaded"},"modules":{"0Harmony":"2.0.1.0","Microsoft.AspNetCore.Connections.Abstractions":"6.0.0.0","Microsoft.Extensions.Logging":"6.0.0.0","Microsoft.Extensions.Logging.Abstractions":"6.0.0.0","Microsoft.Win32.Primitives":"6.0.0.0","NLog":"4.0.0.0","NLog.Extensions.Logging":"1.0.0.0","NLog.Targets.Syslog":"6.0.0.0","Sentry":"4.0.0.0","Sonarr":"10.0.0.34270","Sonarr.Common":"10.0.0.34259","Sonarr.Host":"10.0.0.34270","Sonarr.RuntimePatches":"10.0.0.34260","System.Collections":"6.0.0.0","System.Collections.Concurrent":"6.0.0.0","System.Collections.Immutable":"6.0.0.0","System.Collections.NonGeneric":"6.0.0.0","System.ComponentModel":"6.0.0.0","System.ComponentModel.Primitives":"6.0.0.0","System.ComponentModel.TypeConverter":"6.0.0.0","System.Console":"6.0.0.0","System.Data.Common":"6.0.0.0","System.Data.SQLite":"1.0.115.5","System.Diagnostics.DiagnosticSource":"6.0.0.0","System.Diagnostics.FileVersionInfo":"6.0.0.0","System.Diagnostics.Process":"6.0.0.0","System.Diagnostics.StackTrace":"6.0.0.0","System.Diagnostics.TraceSource":"6.0.0.0","System.Diagnostics.Tracing":"6.0.0.0","System.IO.Compression":"6.0.0.0","System.IO.FileSystem.Watcher":"6.0.0.0","System.IO.MemoryMappedFiles":"6.0.0.0","System.IO.Pipes":"6.0.0.0","System.Linq":"6.0.0.0","System.Linq.Expressions":"6.0.0.0","System.Memory":"6.0.0.0","System.Net.Http":"6.0.0.0","System.Net.Mail":"6.0.0.0","System.Net.NameResolution":"6.0.0.0","System.Net.Primitives":"6.0.0.0","System.Net.Requests":"6.0.0.0","System.Net.Security":"6.0.0.0","System.Net.Sockets":"6.0.0.0","System.Numerics.Vectors":"6.0.0.0","System.ObjectModel":"6.0.0.0","System.Private.CoreLib":"6.0.0.0","System.Private.Uri":"6.0.0.0","System.Reflection.Emit.ILGeneration":"6.0.0.0","System.Reflection.Emit.Lightweight":"6.0.0.0","System.Reflection.Metadata":"6.0.0.0","System.Reflection.Primitives":"6.0.0.0","System.Runtime":"6.0.0.0","System.Runtime.CompilerServices.Unsafe":"6.0.0.0","System.Runtime.InteropServices":"6.0.0.0","System.Runtime.InteropServices.RuntimeInformation":"6.0.0.0","System.Runtime.Intrinsics":"6.0.0.0","System.Security.Cryptography.Algorithms":"6.0.0.0","System.Security.Cryptography.Primitives":"6.0.0.0","System.Security.Cryptography.X509Certificates":"6.0.0.0","System.Text.Encoding.Extensions":"6.0.0.0","System.Text.Encodings.Web":"6.0.0.0","System.Text.Json":"6.0.0.0","System.Text.RegularExpressions":"6.0.0.0","System.Threading":"6.0.0.0","System.Threading.Thread":"6.0.0.0","System.Threading.ThreadPool":"6.0.0.0","netstandard":"2.1.0.0"},"nodestore_insert":1700352201.99307,"received":1700352199.830256,"sdk":{"name":"sentry.dotnet","version":"4.0.0-beta.0","packages":[{"name":"nuget:sentry.dotnet","version":"4.0.0-beta.0"}]},"timestamp":1700352198.989665,"title":"System.Exception: Crash with symbols uploaded","type":"error","user":{"id":"177a8117"},"version":"7"}

[bruno-garcia]

@vaind since you looked at this, the fact that works on bundled code in the PDB but not in source bundles, any hints where the problem might be or where to look?

[zhuxb711]

I also met this issue. C# source bundles available through setting "SentryUploadSources" on MSBuild but in the end no source available for the stack frame.

[vaind]

This has slipped through the cracks. I'll check what's going on. At the first glance, I'd say the debug symbols (PDBs) uploaded already contain some source so the source bundle is not used at all.

[zhuxb711]

@vaind Yes I think so. Sentry said that the source is bundled with pdb file and have 'source' mark on it. But somehow did not used at all.

[vaind]

@vaind Yes I think so. Sentry said that the source is bundled with pdb file and have 'source' mark on it. But somehow did not used at all.

If I remember correctly, PDBs often have source code blocks for generated code, even though you don't enable EmbedAllSources in your csproj. And the symbolicator will only consider the first debug file that has sources and won't look for another one.

This is the relevant code in the Symbolicator:

https://github.com/getsentry/symbolicator/blob/2474a74225d1b8488d909e0441c1996c0078b6af/crates/symbolicator-native/src/symbolication/module_lookup.rs#L312-L333

Notably from objects.find() docs: " /// Fetches matching objects and returns the metadata of the most suitable object." - it returns a single object.

And I've just found out this issue is a duplicate of getsentry/symbolicator#1112

[zhuxb711]

Hope this issue would be able to resolve soon~ Thanks @vaind