Build Debs using our own infra instead of debuild + custom scripting

[jkoritzinsky]

Today, we use some scripting built around debuild to build our deb packages. This approach has a few problems:

• debuild is only available on Debian-based systems, forcing us to package our deb packages in a different container than we build in.
• debuild has a lot of features based around building for the current distro, which we have to work around every so often (Add mechanism to disable shlibs discovery for particular shared libraries #13730)
• debuild assumes that it's targeting the current distro, so sometimes it uses newer features that older distros don't know about, which we only discover in validation runs (Package debs with xz compression #13871)
• debuild's lack of cross-build support makes ARM64 debs difficult to produce.
• The custom scripting we have goes through MSBuild -> JSON -> Bash and is not well-understood.

This PR attempts a new approach: Build Debs from the various components that make up a Deb package.

A deb package is a remarkably simple format. It's an ar archive with 3 files in it. One has one line of text. The other two are (compressed) tarballs. Within the first tarball are text files with well documented formats. Within the second tarball is the actual data as well as some generated files that follow specific formats.

This approach allows us to ensure we don't accidentally adopt new features in the format that aren't supported by old distros.

It also doesn't implement the rest of the package building support that we don't use and have to work around.

Since this doesn't depend on debian-only tools, we can build our packages on Azure Linux and for arm64 without significant trouble.

Also, having our own library to support the "ar" format also solves the biggest part of #14436.

Validated locally that (with the changes in dotnet/runtime#107429) produces identical package contents and metadata (just with gzip compression instead of xz compression).

Blocked on dotnet/runtime#107429

[am11]

Maybe we can include https://github.com/dotnet/sdk/tree/main/documentation/manpages (which we can update at the end of October like did last year).

[jkoritzinsky]

I'd need to look more at that tooling to figure out how it works and how to integrate it well. I like the idea

[NikolaMilosavljevic]

This is great for DEBs produced in runtime. sdk and aspnetcore won't benefit from this until they start using Arcade infra for DEB (and RPM) builds.

cc @wtgodbe @joeloff

[joeloff]

Thanks @NikolaMilosavljevic So once this change flows to SDK we could switch to using this? @baronfel as an FYI

[mmitche]

Absolutely fantastic. Taking a look.

[mmitche]

/cc @ellahathaway

[jkoritzinsky]

Looking at dotnet/sdk, it actually directly uses the existing tooling with its own task (instead of our MSBuild targets).

As a result, I'll need to restore the old tooling just to keep in the package for now.

[ericstj]

I wonder if this functionality would also be helpful for the signing work @mmitche

[jkoritzinsky]

@ericstj My initial reasoning for doing this work was for signing, as it will enable us to do the unwrapping/re-wrapping of debs easily (like how we do for tarballs and zips).

[mmitche]

@ericstj My initial reasoning for doing this work was for signing, as it will enable us to do the unwrapping/re-wrapping of debs easily (like how we do for tarballs and zips).

Yeah, this is absolutely useful.