shim-15.8 for WAXAR

[Jurij-Ivastsuk]

Confirm the following are included in your repo, checking each box:

• completed README.md file with the necessary information
• shim.efi to be signed
• public portion of your certificate(s) embedded in shim (the file passed to VENDOR_CERT_FILE)
• binaries, for which hashes are added to vendor_db ( if you use vendor_db and have hashes allow-listed )
• any extra patches to shim via your own git tree or as files (we use no extra patches)
• any extra patches to grub via your own git tree or as files (no extra patches, we use Canonical upstream for GRUB)
• build logs
• a Dockerfile to reproduce the build of the provided shim EFI binaries

---

What is the link to your tag in a repo cloned from rhboot/shim-review?

---

https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20241129

---

What is the SHA256 hash of your final SHIM binary?

---

SHA256 (shimx64.efi): 1894c7e467991c117162e1e40dd61ed85a5f790a68216fdbee93b2619b70df67
SHA256 (shimaa64.efi): 5c7238473401d791c7f376efee90cf1e9fe23e2bf1814f4795474d57920bdfbf

---

What is the link to your previous shim review request (if any, otherwise N/A)?

---

N/A

[aronowski]

Hello.

I'll start reviewing the application soon, but the first requirement is that all the checkboxes are checked and render well.

Please, fix the formatting and check the missing one - it's supposed to mean "if we have any extra patches, then they have been included."

[Jurij-Ivastsuk]

@aronowski Hi, many thanks for any tips!
I literally understood point 6 as "if you have extra patches for GRUB..." Since we use the GRUB software upstream from Canonical, and don't make any changes there, with patches or otherwise, I figured I shouldn't mark this point as done.

[aronowski]

@Jurij-Ivastsuk, thank you!

I agree that some part in here may not be clear, but I'll try my best to address them. Some documents I've been working on improving during my free time, but haven't requested to incorporate them so far.

Reviewing also may not be easy especially when dealing with environments I'm not that familiar with or analyzing patches for correctness (took me about 8 hours to spot one major error, but I did it). If there's no activity e.g. by the end of next week, ping me.

[Jurij-Ivastsuk]

@aronowski, I understand and I thank you for the work you do in your spare time!

[aronowski]

Hello.

I did scratch the surface of the application. There are some curiosities and things that need to be fixed. I address them as some loose thoughts below.

In the meantime, I'm sending contact verification emails soon. Please, post the random words, that I'm emailing you, in the comments.

---

The NX compatibility bit should be enabled only if the whole bootchain is NX compatible - that's the requirement Microsoft has, as I'm aware right now. Therefore, the patch 530 shall not be used.

Furthermore, the validation function patch does its job in regard to older requirements, which have changed - there's no need to use it today.

Please, update the binaries and the building process as described - I'll then try to rebuild it and check if everything is correct.

---

The 626.patch PR hasn't been merged and debugging the code in my head is out of the question. It's just a one-line change, but no idea about the consequences and possible corner-cases, especially since I'm not a low-level programmer.

I'm not saying that it's unable to be approved as-is - just that I'll need some help with it and wondering, how to get some.

---

Linux required upstream commits - are they applied for the 5.15.1 Debian kernel?

The commit eadb2f47a3ced5c64b23b90fd2a3463f63726066 as far as I know got introduced in Linux v5.19. Is it backported in Debian?

---

There's no ephemeral key - is the strategy sufficient, as it's described right now?

It would be preferable to mention the usage of CONFIG_MODVERSIONS=y and CONFIG_MODULE_SIG_FORCE=y. Furthermore, is a new key being used to sign each new kernel build?

---

The CA certificate waxar.ceris valid for almost 100 years and it's only 2048 bit.

I don't know if Microsoft would be willing to approve this - any chances of generating one with a shorter lifespan and/or using 4096 bit public modulus?

[aronowski]

Verification emails sent.

[Jurij-Ivastsuk]

I have received your email. Here is the decrypted text:
summerhouses ants regionally announces unmasking fusing addling bantered
sprouted chiropractic

[Jurij-Ivastsuk]

@aronowski Hi, thank you very much for the first hints!
Now I have changed the following:

1. patches 530 and 531 (NX compatibility) have been removed
2. docker file adapted accordingly
3. CA certificate waxar.cer is now valid for 50 years and its size is now 4096 byte
4. for kernel building we use CONFIG_MODVERSIONS=y and CONFIG_MODULE_SIG_FORCE=y
5. the new certificate is already used for signing kernel build as well as for signing all modules
   Regarding commit eadb2f47a3ced5c64b23b90fd2a3463f63726066 (CVE-2022-21499) in Debian, I have found the following information:
   bullseye 5.10.197-1 fixed
   bullseye (security) 5.10.205-2 fixed
   bullseye 5.10.120-1 fixed
   bookworm 6.1.66-1 fixed
   Exactly for kernel 5.15.1 no Information could be found...
   As for 626.patch, I understand that you cannot be responsible for all matters. Maybe you can ping someone who does low-level programming and who can properly assess our PR request. Actually I explained the problem and the solution pretty well. Without this patch we can't use shim like this because it keeps giving us errors (screenshot and explanation in PR 626).

[Jurij-Ivastsuk]

@aronowski Hi, should we now switch to 15.8 ?

[aronowski]

Updating should extend the lifecycle of the review if something happened and if reviews from other people from the committee got delayed. I'd do so, especially since I just scratched the surface and will perform a further review, after the latest fixes have been applied.

[Jurij-Ivastsuk]

@aronowski Hi,
the second contact (Lukas Kienbaum) have received your email (shim-words x2). Here is the decrypted text:
hauled multiplication Collins reemerge exhilarates fathers Vijayawada
stubbiest fête refile

[Jurij-Ivastsuk]

@aronowski Hi, did I understand you correctly, we are waiting until the last reviews from the committee are closed, then you will continue with your review with us? I still don't understand if we will stay with version 15.7 or will we logically switch to 15.8. In this case, if I understand correctly, a new review "shim-15.8 for WAXAR" will be requested. Is that the case?

[aronowski]

@Jurij-Ivastsuk

Please, update all the required things to reflect shim 15.8. I'll then perform further review and ask other reviewers for help (you can ping them as well).

---

we are waiting until the last reviews from the committee are closed, then you will continue with your review with us?

I'll try my best when there's optimal environment for me - for instance, the comment from January 22 I posted in the middle of the night in my timezone, as I had some free time and no other things on my mind.

Still, there are other things that can make some applications being reviewed ASAP and some that wait for, let's say, a month or even more. The main thing is that I'm just more familiar with certain distros and their environments.

Some things I gathered as part of the improvements I'd like to introduce to the project, so they are blatantly visible, e.g. here.

---

CA certificate [...] size is now 4096 byte

4096 byte? ;-)

[Jurij-Ivastsuk]

@aronowski Hi, thank you very much for the tips and explanations! I really like your suggestions for improvements. I have to apologize for the incorrect representation of the certificate size: of course it's bits and not bytes. So the CA certificate size is now 4096 bits... ;-)))
I'll let you know when I've made all the adjustments for 15.8.

[Jurij-Ivastsuk]

@aronowski Hi, I have a question, can you please explain to me why the CA certificate must be 4096 bit in size and not 2048 bit?

[aronowski]

@Jurij-Ivastsuk, it's not like it must be this or that, but instead a good practice to follow the recommendations from NIST.SP.800-57pt1r5 - table no. 2 on page 54 and table no. 4 on page 59.

I'll review all the updates soon, once I get more free time and some well-deserved sleep.

[Jurij-Ivastsuk]

@aronowski Hi, please only continue with review if you have slept enough. i have made all adjustments for version 15.8 ;-))

[aronowski]

@Jurij-Ivastsuk,

Exactly for kernel 5.15.1 no Information could be found...

Apologies, that's the Ubuntu Jammy kernel, rather than Debian's as I thought earlier.

So I think our best bet is to wait for an official answer from Canonical kernel maintainers. Please see #368 (comment).

---

please only continue with review if you have slept enough

Let's say 6 hours is enough for now. :-P

I was checking things out and got stuck with the kernel-related thing. Let's hope it gets resolved soon.

[steve-mcintyre]

@Jurij-Ivastsuk Simply deleting old kernel modules from a system is not enough for revocation. You need to have technical measures in place to block them from loading.

There's a few issues here where it's not clear from your submission that you understand how things are meant to work.

[Jurij-Ivastsuk]

@THS-on , @aronowski
Dear @steve-mcintyre thank you for your comments. Our Linux environment is located on the external boot device like CD, USB stick or USB drive (like live-CD). The boot process is initiated by selecting Boot Device from the boot menu.

• When we upgrade our entire system, we have full control over kernel upgrade, modules upgrade in system and modules upgrade in initramfs. Therefore, when we upgrade - we upgrade the entire system, starting from bootloader stage1 and stage2, kernel, initramfs up to all system components and modules. All modules in initramfs and in the system tree are linked to the corresponding kernel. Therefore, I think the treatment of modules from the security perspective is slightly different in our case.

• As I have already mentioned, the simplest mechanism to prohibit a kernel to load older kernel modules, is to use an ephemeral key during build time, but due to the restrictions in our case we cannot use them. Theoretically, there is another possibility to block the loading of old modules, namely to use the blacklisting procedure for modules. But in our case this procedure would not be applicable because the names of most modules remain the same during the transition to a new version.

• Finally, we have a mechanism designed to exclude unwanted modules from loading - signing the modules with a unique license key at compile time. This is the purpose of signing. All modules that do not have the correct license key are ignored.

There's a few issues here where it's not clear from your submission that you understand how things are meant to work

I can't say that I know everything, but unfortunately I can't do anything with this statement of yours. I would be very happy to learn something new from you. If you think you know something more in this context, please share your knowledge with everyone here who is in a technically similar situation and would like to improve their solutions.

[Jurij-Ivastsuk]

Hello dear reviewer @aronowski @THS-on @steve-mcintyre, to answer once again your question about the prevention of loading unwanted modules: we use our own certificate to check the signatures of the modules to be loaded. Our kernel can also be delivered with new keys, if we need it, to prevent old modules from being loaded.

[aronowski]

I'll raise the issue during the next call.

---

If you think you know something more in this context, please share your knowledge with everyone here who is in a technically similar situation and would like to improve their solutions.

I myself was thinking about writing a guide, how to start the shim-related venue from scratch and implement appropriate measures for reasonable security, but taking into account each company's distinct situation, but there's no way I'll be able to account for so many different cases and limitations. Especially doing this in a free time after job hours and considering my own knowledge, experience and limitations.

I could think of something similar to the venue from this thread about smaller companies (one referred there as a fictional MyCompany) and their products deriving from the larger ones, but in the context of shim implementation and the whole bootloader chain (referred as Using downstream implementations from Canonical in your application). Still, as said earlier, I can see too many distinct cases to make it reasonable.

[THS-on]

@Jurij-Ivastsuk regarding the kernel module signing:

• If you need the way to build kernel modules after the initial build, you can include additional kernel specific certificates per each build. Note that it is highly recommended to protect those keys in an HSM or similar. The main reason right now for accepting submissions without ephemeral key signing, is already products that have been signed in the past and when they have a sufficient mitigation strategy.
• How your build is currently documented (https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/blob/waxar-shim-x86_64-aarch64-20240601/Setup_for_module_signing.txt) it looks like the signing keys are in plain text on some build machine and emphermal, which is not sufficient.
• Please elaborate on the implementation with the license key and module signing. I currently don't see the connection to the kernel module loading mechanism

[Jurij-Ivastsuk]

@THS-on Hello, thank you very much for your comments. In our current kernel-mudule signing-setup we currently use two different certificates: 1. our own CA certificate and 2. kernel specific certificate for each compilation.
Can you please explain what you mean by “license key”. Is this a third variant of the certificate?

[THS-on]

In our current kernel-mudule signing-setup we currently use two different certificates: 1. our own CA certificate and 2. kernel specific certificate for each compilation

Can you update your documentation to reflect that? And also describe how your procedure with those keys are in detail?

Can you please explain what you mean by “license key”. Is this a third variant of the certificate?

This refers to your third point from this comment #362 (comment)

[Jurij-Ivastsuk]

Hello @THS-on
Thank you very much for your comments and suggestions.

In order to implement the realization of the “completely secure boot” principle, we have developed an approach with double signing of certain boot components using HSM. We combine the signing of the kernel and modules once with our CA certificate (for kernel) and additionally with unique private key + CA certificate (for modules). This approach for signing the modules is more secure compared to signing only with a unique private key. Additional use of HSM gives us an even higher protection for security because both private keys are only in HSM and never appear as physical files.

We have adapted and updated our setup documentation in line with your comments. Please see here: (https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/blob/waxar-shim-x86_64-aarch64-20240804/Setup_for_module_signing.txt)

[THS-on]

Thanks for the update. Some questions:

• What kind of HSM are you using and who has access to it?
• You seem to sign the installed files, which seems weird. How do you package your kernel? As you are Debian based, have a look at what either Debian or Ubuntu is doing for their kernel packaging
• How does your CA + leaf certificates structure look like?
• Are you rotating the keys or only certificates? Because if I recall correctly the kernel checks on a key basis and not certificate (unless something has changed).

[dbnicholson]

In order to implement the realization of the “completely secure boot” principle, we have developed an approach with double signing of certain boot components using HSM. We combine the signing of the kernel and modules once with our CA certificate (for kernel) and additionally with unique private key + CA certificate (for modules). This approach for signing the modules is more secure compared to signing only with a unique private key. Additional use of HSM gives us an even higher protection for security because both private keys are only in HSM and never appear as physical files.

The kernel only allows and validates a single signature at the end of the module. So, it's not valid to say that modules are signed with both a unique key and your CA certificate's key.

Furthermore, it's more secure to sign modules only with the ephemeral key generated during the kernel build. That way, only modules built for that particular kernel build can be loaded. It ensures that you can't take a vulnerable module from an older kernel build and load it with a current kernel. In that scheme, modules only need to be signed with the ephemeral key during the kernel build. There's no need to keep the ephemeral key or re-sign the modules later. If you did sign the modules with your CA key, then any module ever signed by you would be valid for loading in the current kernel.

The only place this causes an issue is with out of tree modules like the nvidia drivers. In that case, you obviously need to sign them outside of the kernel build. This is what CONFIG_SYSTEM_TRUSTED_KEYS is for and where you'd put your CA certificate. For out of tree modules, you'd sign them with your CA key (or one for a certificate issued by the CA) in an HSM like you would for the kernel itself. This is a weaker but necessary process since you can now load any older signed out of tree module into the current kernel.

Following the standard process (CONFIG_MODULE_SIG_ALL=y) is more secure for handling in tree modules. Do you have out of tree modules that you need to sign? If not, then you don't need to do anything with CONFIG_SYSTEM_TRUSTED_KEYS or do any module signing involving an HSM.

[Jurij-Ivastsuk]

Hello @dbnicholson,
thank you very much for your advice and constructive recommendations.
The use of ephemeral key is certainly the best alternative from a security point of view. However, as I have already explained in an earlier discussion, this is unfortunately technically not an option for us:
#362 (comment)

[Jurij-Ivastsuk]

Hello @THS-on,
Thank you for your questions.

• We use General Purpose HSM, which provides broad cryptographic functionality and can be used for a wide range of applications.
• We can easily replace the double signing of the modules with a single one. This is carried out automatically during kernel compilation with a leaf certificate. This procedure is already described in our setup. The second signing of the modules with our CA certificate will be omitted.
• Kernel compilation and packaging is initiated with:
  make -j6 deb-pkg LOCALVERSION=-devimg-waxar KDEB_PKGVERSION=$(make kernelversion)-2
  The overall process initiated consists of the following steps:

1. kernel compilation
2. module compilation
3. module signing with leaf certificate
4. Debian-packaging of the kernel package for installation
   After the whole process is successfully completed, we get the Debian package for the new version of Kernel:
   linux-image-5.15.1-devimg-waxar_5.15.1-2_amd64.deb
   After the installation of this Debian package we get the kernel and all signed modules installed.

• As I do not have access to my IT-infrastructure at the moment, please have a look at
  https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240804
  where you will find the file waxar.der, our DER coded public CA certificate.
  Please use the command:
  openssl x509 -in waxar.der -inform der -text -noout
  This will allow you to read out the structure of our certificate.
  I can provide the structure of our leaf certificate later if required.

• Rotation only takes place with Key. With each new compilation, a new key is created accordingly, so that the modules are signed with a new leaf certificate each time they are compiled.

[dbnicholson]

Hi @Jurij-Ivastsuk, I'm trying to understand why this is a blocker. Quoting from that comment:

• If we compile a new Linux kernel with the options CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE, the signature keys are handled as follows:
  During the kernel compilation process (or, in our case before compilation), a new key pair (private and public key) is generated for signing the kernel modules. The private key is used to sign the modules during the compilation. The public key is integrated into the compiled kernel. This is then used when loading the modules to check their signature. The generated keys are not stored persistently. A new key pair will be generated with each new kernel compilation. We delete the private key immediately after kernel compilation to prevent misuse. The public key is contained in the compiled kernel and does not need to be stored separately.
• As I wrote earlier in this chat, we can't use the ephemeral key because building and signing are done on different machines and we cannot rule out the possibility that there will be a need to build out-of-tree modules.
• In order to ensure that kernel modules from older kernels cannot be loaded on newer ones, We always remove all old modules after upgrading to a new kernel version.
  The simplest way to prohibit a kernel to load older kernel modules, is to use an ephemeral key during build time, but due to the restrictions in our case, as I already mentioned, we cannot use them.

In this comment it sounds like you're doing exactly what I suggested. An ephemeral key is created during the kernel build and the private key is thrown away after the build. Unless you're following a different procedure, this is what's stored in CONFIG_MODULE_SIG_KEY. If you want to add a 2nd public key for verifying out of tree modules, then add it to CONFIG_SYSTEM_TRUSTED_KEYS.

The in tree modules are already signed with the ephemeral key and will be validated against it with the public certificate stored in CONFIG_MODULE_SIG_KEY. There's no need to sign them again later. The only reason to sign modules outside of the kernel build process is out of tree modules. In that case, build the kernel with that certificate in CONFIG_SYSTEM_TRUSTED_KEYS. So, I don't understand what's unique about your setup that requires you to sign the in tree modules again during the signing phase.

[Jurij-Ivastsuk]

Hi @dbnicholson,
I think I need to explain something here. As you can see, in a short summary on June 7 (ttps://github.com//issues/362#issuecomment-2154452597) I mentioned that we have implemented the recommendations of @aronowski in the activation of the parameters CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE.
I didn't know at that time that this change would lead to the creation of ephemeral keys. Under ephemeral key I understood one of the other implementations. That is why I mentioned that the use of ephemeral key is not the option for us. So, at this point, perhaps there was a misunderstanding in the communication.

Because some reviewers (#362 (comment)) have pointed out that you have to create a procedure to prevent old modules from being loaded, we have proposed an idea with double signing. Although, if the ephemeral key is already in use, this measure is completely unnecessary. The only point the reviewer @THS-on pointed out was the use of HSM when implementing automatic signing during kernel build. Because the argument that the signature key is physically accessible for a certain time during the kernel build, I think his argument is perfectly fine. After all, you want to have the highest possible security so that you can prevent other perhaps unwanted modules from being loaded. We have implemented HSM for the creation of ephemeral keys (https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/blob/waxar-shim-x86_64-aarch64-20240804/Setup_for_module_signing.txt). So, now we hope that all security requirements are fulfilled by us and we would be happy that the review process can be decided positively for us at this point.

[dbnicholson]

@Jurij-Ivastsuk Thanks very much for that explanation. So, the purpose of the HSM within the kernel build is to generate the key and do the signing rather than doing it all with a key generated on disk from the OS random source? That is certainly more secure.

However, this part seems to undo all of that:

Step 2: Additionally signing all modules with our CA-Certificate using HSM
To be sure that modules from older kernels or unwanted modules cannot be loaded, we have used the strategy of double signing of all modules. On the one hand with the unique kernel certificate and on the other hand with our CA certificate. Taking into account that our CA certificate is located in the HSM, this gives us a high level of security in terms of preventing the possibility of infiltration of foreign and unknown modules into the boot process.

If you sign the modules with your CA certificate, then you lose the ability to prevent old modules from being loaded since all modules from every kernel build will be valid. Furthermore, the linux module signature verification process only validates a single signature. It does not support having multiple signatures on a module. Looking at the validation process, it simply copies a single signature of bytes from the end of the module for verification. So, it's only validating the last signature, which is your CA signature. The ephemeral key signature isn't being used at all.

I feel like you should just drop this step. You just said "Although, if the ephemeral key is already in use, this measure is completely unnecessary." The only thing this step is going to do is decrease security.

[Jurij-Ivastsuk]

Hi @dbnicholson,
Thank you for your comment.
I completely agree with you and will skip step 2. Because as you say, this will only reduce security instead of increasing it.
I will remove this step from our setup as soon as possible.

[Jurij-Ivastsuk]

Hi @THS-on, @aronowski and @dbnicholson,

according to our last discussions with @dbnicholson, the area concerning the double signing of the modules has been completely removed from our signing setup. Please have a look at the updated version of our setup here:
https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/blob/waxar-shim-x86_64-aarch64-20240915/Setup_for_module_signing.txt

[aronowski]

The updated setup description seems reasonable at a first glance, but nevertheless I'll need to review the application again, considering how many changes there have been in the meantime. Should be easier, considering that some reviewer questions have been answered here.

Also, pinging others for further help - if you noticed some missing details or curiosities that need clarification from the applicants' side, please point them out.

[aronowski]

Review as per waxar-shim-x86_64-aarch64-20240915

The tag referred in the opening post is still waxar-shim-x86_64-aarch64-20240601. It will need an update once the things below have been clarified and pushed with a new tag.

---

*******************************************************************************
### URL for a repo that contains the exact code which was built to get this binary:
*******************************************************************************
hhttps://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240531

This question is not about the review application URL, please correct it.

---

*******************************************************************************
### What patches are being applied and why:
*******************************************************************************
N/A

OK: the "626" patch seems to be no longer used, although it still resides in the application's repository and the Patches folder is blatantly visible - instead, the former problem, for which the patch was made, is mitigated with the DISABLE_REMOVABLE_LOAD_OPTIONS=y option.

Hint: since I noticed some other errors, so that the application will have to be corrected, you can remove that folder, so it doesn't cause further confusion in case of another re-review.

---

*******************************************************************************
### If your boot chain of trust includes a Linux kernel:
### Is upstream commit [1957a85b0032a81e6482ca4aab883643b8dae06e "efi: Restrict efivar_ssdt_load when the kernel is locked down"](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1957a85b0032a81e6482ca4aab883643b8dae06e) applied?
### Is upstream commit [75b0cea7bf307f362057cc778efe89af4c615354 "ACPI: configfs: Disallow loading ACPI tables when locked down"](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=75b0cea7bf307f362057cc778efe89af4c615354) applied?
### Is upstream commit [eadb2f47a3ced5c64b23b90fd2a3463f63726066 "lockdown: also lock down previous kgdb use"](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=eadb2f47a3ced5c64b23b90fd2a3463f63726066) applied?
*******************************************************************************
Yes

OK: all the issues mitigated by those upstream patches are mitigated and explained as per #368 (the same kernel from the accepted application is reused).

---

*******************************************************************************
### Do you use an ephemeral key for signing kernel modules?
### If not, please describe how you ensure that one kernel build does not load modules built for another kernel.
*******************************************************************************
We do not use an ephemeral key. We use a WAXAR HSM backed key for signing kernel modules.

This is a bigger thing, that got it's own description and there have been some discussions to help with it. It seems OK and thoroughly described, but nevertheless, I wanted to try it out for myself:

$ mkdir -p $HOME/.config/softhsm2/tokens
$ echo "directories.tokendir = $HOME/.config/softhsm2/tokens" > $HOME/.config/softhsm2/softhsm2.conf
$ softhsm2-util --init-token --label HSM --so-pin Secret.123 --pin Secret.123 --free
Slot 0 has a free/uninitialized token.
The token has been initialized and is reassigned to slot 313440693

$ pkcs11-tool --module /usr/lib64/softhsm/libsofthsm.so --login --pin Secret.123 --keypairgen --key-type rsa:2048 --id 01 --label "kernel_signing_key"
Using slot 0 with a present token (0x12aeb9b5)
Key pair generated:
Private Key Object; RSA
  label:      kernel_signing_key
  ID:         01
  Usage:      decrypt, sign, signRecover, unwrap
  Access:     sensitive, always sensitive, never extractable, local
Public Key Object; RSA 2048 bits
  label:      kernel_signing_key
  ID:         01
  Usage:      encrypt, verify, verifyRecover, wrap
  Access:     local

$ openssl req -new -x509 -key <(pkcs11-tool --module /usr/lib64/softhsm/libsofthsm.so --login --pin Secret.123 --read-object --id 01 --type pubkey) -out signing_key.x509 -days 3650
Using slot 0 with a present token (0x12aeb9b5)
Could not find private key from /dev/fd/63
80C2EBD243710000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:151:

Is this an issue regarding SoftHSM or is there something else going on? Decomposing this command into two steps, i.e. dumping the object into a file first, and then using this file directly with openssl, rather than using process substitution, makes no difference.

As far as I can see, the custom sign-file script, replacing the vanilla compiled one, has been architected so that the invocation stays the same, but the first three arguments go unused due to how the operations are delegated to an actual HSM. pkcs11-tool is then used to sign each kernel module.
What OpenSC distribution do you have, so that pkcs11-tool uses the --key-id option? I can't find it anywhere.

I'll need to take another look, once these are clarified.

---

*******************************************************************************
### Do you add a vendor-specific SBAT entry to the SBAT section in each binary that supports SBAT metadata ( GRUB2, fwupd, fwupdate,
 shim + all child shim binaries )?
### Please provide exact SBAT entries for all SBAT binaries you are booting or planning to boot directly through shim.
### Where your code is only slightly modified from an upstream vendor's, please also preserve their SBAT entries to simplify revocat
ion.
If you are using a downstream implementation of GRUB2 (e.g. from Fedora or Debian), please
preserve the SBAT entry from those distributions and only append your own.
More information on how SBAT works can be found [here](https://github.com/rhboot/shim/blob/main/SBAT.md).
*******************************************************************************
sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md
shim,3,UEFI shim,shim,1,https://github.com/rhboot/shim
shim.waxar,1,Waxar GmbH & Co.KG,shim,15.8,https://www.waxar.eu

We use upstreams distro from Canonical for grub since we are not rebuilding it.

I can see this entry being from January 15, 2024, so in the meantime it should have been updated to shim,4. Please, also attach the GRUB2 listing as well despite the entry being manually added to a precompiled binary.

---

*******************************************************************************
### Which modules are built into your signed GRUB2 image?
*******************************************************************************
N/A

In case of reusing the same modules as the upstream release, i.e.:

Ubuntu: grub2 - Version 2.12~rc1-10ubuntu4

Please, provide the list of them here.

---

*******************************************************************************
### How do the launched components prevent execution of unauthenticated code?
*******************************************************************************
N/A

If the setup just reuses upstream mechanisms, please describe them here.

---

*******************************************************************************
### What kernel are you using? Which patches does it includes to enforce Secure Boot?
*******************************************************************************
Kernel version of Debian 5.15.1

Please see #362 (comment), otherwise things might get confusing especially for those, who work with different family of distros.

[Jurij-Ivastsuk]

Hello @aronowski ,
Thank you very much for this review. I have answered all your questions here. I hope I have been able to clarify everything.

URL for a repo that contains the exact code which was built to get this binary:
This question is not about the review application URL, please correct it.

Corrected

OK: the "626" patch seems to be no longer used, although it still resides in the application's repository and the Patches folder is blatantly visible - instead, the former problem, for which the patch was made, is mitigated with the DISABLE_REMOVABLE_LOAD_OPTIONS=y option.
Hint: since I noticed some other errors, so that the application will have to be corrected, you can remove that folder, so it doesn't cause further confusion in case of another re-review.

Folder and file /Patches/626.patch deleted.
Dockerfile lines concerning patch "626" removed.

This is a bigger thing, that got it's own description and there have been some discussions to help with it. It seems OK and thoroughly described, but nevertheless, I wanted to try it out for myself:
openssl req -new -x509 -key <(pkcs11-tool --module /usr/lib64/softhsm/libsofthsm.so --login --pin Secret.123 --read-object --id 01 --type pubkey) -out signing_key.x509 -days 3650
0C2EBD243710000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:151: Could not find private key from /dev/fd/63
Is this an issue regarding SoftHSM or is there something else going on?

You can't export the public key with OpenSSL and a PKCS#11 token? This is strange... The command tries to read a private key while you explicitly ask for the public key (--type pubkey)
Here are some of my suggestions for solving this problem:
a) PKCS#11 engine is not configured properly ?
b) not correct OpenSSL configuration ? Check your OpenSSL configuration file (usually /etc/ssl/openssl.cnf) to make sure that the PKCS#11 engine is set up correctly.
c) authorization problem ? Make sure you have the necessary permissions to access the PKCS#11 module and token.

What OpenSC distribution do you have, so that pkcs11-tool uses the --key-id option? I can't find it anywhere.

We use the opensc for debian 10 “Buster”, version 0.20.0-1
As far as I know, there is no OpenSC distribution that specifically allows pkcs11-tool to use a "--key-id" option.
The option to specify a key ID is typically "--id" rather than "--key-id".

I can see this entry being from January 15, 2024, so in the meantime it should have been updated to shim,4.
Please, also attach the GRUB2 listing as well despite the entry being manually added to a precompiled binary.

Files shimx64.efi and shimaa64.efi have been rebuild with the updated sbat.csv (shim,4)
sha256sum for shimx64.efi (1894c7e467991c117162e1e40dd61ed85a5f790a68216fdbee93b2619b70df67)
sha256sum for shimaa64.efi (5c7238473401d791c7f376efee90cf1e9fe23e2bf1814f4795474d57920bdfbf)
both of them have been updated on all places in the review.

The content of sbat.csv is now:

sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md
shim,4,UEFI shim,shim,1,https://github.com/rhboot/shim
shim.waxar,1,Waxar GmbH & Co.KG,shim,15.8,https://www.waxar.eu

The SBAT listing for GRUB2 from Canonical with the addition for waxar:

sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md.
grub,4,Free Software Foundation,grub,2.12~rc1,https://www.gnu.org/software/grub/.
grub.ubuntu,1,Ubuntu,grub2,2.12~rc1-10ubuntu4,https://www.ubuntu.com/.grub.
peimage,1,Canonical,grub2,2.12~rc1-10ubuntu4,https://salsa.debian.org/grub-team/grub/-/blob/master/debian/patches/secure-boot/efi-use-peimage-shim.patch
grub.waxar,1,Waxar GmbH & Co.KG,grub2,2.12~rc1-10ubuntu4,https://www.waxar.eu

Both listings have been updated in README.md

Which modules are built into your signed GRUB2 image?
Please, provide the list of them here.

Ubuntu's GRUB 2 version 2.12~rc1-10ubuntu4 includes the following modules built into its signed GRUB EFI binary (grubx64.efi):

1. Core modules required for basic functionality
2. Filesystem modules to read various filesystems containing kernel and initramfs images
3. The lvm module for Logical Volume Management support
   The modules are embedded directly in the EFI binary to comply with Secure Boot requirements.

Here's a list of some of the key modules:

1. UEFI modules:
   Modules for ARM systems with UEFI
   Modules for ARM64 (AArch64) systems with UEFI
   Modules for IA32 (32-bit x86) systems with UEFI
   Modules for AMD64 (64-bit x86) systems with UEFI
   Modules for RISC-V 64-bit systems with UEFI
2. BIOS modules:
   Modules for traditional BIOS systems
3. Other architecture-specific modules:
   Modules for Coreboot firmware
4. Filesystem modules:
   ext2/ext3/ext4
   FAT12/FAT16/FAT32
   exFAT
   BtrFS (including RAID0, RAID1, RAID10, gzip and lzo compression)
   ISO9660 (including Joliet and Rock-ridge extensions)
   NTFS (including compression)
   HFS and HFS+
   UFS and UFS2
5. Compression modules:
   Support for reading compressed files
6. Network modules:
   Support for network booting
7. Multiboot modules:
   Support for loading multiple modules as specified in the Multiboot standard
8. Environment modules:
   Support for saving and loading environment variables
   Listing has been updated in README.md

How do the launched components prevent execution of unauthenticated code?
If the setup just reuses upstream mechanisms, please describe them here.

GRUB2 employs several mechanisms to prevent the execution of unauthenticated code when launching components in a secure boot environment:

1. Signature Verification: GRUB2 verifies digital signatures on components it loads, such as kernels, using the shim framework. This ensures only properly signed and authenticated code is executed.
2. Secure Boot Chain: GRUB2 is part of a secure boot chain that starts with UEFI Secure Boot and continues through shim, GRUB2, and the kernel. Each component verifies the next before handing off control.
3. Lockdown Mechanisms: GRUB2 implements lockdown features that restrict certain operations when secure boot is enabled, preventing unauthorized modifications to the boot process.
4. Restricted Boot Options: In secure boot mode, GRUB2 limits available boot options to only those that are signed and authenticated.
5. Configuration File Protection: The GRUB2 configuration file (/boot/grub/grub.cfg) is protected with restrictive permissions to prevent unauthorized modifications.
6. Module Restrictions: GRUB2 either uses the shim framework to verify modules or disables modules entirely when secure boot is enabled.
7. Kernel Verification: For Linux kernels specifically, GRUB2 verifies the kernel signature before loading it.
8. No Unsigned Kernel Loading: In secure boot mode, GRUB2 is configured to not load unsigned kernels.
   These mechanisms work together to create a robust system that prevents the execution of unauthenticated code during the boot process when secure boot is enabled.
   This section has been updated in README.md

What kernel are you using? Which patches does it includes to enforce Secure Boot?
Please see #362 (comment), otherwise things might get confusing especially for those, who work with different family of distros.

Debian's 5.15.1 kernel includes several patches and features to enforce Secure Boot functionality:

1. Key Secure Boot Features
   Shim bootloader: Debian uses a small UEFI bootloader called "shim" that is signed by Microsoft. This allows Debian to sign its own binaries without requiring additional Microsoft signatures
   Debian signing keys: The kernel embeds Debian's own signing keys, which are used to verify the authenticity of Debian-provided UEFI programs and kernels
   Machine Owner Key (MOK) support: The kernel supports using a Machine Owner Key for signing custom kernels and modules to work with Secure Boot
2. Specific Patches and Enhancements
   X.509 certificate loading: The kernel loads compiled-in X.509 certificates to verify signatures
   UEFI key variable checking: The kernel checks for keys in UEFI variables like DB and MokListRT when Secure Boot is enabled
   DKMS module signing: Support for signing Dynamic Kernel Module System (DKMS) modules using a Machine Owner Key
   Initramfs updates: Patches to ensure the initramfs is properly updated when signing new modules
3. Verification Mechanisms
   Module signature verification: The kernel verifies signatures on loadable kernel modules before allowing them to be loaded
   EFI binary verification: Tools like sbverify are included to check signatures on EFI binaries
   This section has been updated in README.md

[aronowski]

Review as per waxar-shim-x86_64-aarch64-20241111

---

URL for a repo that contains the exact code which was built to get this binary:
This question is not about the review application URL, please correct it.

Corrected

This question is not about the review application URL, but from where the code has been taken - if it's just the upstream shim repository, just write the link to it here.

---

Folder and file /Patches/626.patch deleted.
Dockerfile lines concerning patch "626" removed.

Thanks!

---

You can't export the public key with OpenSSL and a PKCS#11 token?

Since the later answer refers to Debian Buster, I used it as a Docker container to double-check things. This happens:

$ docker run -it debian:buster /bin/bash
root@1188d8865dfd:/# mkdir -p $HOME/.config/softhsm2/tokens
root@1188d8865dfd:/# echo "directories.tokendir = $HOME/.config/softhsm2/tokens" > $HOME/.config/softhsm2/softhsm2.conf
root@1188d8865dfd:/# softhsm2-util --init-token --label HSM --so-pin Secret.123 --pin Secret.123 --free
Slot 0 has a free/uninitialized token.
The token has been initialized and is reassigned to slot 202401902
root@1188d8865dfd:/# pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --pin Secret.123 --keypairgen --key-type rsa:2048 --id 01 --label "kernel_signing_key"
Using slot 0 with a present token (0xc10686e)
Key pair generated:
Private Key Object; RSA
  label:      kernel_signing_key
  ID:         01
  Usage:      decrypt, sign, unwrap
Public Key Object; RSA 2048 bits
  label:      kernel_signing_key
  ID:         01
  Usage:      encrypt, verify, wrap
root@1188d8865dfd:/# openssl req -new -x509 -key <(pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --pin Secret.123 --read-object --id 01 --type pubkey) -out signing_key.x509 -days 3650
Using slot 0 with a present token (0xc10686e)
unable to load Private Key
139659649082560:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: ANY PRIVATE KEY

---

We use the opensc for debian 10 “Buster”, version 0.20.0-1
As far as I know, there is no OpenSC distribution that specifically allows pkcs11-tool to use a "--key-id" option.
The option to specify a key ID is typically "--id" rather than "--key-id".

Yes, confirmed on my end, so that the Debian Buster-provided OpenSC tooling has no such option. Although in my case the newest version is:

root@1188d8865dfd:/# dpkg-query --show opensc
opensc  0.19.0-1+deb10u3

I asked, since in this document it says --key-id <key-id>:

Step 8: Modify the sign-file script in the kernel source to use the HSM for signing, 
the script is usually located at /usr/src/linux/scripts/ directory:
-------------sign-file script--------------BEGIN
#!/bin/bash

HASH_ALGO=$1
PRIVATE_KEY=$2
CERT=$3
MODULE=$4

# Use pkcs11-tool to sign the module
pkcs11-tool --module /usr/lib/opensc-pkcs11.so --sign --mechanism RSA-PKCS --input-file $MODULE --output-file $MODULE.sig --login --pin <our-pin> --key-id <key-id>

# Append the signature to the module
cat $MODULE.sig >> $MODULE
-------------sign-file script--------------END

So how can the script work, if no OpenSC distro ever provided the --key-id option?

---

The content of sbat.csv [...]

Looks OK after a rebuild, thanks!

The SBAT listing for GRUB2 from Canonical with the addition for waxar [...]

There are some errors with the listing itself - in the meantime some generation numbers have been bumped. Also, I can see some errors or typos - there are dots at the end of each line of the listing, and there's no newline between grub.ubuntu and grub.peimage.

Please, update the entry, and surround the listing in the new application with three backticks, so the rendering on GitHub doesn't get messed up - right now the whole listing is a one-liner, and the tildes act like strike-through control characters.

---

Which modules are built into your signed GRUB2 image?
Please, provide the list of them here.

Ubuntu's GRUB 2 version 2.12~rc1-10ubuntu4 includes the following modules built into its signed GRUB EFI binary (grubx64.efi):
[...]

That question is about the list of the actual modules, rather than their descriptions - reusing upstream binary without adding own ones means just the same list as they have.

---

*******************************************************************************
### How do the launched components prevent execution of unauthenticated code?
*******************************************************************************
GRUB2 employs several mechanisms to prevent the execution of unauthenticated code when launching components in a secure boot environment:
1. Signature Verification: GRUB2 verifies digital signatures on components it loads, such as kernels, using the shim framework. This ensures only properly signed and authenticated code is executed.
2. Secure Boot Chain: GRUB2 is part of a secure boot chain that starts with UEFI Secure Boot and continues through shim, GRUB2, and the kernel. Each component verifies the next before handing off control.
3. Lockdown Mechanisms: GRUB2 implements lockdown features that restrict certain operations when secure boot is enabled, preventing unauthorized modifications to the boot process.
4. Restricted Boot Options: In secure boot mode, GRUB2 limits available boot options to only those that are signed and authenticated.
5. Configuration File Protection: The GRUB2 configuration file (/boot/grub/grub.cfg) is protected with restrictive permissions to prevent unauthorized modifications.
6. Module Restrictions: GRUB2 either uses the shim framework to verify modules or disables modules entirely when secure boot is enabled.
7. Kernel Verification: For Linux kernels specifically, GRUB2 verifies the kernel signature before loading it.
8. No Unsigned Kernel Loading: In secure boot mode, GRUB2 is configured to not load unsigned kernels.
These mechanisms work together to create a robust system that prevents the execution of unauthenticated code during the boot process when secure boot is enabled.

While technically true, the question doesn't require such a thorough answer, but a simple one-liner would do just fine if mentioning some Canonical-specific implementation and the internal HSM-based module signing (something cool, that I haven't seen anywhere else).

---

*******************************************************************************
### What kernel are you using? Which patches does it includes to enforce Secure Boot?
*******************************************************************************
Kernel version of Debian 5.15.1

Debian's 5.15.1 kernel includes several patches and features to enforce Secure Boot functionality:
1. Key Secure Boot Features
    Shim bootloader: Debian uses a small UEFI bootloader called "shim" that is signed by Microsoft. This allows Debian to sign its own binaries without requiring additional Microsoft signatures
    Debian signing keys: The kernel embeds Debian's own signing keys, which are used to verify the authenticity of Debian-provided UEFI programs and kernels
    Machine Owner Key (MOK) support: The kernel supports using a Machine Owner Key for signing custom kernels and modules to work with Secure Boot
2. Specific Patches and Enhancements
    X.509 certificate loading: The kernel loads compiled-in X.509 certificates to verify signatures
    UEFI key variable checking: The kernel checks for keys in UEFI variables like DB and MokListRT when Secure Boot is enabled
    DKMS module signing: Support for signing Dynamic Kernel Module System (DKMS) modules using a Machine Owner Key
    Initramfs updates: Patches to ensure the initramfs is properly updated when signing new modules
3. Verification Mechanisms
    Module signature verification: The kernel verifies signatures on loadable kernel modules before allowing them to be loaded
    EFI binary verification: Tools like sbverify are included to check signatures on EFI binaries

Which Debian release ships the kernel 5.15.1? I've only seen this kernel shipped with Ubuntu Jammy.

What exact Debian-specific downstream patches are applied, that implement the described functionality, that are not present in upstream kernels?

[Jurij-Ivastsuk]

Hi @aronowski, thanks for your review and valuable comments!
Here are my answers to your questions. Hopefully there are no more spelling mistakes and the answers answer all your questions:

This question is not about the review application URL,
but from where the code has been taken - if it's just the upstream shim repository, just write the link to it here.

https://github.com/rhboot/shim.git
Corrected in README.md

Since the later answer refers to Debian Buster, I used it as a Docker container to double-check things. This happens:

Dear @aronovski, I have checked and corrected all the lines in the code again. You can find the corrected version here. Since you use Softhsm2 for your tests, I have created a document especially for this case. Please have a look at this document. You can enter the commands line by line and test the most important parts of the setup.

Yes, confirmed on my end, so that the Debian Buster-provided OpenSC tooling has no such option. Although in my case the newest version is:

In the meantime, our Linux computer has been upgraded. Now we have done the whole test series on the Debian version:
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
opensc version: 0.21.0-1
softhsm2 version: 2.6.1-2

I asked, since in this document it says --key-id

Sorry, there was a typing error, obviously --id instead of --key-id
Corrected in Setup_for_module_signing.txt.

The SBAT listing for GRUB2 from Canonical with the addition for waxar
There are some errors with the listing itself - in the meantime some generation numbers have been bumped. Also, I can see some errors or typos

SBAT listing for GRUB2 corrected:

sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md
grub,4,Free Software Foundation,grub,2.12~rc1,https://www.gnu.org/software/grub/
grub.ubuntu,1,Ubuntu,grub2,2.12~rc1-10ubuntu4,https://www.ubuntu.com/.grub
peimage,1,Canonical,grub2,2.12~rc1-10ubuntu4,https://salsa.debian.org/grub-team/grub/-/blob/master/debian/patches/secure-boot/efi-use-peimage-shim.patch
grub.waxar,1,Waxar GmbH & Co.KG,grub2,2.12~rc1-10ubuntu4,https://www.waxar.eu

updated in README.md

Which modules are built into your signed GRUB2 image?
That question is about the list of the actual modules, rather than their descriptions - reusing upstream binary without adding own ones means just the same list as they have.

Yes we use upstreams distro from Canonical for grub since we are not rebuilding it and the list remains obviously the same.
I tried to analyze the grub2 binary with the help of “strings”, but because of the strings command that will only recognize strings
larger than 4 characters, I could not list all the modules. Finally I found the following information:
Grub2 modules:
acpi afsplitter all_video bitmap bitmap_scale boot btrfs bufio cat chain configfile
cpuid crypto cryptodisk datetime disk diskfilter echo efi_gop efinet efi_uga ext2 extcmd
fat font fshelp gcry_arcfour gcry_blowfish gcry_camellia gcry_cast5 gcry_crc gcry_des gcry_dsa
gcry_idea gcry_md4 gcry_md5 gcry_rfc2268 gcry_rijndael gcry_rmd160 gcry_rsa gcry_seed gcry_serpent
gcry_sha1 gcry_sha256 gcry_sha512 gcry_tiger gcry_twofish gcry_whirlpool gettext gfxmenu gfxterm
gfxterm_background gzio halt help hfsplus iso9660 jpeg keystatus linux linuxefi loadenv loopback
ls luks lvm lzopio mdraid09 mdraid1x mmap mpi net normal ntfs password_pbkdf2 pbkdf2 pgp png probe
procfs raid5rec raid6rec regexp relocator search search_fs_file search_fs_uuid search_label sleep
smbios squash4 terminal trig video video_bochs video_cirrus video_colors xfs xzio zfs zfscrypt
zfsinfo zstd cat fat ls lvm pgp png zfs

Please note, this list may not include all modules.
updated in README.md

What kernel are you using? Which patches does it includes to enforce Secure Boot?
Which Debian release ships the kernel 5.15.1? I've only seen this kernel shipped with Ubuntu Jammy.

According to the information I have, there is no official Debian version with the kernel 5.15.1.
Debian 11 "Bullseye" which was released in August 2021 included kernel 5.10.
Debian 12 "Bookworm" is shipped with the kernel 6.1.
The Linux kernel 5.15 was released in October 2021 right after the release of Debian 11.
We took this kernel version because at that time it seemed to us that this version of the kernel
allowed support for more hardware components. We also stayed with this kernel version because we had no problems with device support.

What exact Debian-specific downstream patches are applied, that implement the described functionality, that are not present in upstream kernels?

Unfortunately I can't find the exact differences between kernel 5.10 and 5.15 regarding pathes.
Here is the list of patches applied in 5.15.1:

2021-11-04 14:12 ` [PATCH 5.15 01/12] sfc: Fix reading non-legacy supported link modes Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 02/12] media: firewire: firedtv-avc: fix a buffer overflow in avc_ca_pmt() Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 03/12] Revert "xhci: Set HCD flag to defer primary roothub registration" Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 04/12] Revert "usb: core: hcd: Add support for deferring " Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 05/12] drm/amdkfd: fix boot failure when iommu is disabled in Picasso Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 06/12] Revert "soc: imx: gpcv2: move reset assert after requesting domain power up" Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 07/12] ARM: 9120/1: Revert "amba: make use of -1 IRQs warn" Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 08/12] Revert "wcn36xx: Disable bmps when encryption is disabled" Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 09/12] drm/amdgpu: revert "Add autodump debugfs node for gpu reset v8" Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 10/12] drm/amd/display: Revert "Directly retrain link from debugfs" Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 11/12] Revert "drm/i915/gt: Propagate change in error status to children on unhold" Greg Kroah-Hartman
2021-11-04 14:12 ` [PATCH 5.15 12/12] ALSA: usb-audio: Add quirk for Audient iD14 Greg Kroah-Hartman

For details please see here.