Getting the ZFS "magic" on GNU Guix System

[raid5atemyhomework]

I've been doing some work recently to get ZFS into GNU Guix System. Guix System is a GNU/Linux distribution built around the Guix functional package manager. Think "Nix, but using Scheme and only-free-as-in-freedom-software". Guix is willing to package ZFS, on the basis of the fact that users are free to do what they want on hardware they own, so they can use the Guix build system to build ZFS locally and link it to their kernel and run the combined kernel+ZFS, users are just not allowed to redistribute the combined work (and the guix repositories don't contain the binaries, users have to compile it, but it's automated by the Guix package manager). However, support for actually using ZFS is nonexistent (and is what I am working on).

ZFS guides tout many things that ZFS can do automatically:

• Automatic import of pools and automatic mounting of datasets at startup.
• Can be configured to automatically add a hot spare in case of a drive failure during operation.
• ZVOLs become accessible by /dev/zvol/pool/dataset.
• etc.

It turns out that many of the things that ZFS guides take for granted are really magic that is implemented, on Linux, by a variety of programs.

For example, the ZFS kernel module has to be loaded at some point, and that has to be done by the Linux init program. The ZFS release has a bunch of systemd services to do so. Guix does not use systemd, it uses GNU Shepherd. So I have to go understand the load-the-module and other things that the systemd services do and reimplement it in GNU Shepherd.

And of course the ZFS automounting behavior turns out to be performed by a systemd service as well.

ZVOLs getting into /dev/zvol/* are actually implemented by udev. The ZFS release has a bunch of udev rules which implement this. Fortunately Guix uses udev also, but Guix is different enough that I need to add special support for installing the udev rules provided by ZFS into the appropriate place in the Guix system.

And then there's ZED, which does a lot of the other magic, like emailing you when a scrub is finished, or replacing a failing device with a hotspare (if you enable the appropriate script).

So I have a bunch of things I need to ask about ZFS and the magic bits that have to be integrated into the distribution in order to make ZFS work the way guides say ZFS is supposed to work.

First, lemme know if the below set of things to make ZFS magic work on Linux is complete. For now I'm targeting /home on ZFS, not yet / or /boot on ZFS, because just targeting /home seems to be enough of a challenge. So I'm going to skip over initrd parts, and start after the bootup has pivoted from the initrd filesystem to the "real" / root filesystem.

• init
  • Load ZFS kernel module. Also, somehow feed any sysad-selected module parameters to the ZFS kernel module. /etc/modprobe.d/zfs.conf is traditional, but Guix developers want to make their own mechanism. Then there's the issue with / on ZFS where you somehow have to pass in the parameters even earlier, possibly before /etc/modprobe.d is accessible, but, sigh. It's why I don't want to think of / on ZFS for Guix yet.
  • Import pools without mounting them yet, must happen after kernel module loading. Can be done by zpool import -a -N. However, traditionally it seems to be done by zpool -c /etc/zfs/zpool.cache -a -N. How important is the cache file?? Also do I need to do anything at shutdown?
  • Mount datasets, must happen after pool importation. Done by zfs mount -a. On shutdown, do zfs unmount -a.
  • Wait for ZVOLs to appear on /dev/zvol/*, must happen after pool importation. Done by zvol_wait. Also need to have udev rules otherwise zvol_wait will hang forever, because what the ZFS kernel module actually does is create a /dev/zd* device.
  • Launch ZED. Not sure whether it must happen after pool importation. Presumably has to happen after ZFS kernel module loading at least. Done by zed, but I may need to explicitly pass in locations of PID file, zedlet dir, etc.
  • Start ZFS sharing, must happen after dataset mounting. Done by zfs share -a. Must also happen after samba or NFS services are loaded (?).
• udev
  • Need to install the ZFS rules.
  • An issue is that the udev rules refer to /sbin/modprobe. Guix does not have a traditional /bin or /sbin directory, instead a tool that uses modprobe or any other tool should refer to it via an absolute path in /gnu/store which contains the specific version that the tool is known to work with. So I have to modify the rules to change this external dependency and point it to the ZFS dependency on the kmod package.
• ZED
  • Needs a bunch of files in /etc/zfs/zed.d. They are symlinks to zedlets that are built into ZFS. The sysad is supposed to remove or add symlinks, and can add custom zedlets as well.
  • In Guix, the preferred way is to not edit anything in /etc, at all. Instead, you have an operating-system file that you process and then it spits out a new configuration that includes stuff in /etc and so on. What I plan to do is that the configuration file can indicate whether a particular ZED trigger should be taken from the ZFS package (cmd/zed/zed.d/*), should be disabled (if it is like scrub_finish-notify.sh which is enabled by default), or should run custom Scheme code. And for zedlets that come with ZFS, what the various configurations that should go into the zed.rc file.

Does that make sense?

Now an issue I have is the cache file. For now I'm fine with doing an all-devices scan. For one, /etc/zfs does not get created by default and I have to write the code that does that when ZFS gets installed. Do I really need the cache file? The cache file is not created if /etc/zfs directory does not exist and currently I don't have this path existing.

Also, how important is using /dev/disk/by-id on Linux nowadays? I tried in a Guix VM with ZFS installed, and created a pool with SLOGs and L2ARCs in various disks, then shut down the VM and restarted it with the devices in jumbled order, and the plain, non-cache using zpool import -a -N semed to be able to import the pool with the jumbled devices correctly. My understanding is that the /dev/disk/by-id/* paths are stored in the cache file since the motherboard could jumble up how devices are sent to the Linux kernel, so it seems to be working with ZFS 2.0.1 at least even without the cache file.

[behlendorf]

It's great to hear about OpenZFS being packaged for another Linux distribution! Let me try and field a few of your questions.

You've definitely touched on most (maybe all) of the major integration issues required to get OpenZFS working as intended. What you're planning makes sense and we've tried our best to provide flexible mechanisms to tackles all of these integrations issues. But since each distribution has their own way of doing things you'll need to adapt as appropriate for a GNU Guix System. By all means leverage as much of the provided "magic" integration code as makes sense.

How important is the cache file?? Also do I need to do anything at shutdown?

The cache file is optional but histroically preferred. It's primary purpose is to provide a quick way to determine which pools should be imported at boot time and by what paths. You can always run zpool import to scan for any available pools to import. However, this depends on all of the the block devices being available which may not be as reliable during the boot process. Scanning will also be slower for pools constructed from tens, or hundreds, of block devices so it's nice to have both options. As for shutdown if you do want to use the cache file you'll want to unmount the filesystems but not export the pool. Exporting the pool will remove it from the cache file, meaning it it won't be found be a zpool import -cfor the next boot.

Wait for ZVOLs to appear on /dev/zvol/*, must happen after pool importation.

We provide the zvol_wait service to make it easier for end users to script starting services which rely of the zvols. However, at the end of the day they're just normal block devices which can be handled in whatever way is right for the platform. As you mentioned, you will need a udev rule installed somewhere to created the /dev/zvol/ links. This isn't really different than the other common /dev/disk/ symlinks created by udev.

An issue is that the udev rules refer to /sbin/modprobe.

This may not be an issue for you. The sole purpose of this rule is to trigger loading the zfs kmod when a block device containing a pool is detected. If on a GNU Guix System the kmod will always be built in to the kernel then it'll always be available and there's nothing to load.

Launch ZED. Not sure whether it must happen after pool importation

The only requirement on Linux is that this happens after the kernel modules are loaded. It can be started either before or after pools are imported.

Also, how important is using /dev/disk/by-id on Linux nowadays?

It's recommended, but not because OpenZFS won't be able to import a pool if the paths change. The zpool import command can reliably locate the pool's devices regardless of how you jumble up the drives. It's recommended because it's helpful to have consistently named vdevs in the zpool status output regardless of how your system may reorder them. It's also worth mentioning that a preferred path name is stored in the pool configuration on each drive. OpenZFS will prefer to open the device by this path. However if you're doing a scanning import and the preferred path isn't available, but another /dev/ path is, OpenZFS will fallback to using that alternate path.

Hopefully, that helps answer some of your questions!

[raid5atemyhomework]

The cache file is optional but histroically preferred

Well, Guix would prefer to not have (or at least greatly reduce) mutable files in /etc/, ideally everything in /etc/ should be generated from the operating-system declaration. For now I'm using scanning imports. zpool import -a -N will ignore /etc/zfs/zpool.cache if it exists?

I notice there's a module parameter that indicates the path to the cache file, does it actually work on Linux? I tried disabling the zpool import -a -N with a /etc/zfs/zpool.cache file existing, but it didn't look like pools got imported.

If for example Guix decides it's better to move the cachefile to /var/zfs/zpool.cache, how do I ensure that e.g. zpool create will write to /var/zfs/zpool.cache instead of /etc/zfs/zpool.cache like it defaults to? The Guix build system is flexible enough to do simple search-and-replace of the code before compiling so I think it's possible to just modify the source code (need it in ZFS on Guix anyway, since Guix doesn't have a traditional /bin/mount and /bin/umount either, so those have to get redirected to Linux utilities in /gnu/store).

Is there a way to refresh the cachefile? Might be something good to do on shutdown...

We provide the zvol_wait service

Yes, I saw that as well. I already added support for this and even created a way to hook it into the existing (not yet very complete) framework Guix has for waiting for devices to come online before mounting filesystems.

If on a GNU Guix System the kmod will always be built in to the kernel

I don't intend to do that, licensing concerns mean that Guix can't distribute such a kernel, we can only create instruction for the build system to automate the user's action of compiling, so every user who wants to use ZFS on Guix would be forced to recompile a kernel at each update of the kernel or ZFS. At least as a kernel module only the ZFS would be recompiled.

This might not be clear but basically how Guix works is, is that in principle it "should" download sources and compile, for all packages, starting from a tiny self-hosting hex0 binary editor that is used to bootstrap a simple Scheme interpreter that is used to bootstrap a simple C compiler that is then used to compile gcc that is then used to compile the rest of the system. However, this would make building and updating a Guix system pretty horrible, so Guix has a concept of "substitutes", where instead of downloading source it downloads binaries provided by a build farm, and a method to "challenge" the build farm by downloading source for a particular package locally and checking the compiled binary matches the build farm's output. Guix provides substitute binaries for the Linux kernel, but can't provide substitute binaries for ZFS because license issues, however it can package instructions on how to download and build ZFS from source, since this is the basic operation that Guix provides anyway and the binaries are just an optimization. Because of the recompilation requirement it's easier to just have ZFS remain a kernel module.

On the other hand I am putting an init service that loads the ZFS kernel module, so it might not be necessary to put them in udev rules too. Not sure how to load ZFS kernel module while still in initrd though, what do other distros that support / on ZFS do (and how do they pass in kernel module parameters?)?

The only requirement on Linux is that this happens after the kernel modules are loaded.

Thanks

It's recommended because it's helpful to have consistently named vdevs in the zpool status output regardless of how your system may reorder them

Ah, okay, thanks. I guess I'll add that in the manual for Guix.

[behlendorf]

For now I'm using scanning imports. zpool import -a -N will ignore /etc/zfs/zpool.cache if it exists?

Yes, that's right. The cache file will be ignored by default unless zpool import -c is specified. Sticking to scanning at least initially sounds totally reasonable to me.

You have a couple options for setting the default cache file location. The easiest way you've already discovered and that is to set the default path with the spa_config_path module option. The downside of course is that if for some reason it doesn't get set the code will revert to its usual /etc/ default. Alternately, you could patch the source to set a new default by changing the ZPOOL_CACHE definition here.

One other thing worth mentioning is that each pool contains a "cachefile" property which can be used to override the default location on a per pool basis. This does include a "none"option which means the pool will never get added to the cache file. If Guix is handling pool creation you could consider setting this as another option.

Is there a way to refresh the cachefile? Might be something good to do on shutdown...

The cache file is automatically updated whenever the pool configuration is changed. So you shouldn't need to regenerate it, but it can be done manually by running zpool set cachefile=/etc/zfs/zpool.cache tank.

At least as a kernel module only the ZFS would be recompiled.

Absolutely, this is what's done for several other distributions. The kernel module is distributed as a DKMS package and then binaries are provided for user space from the package manager. A copy of the kmods and utilities is added to the initrd and loaded by whatever infrastructure the distribution uses. It seems these days theirs a great variety of systems in use. There are some scripts to help with this under the contrib directory for both dracut and initramfs style setups.

[raid5atemyhomework]

Thank you!

There are some scripts to help with this under the contrib directory for both dracut and initramfs style setups.

Guix uses its own homegrown initrd generator, but I'll try to take a look into this.

A general policy Guix has is that tools put into initrd should be statically-linked with all libraries, is there an easy way to compile the tools only (without the kmod) statically linked? In Guix this is generally done as a separate variant package with a /static suffix. I'm thinking it would be nice if it were possible to compile only zpool with statically-linked libraries, I imagine adding LDFLAGS=-static to the configure script, then have the "build" part use make cmd/zpool/zpool, then have the "install" copy the zpool binary to the build system output directory.

What I'm thinking is that, for /-on-ZFS for Guix, we'd have to do:

• At initrd generation, include the ZFS kmods and the statically-linked zpool.
• At boot while initrd is the root filesystem, load the ZFS module (Guix uses syscalls for loading modules this early in the boot process), zpool import -N rootpool.
  • If my understanding is correct, that will cause ZFS to scan for devices to look for the rootpool, then assemble the array and not mount anything inside it (important since the initrd root filesystem is not yet the final filesystem.
• Mount the rootpool/rootfilesystem onto the temporary /root location (which will be pivoted to later). Guix actually uses a syscall for mounting filesystems while still in the initrd instead of installing mount into the initrd, I presume this will work correctly as long as the rootpool/rootfilesystem has mountpoint=legacy?

If so, it seems to me that initrd should contain only the zpool tool. I notice that zpool and zfs mount by actually execing into the mount tool and secretly passing in -o ...,zfsutil. Just to confirm, if I use zpool import -N rootpool, then will zpool not exec into mount? In initrd Guix doesn't have the mount tool, as noted above it uses a direct syscall instead.

Alternately, is there a convenient syscall I can instead use to have ZFS import just rootpool without mounting anything inside it?

[behlendorf]

is there an easy way to compile the tools only (without the kmod) statically linked?

There may be but this isn't something I've ever really tried. It's easy enough to instruct configure to just build the user components, you can do so like this ./configure --with-config=user. I'm not sure what would be needed to get everything built statically. But it wouldn't surprise me if it was possible with some adjustments to the build system.

If my understanding is correct, that will cause ZFS to scan for devices to look for the rootpool, then assemble the array and not mount anything inside it (important since the initrd root filesystem is not yet the final filesystem.

That's right.

I presume this will work correctly as long as the rootpool/rootfilesystem has mountpoint=legacy?

Yes, it should. The zfs command actually prefers to call mount(2) directly now. It used to exec mount(8) but no longer does by default. So you could probably use a statically built version of the zfs tool to do the mount if that's preferable (commit 501a151).

There is an ioctl() interface which for all of the administrative commands, but it's not something you'd want to call directly. There's a lot of user space code involved in scanning for the pool and providing a configuration to the kernel which can be used to import it.