Zpool wont import anymore on startup

[mstemmer]

It must have happend with some updates, but in Ubuntu 22.04 suddenly my zpool wouldnt import anymore automatically upon reboot. Never had that the years before...Works fine when using zpool import -a though...
Now, I have updated Ubuntu to 24.01 (6.8.0-44-generic) and zfsutils runs with version 2.2.2. Still the same issue.
I have found that the import by cache service reports a dependecy error, but cant track down, what this means.

○ zfs-import-cache.service - Import ZFS pools by cache file
     Loaded: loaded (/usr/lib/systemd/system/zfs-import-cache.service; enabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:zpool(8)

Sep 13 14:24:37 L-xxxx systemd[1]: Dependency failed for zfs-import-cache.service - Import ZFS pools by cache file.
Sep 13 14:24:37 L-xxxx systemd[1]: zfs-import-cache.service: Job zfs-import-cache.service/start failed with result 'dependency'.

Tried to set the cachefile again, but no change...
Also any changes I make in /etc/defaults/zfs are ignored...why?
Any ideas would be very welcome!

[n0xena]

although I'm no expert I found this info:

a failure with reason "dependency" hints towards any other service this service depends on has failed

zfs-import-cache has these dependencies:

Requires=systemd-udev-settle.service
After=systemd-udev-settle.service
After=cryptsetup.target
After=multipathd.service
After=systemd-remount-fs.service

zfs-import-cache failing with reason "dependency" means that any of the above resulted in a failed state

I would start by investigating the status of the above services at the time zfs-import-cache is started - this should hint toward the cause

[mstemmer]

Thank you for the hint!
It was indeed the systemd-udev-settle.service, which stalled zfs and caused it to crash during boot.

I removed that dependecy from zfs-import-cache.service, zfs-import-scan.service and zfs-load-module.service and everything works fine again! Seems that systemd-udev-settle.service is not needed at all for zfs to initiate?!

Anyways, this fixed my problem.

[n0xena]

well, I won't call it a "fix" - rather you got rid of one of the symptoms - it's like suppressing coughing instead of curing the infection

anyway - instead of just remove udev from the depencies you shoukd rather investigate why it fails
udev is used for stuff like persistent device naming which should be preferred over kernel naming which can change on reboot
check if you have bad/outdated udev rules and also if udev-settle fails itself or if it, too, fails by dependency and walk down this path until you find the actual root cause and fix it

[mstemmer]

It counts as a "fix" for me, but I get your point!

systemd-udev-settle.service - Wait for udev To Complete Device Initialization
     Loaded: loaded (/usr/lib/systemd/system/systemd-udev-settle.service; static)
     Active: inactive (dead)
       Docs: man:systemd-udev-settle.service(8)

No errors logged here. Any good idea how to check bad/outdated udev rules?

[n0xena]

https://www.freedesktop.org/software/systemd/man/latest/systemd-udev-settle.service.html is quite interesting:

Using this service is not recommended.

So it seems this actually is an issue of ZFS to "abuse" udev-settle to wait for udev have discovered all hardware.

I get the idea: ZFS depends on all storage discovered and online before scanning for pools to import

btw: I won't use zfs-import-scan anyway as all pools the systems uses and expects to be online should be in cache anyway - import-scan is only useful if you expect to discover non-cached pools at boot

so your "solution" to remove udev-settle seems to be the right direction, at least seen from udev

actually I would raise this as an issue to be checked and maybe removed in the future
zfs brings its own i/o stack anyway so it should be able to implement a similar function without havin to wait for udev

[mstemmer]

That is interesting. I am using a SuperMicro server, which might also be part of why udev-settle doesnt seem happy during boot...
Alright, seems that this indeed is not needed and might be removed in future ZFS versions.

Also, yes, I am actually using zfs-import-cache and might not have been necessary to modify also zfs-import-scan. I did btw also remove udev-settle from zfs-load-module.service (updated my response further up).

[AllKind]

Also any changes I make in /etc/defaults/zfs are ignored...why?

Because that file is only for systems that do not use systemd, but instead use the classic init scripts.