zfs diff shows no changes but written property has random values as high as 800M, with encrypted datasets

[digitalsignalperson]

Hi, I'm looking for help on how to investigate my datasets showing no output with zfs diff, but the written property is not zero (ranging from ~100K to ~700-800M). If I try to receive a new incremental snapshot to any of these datasets with this state, it fails with "cannot receive incremental stream: destination {path} has been modified". While I could use recv -F, I want to figure out what is happening and what these unexpected writes are. And is this a bug, or an issue with my system, etc.

This system received ~100 raw encrypted incremental snapshots about a week ago with the latest snapshot @replication-2024-01-29. All the datasets were mounted since then, but nothing should have been written to any/most of them.

I can make a @temp snapshot and send an incremental for any of these datasets from @replication-2024-01-29 to @temp and then look at it with zstream dump -v or -d, but I'm not sure how to interpret this data to figure out what it means.

Doing a scrub now to confirm there is no corruption.
I've also confirmed I can use -F on any of them and receive new snapshots.

[digitalsignalperson]

for example
zfs diff xpool/enc/0006@replication-2024-01-29 shows nothing. The written property says 11.3M
then I do touch /mnt/0006/test
then zfs diff xpool/enc/0006@replication-2024-01-29 shows

+        /mnt/0006/test
M       /mnt/0006/

so it does show diffs for changes to files. Just not this mystery data.

zfs snapshot xpool/enc/0006@test
zfs send -i @replication-2024-01-29 xpool/enc/0006@test | zstream dump -d > dumped

if I look through this, potentially relevant bits:

• some sections with a lot of zeros
• sections with FREEOBJECTS / OBJECT and hex data with user.DOSATTRIB (seems like TONS of these)
• similar but with system.posix_acl_access
• and lots of these hex sections starting with ZP/

I have atime off. Maybe I need to look for something recursively settings permissions/ACLs?

[rincebrain]

DOSATTRIB would usually be Samba-related, I think.

[digitalsignalperson]

weird cause I'm not using samba at all

[rincebrain]

There are other ways, that's just the most common that came to mind, since I think that's the style used for smuggling Windows/DOS style attributes into xattr bits.

[digitalsignalperson]

OK this is weird. So I just force received new data on all of them. All datasets now have written 0 except one. If I unmount it, roll it back, written is 0. If I then mount it, the written property within 2 seconds jumps to 788M. I can repeat this, unmount, rollback, mount, with a watch -n 0.5 zfs get written -t filesystem -o name,value and see it super quickly jump to 788M.

I tried setting inotifywait -f -m on the mount point immediately after mounting, but it shows nothing while the used property grows.

Ummm ok and if I zfs umount -a, then zfs mount -a, instantly all of the datasets get random written values.

zfs umount -a
zfs list -t snapshot -o name -H | grep replication-2024-02-03 | xargs -I {} zfs rollback {}
zfs mount 
# confirms nothing mounted
zfs get written -t filesystem -o name,value
# written is 0 for all
zfs mount -a
zfs get written -t filesystem -o name,value
# now all of them get random written values

I can repeat this and it does the same

[digitalsignalperson]

Ah I see. Is there any open issues on this?

[rincebrain]

#12580 at least

[digitalsignalperson]

It's super repeatable for me right now. But for me it doesn't matter if zfs recv -u is used or not. I just tried receiving with the encryption key unloaded, then loaded it, then mounted it. Still grew instantly to 788M written. And on the sending side, I can take a new snapshot, send a new incremental again, and it still results in the same thing of the receiving side changing after I mount it.

Is there any queue stuff I can check to confirm if it's the same thing? Like your last comment in the linked issue mentioning deadlist

[rincebrain]

I mean, receive -u just means it doesn't instantly happen, it wouldn't stop it from happening on mount, that's all that -u is doing here.

[digitalsignalperson]

just noted below that if readonly=on during mount, it prevents the replay/written thing. You can then turn readonly=off and it doesn't do the replay. But if you umount and mount again with readonly=off, it does do the replay write stuff.

zfs umount -a
zfs list -t snapshot -o name -H | grep replication-2024-02-03 | xargs -I {} zfs rollback {}
zfs set readonly=on xpool/enc
zfs mount -a
zfs mount  # confirms they are all mounted
watch -n 0.5 zfs get written -t filesystem  # everything stays 0
zfs set readonly=off xpool/enc
watch -n 0.5 zfs get written -t filesystem  # everything stays 0
zfs umount -a
zfs mount -a
watch -n 0.5 zfs get written -t filesystem  # everything grows

[rincebrain]

zfs diff is also incomplete in the general case, so you shouldn't rely on it. Notably, it won't be able to map names for things with hardlinks in many cases, so it'll just print an error about trying to resolve the object id (and, on much older versions of OpenZFS, just error out there).

See #6335

(There are other cases that are rarer that it outputs incomplete or incorrect information, but they're much rarer, and without this fixed, I've personally just never cared enough to run them down, since it's already too unreliable.)

[mpeter50]

Does that mean that there are cases where it wont even show an error when producing incorrect output?
If so, it is such a shame that zfs diff is unreliable. It is the only tool that could diff snapshots in a timely manner, and without a large system load.

[IvanVolosyuk]

I usually make receive destination read-only 'zfs set readonly=on dataset' to avoid this kind of surprises.

…

On Sun, Feb 4, 2024 at 11:43 PM andy ***@***.***> wrote: It's super repeatable for me right now. But for me it doesn't matter if zfs -u is used or not. I just tried receiving with the encryption key unloaded, then loaded it, then mounted it. Still grew instantly to 788M written. And on the sending side, I can take a new snapshot, send a new incremental again, and it still results in the same thing of the receiving side changing after I mount it. Is there any queue stuff I can check to confirm if it's the same thing? Like your last comment in the linked issue mentioning deadlist — Reply to this email directly, view it on GitHub <#15853 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HP7U62PMEL3KVQMQA3YR57ARAVCNFSM6AAAAABCYXKYB6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGNRQGQZTM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[digitalsignalperson]

This is interesting that if zfs umount -a then rollback, then zfs set readonly=on xpool/enc, then zfs mount -a, I don't get the random write values, and then if I zfs set readonly=off xpool/enc, the write values are still zero. But then if I zfs umount -a and zfs mount -a again, NOW it applies all the random writes. So mounting with readonly=on prevents whatever replay thing from getting triggered maybe?

I'm not sure if in general I can use readonly=on for my application, but maybe I can figure out a workaround.
My setup was to allow read/write on multiple destinations, and whenever a new change is written (which is rare, so no conflicts) it gets snapshotted and shared between multiple systems. This was working without any issue until now trying to make the same system with raw encrypted send/recv.

[rincebrain]

Well, yes, it doesn't remount the dataset when you do that. (I think?)

Or, if it does, then not replaying the delete queue is a bug, probably.

[digitalsignalperson]

What do you mean by doesn't remount?

I can do this:

zfs umount -a
zfs list -t snapshot -o name -H | grep replication-2024-02-03 | xargs -I {} zfs rollback {}
zfs set readonly=on xpool/enc
zfs mount -a
zfs mount  # confirms they are all mounted
watch -n 0.5 zfs get written -t filesystem  # everything stays 0
zfs set readonly=off xpool/enc
watch -n 0.5 zfs get written -t filesystem  # everything stays 0

and the written prop stays zero. And everything is mounted read-write. I left it like this for 16 hours and written is still zero.

But then if I simply

zfs umount -a
zfs mount -a
watch -n 0.5 zfs get written -t filesystem  # everything grows

I see the written prop quickly grows by those random values

[IvanVolosyuk]

Sounds like there is a service which scans all the newly mounted filesystems and does something with them. It doesn't scan read-only filesystems or unable to change anything there. Do you have atime=on? Try atime=off may be, but for reliable 'zfs receives' I recommend having receive destination readonly anyway.

…

On Mon, Feb 5, 2024 at 4:17 PM andy ***@***.***> wrote: What do you mean by doesn't remount? I can do this: zfs umount -a zfs list -t snapshot -o name -H | grep replication-2024-02-03 | xargs -I {} zfs rollback {} zfs set readonly=on xpool/enc zfs mount -a zfs mount # confirms they are all mounted watch -n 0.5 zfs get written -t filesystem # everything stays 0 zfs set readonly=off xpool/enc watch -n 0.5 zfs get written -t filesystem # everything stays 0 and the written prop stays zero. And everything is mounted read-write. I left it like this for 16 hours and written is still zero. But then if I simply zfs umount -a zfs mount -a watch -n 0.5 zfs get written -t filesystem # everything grows I see the written prop quickly grows by those random values — Reply to this email directly, view it on GitHub <#15853 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HJ2OMMQSOARJQPF6ITYSBTONAVCNFSM6AAAAABCYXKYB6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGNRVGQ4TO> . You are receiving this because you commented.Message ID: ***@***.***>

[digitalsignalperson]

Indeed atime is off. There shouldn't be any weird services running or anything scanning for files. It's just booted into a virtual terminal on a minimal arch install. I'll try firing up a clean VM and send it the same data to see if it does the same thing

[rincebrain]

Ivan, no, that's not what's happening. Go read #12580.

[digitalsignalperson]

I notice if I zfs diff between snapshots, I see a line with (on_delete_queue), so evidence towards the issue you link

e.g. for xpool/enc/0010 between snapshots the zfs diff contains either

M /mnt/0010(on_delete_queue)
or
+ /mnt/0010(on_delete_queue)

where /mnt/0010 is the mountpoint.

Edit: but not all the affected datasets show this, only some

[IvanVolosyuk]

Interesting issue. So, if read-only property change affects the outcome - this looks like an accounting bug at least or even storage data leak for this edge-case.

…

On Mon, Feb 5, 2024 at 6:02 PM Rich Ercolani ***@***.***> wrote: Ivan, no, that's not what's happening. Go read #12580. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[rincebrain]

What?

[digitalsignalperson]

reproducer here #12580 (comment)

it only reproduces if using encryption