Question: Is there a reliable way to programmatically determine whether a particular dataset is currently being received into?

[jgottula]

Background information

I've recently been doing some send-and-receive operations on my local system in an attempt to defrag some very-fragmented datasets. And those datasets happen to have hourly auto-snapshots enabled.

The problem I've been encountering is: if the receive operation takes more than an hour (and for most of these datasets, it definitely does), then I'm guaranteed that zfs-auto-snapshot will kick in during that time interval, and it will create a new hourly snapshot on the receiver dataset. However, the receiver dataset is in the middle of receiving a bunch of old snapshots from the sending dataset. So what happens is, the first few snapshots will get received from the sender dataset; then zfs-auto-snapshot steps in and "helpfully" creates a brand new snapshot on top of the most recently received snapshot; and then, shortly thereafter, the whole zfs recv operation fails.

(Let's say that the sender dataset had snapshots A, B, C, D, E. And at the top of the hour, the receiver dataset has, so far, received snaps A, B, C, and is currently working on receiving D. Well now, since it's the top of the hour, zfs-auto-snapshot creates an auto snapshot Z on top of C; and so when zfs recv attempts to put the C-->D data into place, it obviously can't deal with the discrepancy between its expectation that the most recent snap is C and the reality that the most recent snap is actually Z; and so the receive fails.)

The commands I'm using are:

zfs snap -r pool/Data@defrag-2020feb
zfs send -RLecv pool/Data@defrag-2020feb | zfs recv -sv pool/Data2

I do want to keep the auto-snapshot properties enabled on the datasets that I'm send/recv'ing. But I don't want zfs-auto-snapshot to touch any datasets which are currently being recv'd to. And as it happens, I've already been working on my own custom hacked-up version of zfs-auto-snapshot anyway, so this seemed like a straightforward feature I could add into it.

But when I got down to the details of "how do I determine whether a particular dataset is currently the target of a receive operation", things got nebulous. The best I could come up with was to query the receive_resume_token property, and use that basically as a "yeah this dataset is still being received" indication, when it's set to something non-blank. This does require that the zfs recv operation is done with the -s flag, which is definitely not optimal. And it feels like a hack.

Another thing I found, but was much less confident in, was the presence of pool/Data2/%recv, when explicitly asked for in the invocation of zfs list. But I don't know as much about the reliability of this as an indicator in all cases; and it's also inefficient to have to explicitly query for sub-datasets that may or may not exist (and which will almost always not exist) for every single dataset that I want to do auto-snaps on.

My question

So ultimately I just got to wondering why there isn't a simple, reliable way to find out whether a particular dataset is currently the target of a receive operation. And I grant that perhaps there is a way and I've just overlooked it; but I've done a lot of digging, and I'm pretty sure that there just isn't one. And that seems kind of odd to me.

I did find dmu_objset_is_receiving in the module code, which appears to be a function that pretty much does exactly what I want; but even just accessing that function from a C program via libzfs calls seems like it would be quite a pain. And I'm not even 100% certain whether that function is accessible from user mode in the first place.

Am I asking the wrong question?

I openly admit that, in the pursuit of fixing things, I sometimes go too far down one particular path and end up asking "the wrong question", instead of backing up and stopping at a more reasonable point earlier along the chain of logic.

So, I'm potentially open to other solutions here that maybe don't need to resort to finding a reliable way to tell whether a dataset is currently being recv'd to. (I do still think that it would be useful to be able to easily access that information though.)

Sort of along these lines, one thought that crossed my mind was, why is it even allowed to make a new snapshot on a dataset, when that dataset is currently being recv'd to? Seems kind of wrong. But perhaps it's justifiable in certain circumstances, where I guess maybe you'd want a receive-side snapshot to cause the recv operation to abort in deference to it, for whatever reason? It's hard to think of circumstances where that would make sense... but perhaps it would be less nonsensical when dealing with something like an incremental single-snapshot receive, rather than a replication receive. I don't know.

Side note: strange error message

The error that I get in the situation described above, when zfs recv fails, is very odd.

cannot receive incremental stream: kernel modules must be upgraded to receive this stream.

The sender and receiver datasets are literally on the same system; and the problem is definitely happening due to a new snapshot being inserted in the middle of the receive process. But for some reason, I guess the kernel module is sending back ENOTSUP and so I get this bizarre error message. (I haven't yet gone to the trouble to try to trace back where that ENOTSUP is specifically coming from.)

System information

Type	Version/Name
Distribution Name	Arch Linux
Linux Kernel	5.4.5
Architecture	AMD64
ZFS Version	0.8.0-495_g82e996c26
SPL Version	0.8.0-495_g82e996c26

[GregorKopka]

one thought that crossed my mind was, why is it even allowed to make a new snapshot on a dataset, when that dataset is currently being recv'd to? Seems kind of wrong.

I agree. Does not make sense, in case I want to abort the recv there is kill.

To test a dataset for being in recv... maybe something like ps -C zfs -o args | grep <dataset> ?
I would possibly do it twice, once prior to the snapshot and once afterwards - and remove the new snapshots in case the second invocation had found something.

[allanjude]

If the zfs recv operation is ongoing, the dataset will be locked, and a snapshot will not be able to be created:

# zfs recv zroot/testing < replication_stream.zfs &
# zfs snapshot zroot/testing@testing
cannot create snapshot 'zroot/testing@testing': dataset is busy