Improve ergonomics on following Kad querys

[drHuangMHT]

When you try to query for a node in the DHT, you'll be given a QueryId and you need to listen to the events emitted by swarm to see how the query progresses. It's not hard to implement this myself, but I think we can do better. Instead of giving QueryId, we can give a mpsc channel to track the progress of a query. I think it should be easy since we do keep a pool for all active queries.
Again, it's totally fine for me or others to implement something like event listeners and it will still be a thing even not for following DHT queries, but I'm still happy to see it's made built-in.

[thomaseizinger]

I think that is not a bad idea although we haven't done something like that before.

@mxinden thoughts?

[mxinden]

Overall, yes, I think this is a good user experience improvements. I don't yet know how to solve the edge case where the user doesn't read from the channel. Should the libp2p-kad NetworkBehaviour await a new slot on the channel?

[drHuangMHT]

What about an unbounded channel? If the user fires and forget, then just cause a memory leak, otherwise why the user initiates the query?

[drHuangMHT]

The elephant in the room is how to adapt to other asynchronous libraries. That involves another generics parameter to be typed in by users.

[thomaseizinger]

I am not sure if implementing back-pressure towards the application code makes sense. If I understand the kademlia code correctly, then for queries such as GetClosestPeers, we have an allowed parallelism configuration. Once that is reached, we wait until some of the queries finish. We could do something similar where we don't issue more network requests if the user doesn't give us a slot in the channel that we should send the results in.

However, network requests are likely much slower than the application code. Chances are we will never hit this backpressure mechanism because the application code will read the results as they come in.

The elephant in the room is how to adapt to other asynchronous libraries. That involves another generics parameter to be typed in by users.

I am not sure what you mean here. The channels in futures are executor-agnostic.

[drHuangMHT]

We could do something similar where we don't issue more network requests if the user doesn't give us a slot in the channel that we should send the results in.

Well it sounds... Yeah that's the way to do it.

I am not sure what you mean here. The channels in futures are executor-agnostic.

Didn't know that, that's good.

[drHuangMHT]

In asynchronous context the order of registering event listeners and emitting "query progressed" event is not guaranteed, which means loss of event(use QueryId to follow a specific query, QueryId obtained after the query be issued.) or overhead(broadcast event to all listeners interested in this event, not a specific QueryId). To me neither seems acceptable, but for reliability the latter is preferred. But to be honest, who would issue tons of queries at the same time with tons of listeners? The problem may not actually be a problem in practice.

[thomaseizinger]

I am in favor of the proposed API but it is going to be tricky to implement. We need to poll all those channels internally and make sure the channel of one query isn't stalled by the user not reading from another channel.

[drHuangMHT]

So is it really worth the hassle? I think we need to weigh the pros and cons more carefully.

[thomaseizinger]

I think it is a plus in ergonomics for the user. Internal complexity is a fine trade-off for that. That is kind of the point of abstractions and libraries, isn't it?

I don't think it is very high priority though.