On a more dynamic approach for writing network behaviours

[thomaseizinger]

Context & Problem statement

In rust-libp2p, users write NetworkBehaviours to implement the p2p network part of their application. NetworkBehaviours are composed together and driven by a Swarm. A more abstract way of looking at this is that NetworkBehaviours are plugins and Swarm is a kind of runtime or host for these plugins.

In any plugin system, one needs to define an API that is used between the host and the plugins. For us, this API takes the form of the NetworkBehaviour trait. This API - naturally - grew over time as rust-libp2p is changed to support more and more usecases. One way of dealing with the increased API surface has been proposed in #2832.

Whilst being a useful change, I think it only scratches the surface of the situation. The NetworkBehaviour and by extension the ConnectionHandler API are inherently stateful. Even the simplest of usecases require users to "listen" to certain callbacks, extract data they are interested in, store it in the behaviour and later reason about this state again to perform the actions their network code would like to perform.

This approach creates a problem: It is very difficult to modularise and thus reuse state which I believe is one of the core reasons why we see duplication of certain patterns across various NetworkBehaviours. The statefulness also scales bad with the number of requirements for a single NetworkBehaviour. Splitting networking logic into individual protocols helps here because we can compose multiple NetworkBehaviours together. Yet, it is not the ideal solution. Within one NetworkBehaviour, we may still collect all kinds of state of several requirements that are all slightly related but without an ability to separate these requirements on a source-code level.

Prior art

Games and game engines actually have a very similar problem. Games are littered with state that all needs to be kept in sync and updated consistently. Modern game engines do this with an ECS (entity-component-system). An ECS is pretty much the opposite of object-orientation. Instead of grouping together data and calling functions on it, data (entities) is stored separately and modified via systems. In an ideal world, a system only deals with one functional requirement. A system somehow expresses, which data it is modifying and the entire game is then created by composing the system together.

Bevy is a new but already very popular ECS-driven game engine in Rust. Here is a heavily commented example of what this looks like: https://github.com/bevyengine/bevy/blob/main/examples/ecs/ecs_guide.rs

One interesting thing here is that the logic for checking whether a player won is separate from the logic that the game is over: https://github.com/bevyengine/bevy/blob/c4d1ae0a4744ebed8366ff6d4ea5bd77af878744/examples/ecs/ecs_guide.rs#L111-L140

This modularization is very powerful and works because each system has "dependencies" that are expressed in its function signature. This crazy trait allows system functions to have different parameters yet all of them can be passed to App::add_system.

axum - a web library built on top of hyper - uses a very similar¹ approach to define its request handlers: https://github.com/tokio-rs/axum/blob/main/examples/jwt/src/main.rs

Idea

The idea I'd like to discuss is whether we should attempt at providing such an interface in rust-libp2p. Here is an example of a system:

async fn identify_system(endpoint: Endpoint, incoming_stream: IncomingStream) {
    incoming_stream.send(Identify { observed_address: endpoint.get_address() }).await;
}

The idea would be that one can take a requirement of the protocol and implement it in the form of a function that just takes all required dependencies as parameters. The runtime would call this system / function every time there is a new IncomingStream. No tracking of state needed, it is just there as function arguments. Obviously, many details would need to be fleshed out but I believe that this idea has a lot of potential.

Expected benefits

• No more passing of data back and forth between ConnectionHandler and NetworkBehaviour
• It would be possible to offer much more fine-grained to control to people over where they'd like their system to run, i.e. bevy I believe runs as many systems in parallel as possible but one can also define a specific order. We could do something similar in terms of a task / thread pool.
• Modularisation of functionality repeated across NetworkBehaviour: Similar to how axum provides a few core utilities to extract data out of an HTTP request, we could easily offer building blocks that gather certain data together, for example the list of currently connected peers.
• One could even imagine that plugins could exchange information this way. Imagine the ping plugin "publishes" the latency to each peer somehow and other plugins can just request this as part of a system.

Potential downsides

With great power comes great responsibility. A dynamic system such as the above provides a lot of flexibility and thus little guidance over what once can / should do. The current API is fairly rigid but it guides the user to some degree. The proposed API would probably require more documentation but projects like bevy show that it can be done successfully and actually leads to more maintainable code down the line.

Footnotes

1. I learned recently that it is actually inspired by bevy. ↩

[mxinden]

Sorry for the late reply.

I don't think I understand ECS well enough (after having read the bevy example) in order to understand the identify example above.

It might be easier to tackle this in a synchronous discussion. I am adding it to the rust-libp2p community sync agenda again.

[MarcoPolo]

I like the idea! I'm generally a fan of ECS systems (I built a game engine using it ~9 years ago!). I think there are some subtleties with networking since everything is async/distributed that may be a little tricky. But I think it seems fun and worth exploring.

Bevy is an interesting example. It's wild how they can leverage the type system so much. I don't really understand how it works, but it looks cool.

I'd encourage a prototype and an example implementation of a protocol to see how it feels. Happy to beta test the system as well :)

[thomaseizinger]

It is explained here: https://github.com/alexpusch/rust-magic-function-params

Not actually that magical once it is uncovered but I guess that applies to all types of magic?