prov/verbs: Add support for IBV_ACCESS_RELAXED_ORDERING

[sydidelot]

IBV_ACCESS_RELAXED_ORDERING allows the system to reorder
Send/Write/Atomic operations to improve performance.

The patch enables IBV_ACCESS_RELAXED_ORDERING if the application
has requested no ordering in TX/RX attributes.

[shefty]

I was wondering if we could OR in the msg_order values every time we allocate an endpoint. That at least updates the tx/rx_msg_order fields. However, it's possible for the app to register memory prior to creating any endpoints. In that case, we would incorrectly use relaxed ordering.

So, I'm wondering if we have to have a domain that only supports relaxed ordering. If the app calls fi_info with no ordering set, we can return that domain as one of the options. Endpoints allocated on that domain are only unordered.

This brings up that I think most apps need ordering for sends. AFAICT, the relax ordering here mostly impacts RMA ordering: WAW, RAR, etc. We may only need to check that those bits are not set.

[sydidelot]

@shefty I have updated the PR based on your suggestion to create one domain for relaxed ordering.
The PR only supports msg endpoints for now, but I believe the feature could easily be extended for XRC and DGRAM domains.
I verified the code with a modified version of fi_pingpong that sets tx_attr->msg_order = rx_attr->msg_order = 0 before calling fi_getinfo().

[shefty]

CI failure is related. Changes break verbs;ofi_rxd. Most or all fabtests fail, but here's a simple one:

fi_getinfo_test -s cb14-ib0 cb14-ib0 -p "verbs;ofi_rxd"
13:41:03    timestamp: 20231010-203537+0000
13:41:03    result: Fail

Quickly scanning the changes didn't point anything to me. AFAICT, dgram endpoints shouldn't have been impacted by this change.

[iziemba]

Question: Is the correct behavior to zero the message order flags? Or, is the correct behavior to set max_order_waw_size to zero? The thinking here is that for FI_MSG EPs, the operations would still be ordered in the network. It is target PCIe write operations which will be unordered with respect to each other.

[shefty]

IMO, the feature isn't well enough documented at the verbs layer to figure out what exactly it does. The PR comment indicates that sends could be received out of order. Is that true, or would sends still match receive buffers in order? Hmm... the latter seems reasonable, since matching would be done by the NIC. That implies that WAW ordering, and RAR I guess, is not guaranteed.

[thomasgillis]

Following a recent issue I have with PCIe ordering (#9621), I am worried that the possible reordering of the PCIe writes at the target will break the completion ordering of RDMA_WRITE_WITH_IMM_DATA.
I couldn't find many details on IBV_ACCESS_RELAXED_ORDERING, is there any guarantee that it will not break that?

[shefty]

I haven't found any information on what IBV_ACCESS_RELAXED_ORDERING actually does. If completions can be written before the message data is received, then it's use seems suspect. I will try to find out more details.

[shefty]

Based on initial feedback, IBV_ACCESS_RELAXED_ORDERING should not impact completions. After reading a completion, all data should be present. I'm still investigating what it does beyond that and if it impacts message ordering.

[shefty]

See linux-rdma/rdma-core#1413 for a proposed update to better document the change in verbs API behavior. My understanding is that @iziemba 's earlier proposal to disable waw ordering is correct.

[sydidelot]

@shefty Thanks for the feedback and for the clarification of IBV_ACCESS_RELAXED_ORDERING.
Should we enable relaxed ordering if the user provides hints->ep_attr->max_order_waw_size == 0 as suggested by @iziemba earlier in the discussion?

[shefty]

I honestly still don't know what that flag does. It sets the relaxed ordering bit for PCI transfers, but it's unclear when that bit is set.
I submitted text for it to libibverbs, which was merged, but I don't know if that text was actually correct. WAW ordering may actually be maintained, so the situation may be better than it seems. I'm still trying to determine this.

The flag applies at the receiving side. If WAW ordering is impacted, I would also check/update the msg_order flags.

[sydidelot]

I just discovered that the new CXI provider has support for Relaxed Ordering: https://github.com/ofiwg/libfabric/blob/main/man/fi_cxi.7.md#pcie-ordering
We could check the source code to see how it's implemented in this provider :-)