prov/efa: Do not create SHM peer when SHM is disabled

When the user sets FI_EFA_ENABLE_SHM_TRANSFER=0, do not create a SHM endpoint. The current behavior is that we just don't send via SHM when one sets FI_EFA_ENABLE_SHM_TRANSFER=0, but we still create and poll the peer endpoint. New behavior is to not create SHM endpoint when peer is disabled. Signed-off-by: Seth Zegelstein <[email protected]>
ofiwg · Sep 26, 2023 · 6b62f34 · 6b62f34
1 parent 55f7fd1
commit 6b62f34
Show file tree

Hide file tree

Showing 2 changed files with 6 additions and 2 deletions.
diff --git a/prov/efa/src/efa_domain.c b/prov/efa/src/efa_domain.c
@@ -140,7 +140,11 @@ static int efa_domain_init_rdm(struct efa_domain *efa_domain, struct fi_info *in
 {
 	int err;
 
-	efa_shm_info_create(info, &efa_domain->shm_info);
+	efa_domain->shm_info = NULL;
+	if (efa_env.enable_shm_transfer)
+		efa_shm_info_create(info, &efa_domain->shm_info);
+	else
+		EFA_INFO(FI_LOG_CORE, "EFA will not use SHM for intranode communication because FI_EFA_ENABLE_SHM_TRANSFER=0\n");
 
 	if (efa_domain->shm_info) {
 		err = fi_fabric(efa_domain->shm_info->fabric_attr,

diff --git a/prov/efa/src/efa_env.c b/prov/efa/src/efa_env.c
@@ -182,7 +182,7 @@ void efa_env_define()
 	fi_param_define(&efa_prov, "tx_queue_size", FI_PARAM_INT,
 			"Defines the maximum number of unacknowledged sends with the NIC.");
 	fi_param_define(&efa_prov, "enable_shm_transfer", FI_PARAM_INT,
-			"Enable using SHM provider to perform TX operations between processes on the same system. (Default: 1)");
+			"Enable using SHM provider to perform TX/RX operations between processes on the same system. (Default: 1)");
 	fi_param_define(&efa_prov, "use_zcpy_rx", FI_PARAM_INT,
 			"Enables the use of application's receive buffers in place of bounce-buffers when feasible. (Default: 1)");
 	fi_param_define(&efa_prov, "set_cuda_sync_memops", FI_PARAM_INT,