From c8d681221543e2ee3b4906380216deace510903a Mon Sep 17 00:00:00 2001 From: xbw <78337767+xbw22109@users.noreply.github.com> Date: Tue, 29 Apr 2025 03:57:14 +0000 Subject: [PATCH] Updated the documentation about shared memory. Changes to be committed: modified: docs/launching-apps/localhost.rst modified: docs/tuning-apps/networking/shared-memory.rst Signed-off-by: xbw <78337767+xbw22109@users.noreply.github.com> --- docs/launching-apps/localhost.rst | 78 ----------- docs/tuning-apps/networking/shared-memory.rst | 121 +++++++++++++++--- 2 files changed, 102 insertions(+), 97 deletions(-) diff --git a/docs/launching-apps/localhost.rst b/docs/launching-apps/localhost.rst index 0cb6585a429..37876952ae5 100644 --- a/docs/launching-apps/localhost.rst +++ b/docs/launching-apps/localhost.rst @@ -35,81 +35,3 @@ peers: and destination MPI processes can share memory (e.g., via SYSV or POSIX shared memory mechanisms). -Shared memory MPI communication -------------------------------- - -.. error:: TODO This should really be moved to the networking section. - -The ``sm`` BTL supports two modes of shared memory communication: - -#. **Two-copy:** Otherwise known as "copy-in / copy-out", this mode is - where the sender copies data into shared memory and the receiver - copies the data out. - - This mechanism is always available. - -#. **Single copy:** In this mode, the sender or receiver makes a - single copy of the message data from the source buffer in one - process to the destination buffer in another process. Open MPI - supports three flavors of shared memory single-copy transfers: - - * `Linux KNEM `_. This is a - standalone Linux kernel module, made specifically for HPC and MPI - libraries to enable high-performance single-copy message - transfers. - - Open MPI must be able to find the KNEM header files in order to - build support for KNEM. - - * `Linux XPMEM `_. Similar to - KNEM, this is a standalone Linux kernel module, made specifically - for HPC and MPI libraries to enable high-performance single-copy - message transfers. It is derived from the Cray XPMEM system. - - Open MPI must be able to find the XPMEM header files in order to - build support for XPMEM. - - * Linux Cross-Memory Attach (CMA). This mechanism is built-in to - modern versions of the Linux kernel. Although more performance - than the two-copy shared memory transfer mechanism, CMA is the - lowest performance of the single-copy mechanisms. However, CMA - is likely the most widely available because it is enabled by - default in several modern Linux distributions. - - Open MPI must be built on a Linux system with a recent enough - Glibc and kernel version in order to build support for Linux CMA. - -Which mechanism is used at run time depends both on how Open MPI was -built and how your system is configured. You can check to see which -single-copy mechanisms Open MPI was built with via two mechanisms: - -#. At the end of running ``configure``, Open MPI emits a list of - transports for which it found relevant header files and libraries - such that it will be able to build support for them. You might see - lines like this, for example: - - .. code-block:: text - - Shared memory/copy in+copy out: yes - Shared memory/Linux CMA: yes - Shared memory/Linux KNEM: no - Shared memory/XPMEM: no - - The above output indicates that Open MPI will be built with 2-copy - (as mentioned above, 2-copy is *always* available) and with Linux - CMA support. KNEM and XPMEM support will *not* be built. - -#. After Open MPI is installed, the ``ompi_info`` command can show - which ``smsc`` (shared memory single copy) components are - available: - - .. code-block:: text - - shell$ ompi_info | grep smsc - MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.1.0) - - This Open MPI installation only supports the Linux CMA single-copy - mechanism. - -.. note:: As implied by the SMSC component names, none of them are - supported on macOS. macOS users will use the two-copy mechanism. diff --git a/docs/tuning-apps/networking/shared-memory.rst b/docs/tuning-apps/networking/shared-memory.rst index caf723de4cc..7c40693cd76 100644 --- a/docs/tuning-apps/networking/shared-memory.rst +++ b/docs/tuning-apps/networking/shared-memory.rst @@ -1,11 +1,9 @@ Shared Memory ============= -.. error:: TODO This section needs to be converted from FAQ Q&A style - to regular documentation style. -What is the sm BTL? -------------------- +The sm BTL +---------- The ``sm`` BTL is a low-latency, high-bandwidth mechanism for transferring data between two processes via shared memory. This BTL @@ -26,8 +24,8 @@ can only be used between processes executing on the same node. ///////////////////////////////////////////////////////////////////////// -How do I specify use of sm for MPI messages? --------------------------------------------- +Specifying the Use of sm for MPI Messages +----------------------------------------- Typically, it is unnecessary to do so; OMPI will use the best BTL available for each communication. @@ -44,8 +42,8 @@ communications. For example: ///////////////////////////////////////////////////////////////////////// -How can I tune these parameters to improve performance? -------------------------------------------------------- +Tuning Parameters to Improve Performance +---------------------------------------- Mostly, the default values of the MCA parameters have already been chosen to give good performance. To improve performance further @@ -71,21 +69,106 @@ performance for memory. to resource congestion, but you can increase this parameter to pre-reserve space for more fragments. +* ``btl_sm_backing_directory``: Directory to place backing files for + shared memory communication. This directory should be on a local + filesystem such as ``/tmp`` or ``/dev/shm`` (default: (linux) ``/dev/shm``, + (others) session directory) + ///////////////////////////////////////////////////////////////////////// -Where is the shared memory mapped on the filesystem? +Shared Memory Mechanisms +------------------------ + +The ``sm`` BTL supports two modes of shared memory communication: + +#. **Two-copy:** Otherwise known as "copy-in / copy-out", this mode is + where the sender copies data into shared memory and the receiver + copies the data out. + + This mechanism is always available. + +#. **Single copy:** In this mode, the sender or receiver makes a + single copy of the message data from the source buffer in one + process to the destination buffer in another process. Open MPI + supports three flavors of shared memory single-copy transfers: + + * `Linux KNEM `_. This is a + standalone Linux kernel module, made specifically for HPC and MPI + libraries to enable high-performance single-copy message + transfers. + + Open MPI must be able to find the KNEM header files in order to + build support for KNEM. + + * `Linux XPMEM `_. Similar to + KNEM, this is a standalone Linux kernel module, made specifically + for HPC and MPI libraries to enable high-performance single-copy + message transfers. It is derived from the Cray XPMEM system. + + Open MPI must be able to find the XPMEM header files in order to + build support for XPMEM. + + * Linux Cross-Memory Attach (CMA). This mechanism is built-in to + modern versions of the Linux kernel. Although more performance + than the two-copy shared memory transfer mechanism, CMA is the + lowest performance of the single-copy mechanisms. However, CMA + is likely the most widely available because it is enabled by + default in several modern Linux distributions. + + Open MPI must be built on a Linux system with a recent enough + Glibc and kernel version in order to build support for Linux CMA. + +Which mechanism is used at run time depends both on how Open MPI was +built and how your system is configured. You can check to see which +single-copy mechanisms Open MPI was built with via two mechanisms: + +#. At the end of running ``configure``, Open MPI emits a list of + transports for which it found relevant header files and libraries + such that it will be able to build support for them. You might see + lines like this, for example: + + .. code-block:: text + + Shared memory/copy in+copy out: yes + Shared memory/Linux CMA: yes + Shared memory/Linux KNEM: no + Shared memory/XPMEM: no + + The above output indicates that Open MPI will be built with 2-copy + (as mentioned above, 2-copy is *always* available) and with Linux + CMA support. KNEM and XPMEM support will *not* be built. + +#. After Open MPI is installed, the ``ompi_info`` command can show + which ``smsc`` (shared memory single copy) components are + available: + + .. code-block:: text + + shell$ ompi_info | grep smsc + MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.1.0) + + This Open MPI installation only supports the Linux CMA single-copy + mechanism. + +.. note:: As implied by the SMSC component names, none of them are + supported on macOS. macOS users will use the two-copy mechanism. + +///////////////////////////////////////////////////////////////////////// -.. error:: TODO Is this correct? +Shared Memory Mapping on the Filesystem +--------------------------------------- -The file will be in the OMPI session directory, which is typically -something like ``/tmp/openmpi-sessions-USERNAME@HOSTNAME/*``. -The file itself will have the name -``shared_mem_pool.HOSTNAME``. For example, the full path could be -``/tmp/openmpi-sessions-johndoe@node0_0/1543/1/shared_mem_pool.node0``. +The default location of the file is in the ``/dev/shm`` directory. If ``/dev/shm`` +does not exist on the system, the default location will be the OMPI session +directory. The path is typically something like: +``/dev/shm/sm_segment.nodename.user_id.job_id.my_node_rank``. +For example, the full path could be: ``/dev/shm/sm_segment.x.1000.23c70000.0``. -.. error:: TODO The filename above will certainly be wrong. +You can use the MCA parameter ``btl_sm_backing_directory`` to place the +directory in a non-default location. -To place the session directory in a non-default location, use the MCA parameter -``orte_tmpdir_base``. +.. note:: The session directory can be customized via + PRRTE using ``--prtemca prte_tmpdir_base /path/to/somewhere``. -.. error:: TODO The MCA param name above is definitely wrong. +.. note:: Even when using single-copy methods like CMA, a shared memory file is still + created for managing connection metadata.