Skip to content

Commit 0861121

Browse files
xbw22109xbw22109
committed
Updated the documentation about shared memory.
 Changes to be committed: modified: docs/launching-apps/localhost.rst modified: docs/tuning-apps/networking/shared-memory.rst Signed-off-by: xbw <[email protected]>
1 parent 1d301f7 commit 0861121

File tree

2 files changed

+103
-97
lines changed

2 files changed

+103
-97
lines changed

‎docs/launching-apps/localhost.rst

-78
Original file line numberDiff line numberDiff line change
@@ -35,81 +35,3 @@ peers:
3535
and destination MPI processes can share memory (e.g., via SYSV or
3636
POSIX shared memory mechanisms).
3737

38-
Shared memory MPI communication
39-
-------------------------------
40-
41-
.. error:: TODO This should really be moved to the networking section.
42-
43-
The ``sm`` BTL supports two modes of shared memory communication:
44-
45-
#. **Two-copy:** Otherwise known as "copy-in / copy-out", this mode is
46-
where the sender copies data into shared memory and the receiver
47-
copies the data out.
48-
49-
This mechanism is always available.
50-
51-
#. **Single copy:** In this mode, the sender or receiver makes a
52-
single copy of the message data from the source buffer in one
53-
process to the destination buffer in another process. Open MPI
54-
supports three flavors of shared memory single-copy transfers:
55-
56-
* `Linux KNEM <https://knem.gitlabpages.inria.fr/>`_. This is a
57-
standalone Linux kernel module, made specifically for HPC and MPI
58-
libraries to enable high-performance single-copy message
59-
transfers.
60-
61-
Open MPI must be able to find the KNEM header files in order to
62-
build support for KNEM.
63-
64-
* `Linux XPMEM <https://github.com/hjelmn/xpmem>`_. Similar to
65-
KNEM, this is a standalone Linux kernel module, made specifically
66-
for HPC and MPI libraries to enable high-performance single-copy
67-
message transfers. It is derived from the Cray XPMEM system.
68-
69-
Open MPI must be able to find the XPMEM header files in order to
70-
build support for XPMEM.
71-
72-
* Linux Cross-Memory Attach (CMA). This mechanism is built-in to
73-
modern versions of the Linux kernel. Although more performance
74-
than the two-copy shared memory transfer mechanism, CMA is the
75-
lowest performance of the single-copy mechanisms. However, CMA
76-
is likely the most widely available because it is enabled by
77-
default in several modern Linux distributions.
78-
79-
Open MPI must be built on a Linux system with a recent enough
80-
Glibc and kernel version in order to build support for Linux CMA.
81-
82-
Which mechanism is used at run time depends both on how Open MPI was
83-
built and how your system is configured. You can check to see which
84-
single-copy mechanisms Open MPI was built with via two mechanisms:
85-
86-
#. At the end of running ``configure``, Open MPI emits a list of
87-
transports for which it found relevant header files and libraries
88-
such that it will be able to build support for them. You might see
89-
lines like this, for example:
90-
91-
.. code-block:: text
92-
93-
Shared memory/copy in+copy out: yes
94-
Shared memory/Linux CMA: yes
95-
Shared memory/Linux KNEM: no
96-
Shared memory/XPMEM: no
97-
98-
The above output indicates that Open MPI will be built with 2-copy
99-
(as mentioned above, 2-copy is *always* available) and with Linux
100-
CMA support. KNEM and XPMEM support will *not* be built.
101-
102-
#. After Open MPI is installed, the ``ompi_info`` command can show
103-
which ``smsc`` (shared memory single copy) components are
104-
available:
105-
106-
.. code-block:: text
107-
108-
shell$ ompi_info | grep smsc
109-
MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.1.0)
110-
111-
This Open MPI installation only supports the Linux CMA single-copy
112-
mechanism.
113-
114-
.. note:: As implied by the SMSC component names, none of them are
115-
supported on macOS. macOS users will use the two-copy mechanism.

‎docs/tuning-apps/networking/shared-memory.rst

+103-19
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,9 @@
11
Shared Memory
22
=============
33

4-
.. error:: TODO This section needs to be converted from FAQ Q&A style
5-
to regular documentation style.
64

7-
What is the sm BTL?
8-
-------------------
5+
The sm BTL
6+
----------
97

108
The ``sm`` BTL is a low-latency, high-bandwidth mechanism for
119
transferring data between two processes via shared memory. This BTL
@@ -26,8 +24,8 @@ can only be used between processes executing on the same node.
2624

2725
/////////////////////////////////////////////////////////////////////////
2826

29-
How do I specify use of sm for MPI messages?
30-
--------------------------------------------
27+
Specifying the Use of sm for MPI Messages
28+
-----------------------------------------
3129

3230
Typically, it is unnecessary to do so; OMPI will use the best BTL available
3331
for each communication.
@@ -44,8 +42,8 @@ communications. For example:
4442
4543
/////////////////////////////////////////////////////////////////////////
4644

47-
How can I tune these parameters to improve performance?
48-
-------------------------------------------------------
45+
Tuning Parameters to Improve Performance
46+
----------------------------------------
4947

5048
Mostly, the default values of the MCA parameters have already
5149
been chosen to give good performance. To improve performance further
@@ -71,21 +69,107 @@ performance for memory.
7169
to resource congestion, but you can increase this parameter to
7270
pre-reserve space for more fragments.
7371

72+
* ``btl_sm_backing_directory``: Directory to place backing files for
73+
shared memory communication. This directory should be on a local
74+
filesystem such as /tmp or /dev/shm (default: (linux) /dev/shm,
75+
(others) session directory)
76+
7477
/////////////////////////////////////////////////////////////////////////
7578

76-
Where is the shared memory mapped on the filesystem?
79+
Shared Memory Mechanisms
80+
------------------------
81+
82+
The ``sm`` BTL supports two modes of shared memory communication:
83+
84+
#. **Two-copy:** Otherwise known as "copy-in / copy-out", this mode is
85+
where the sender copies data into shared memory and the receiver
86+
copies the data out.
87+
88+
This mechanism is always available.
89+
90+
#. **Single copy:** In this mode, the sender or receiver makes a
91+
single copy of the message data from the source buffer in one
92+
process to the destination buffer in another process. Open MPI
93+
supports three flavors of shared memory single-copy transfers:
94+
95+
* `Linux KNEM <https://knem.gitlabpages.inria.fr/>`_. This is a
96+
standalone Linux kernel module, made specifically for HPC and MPI
97+
libraries to enable high-performance single-copy message
98+
transfers.
99+
100+
Open MPI must be able to find the KNEM header files in order to
101+
build support for KNEM.
102+
103+
* `Linux XPMEM <https://github.com/hjelmn/xpmem>`_. Similar to
104+
KNEM, this is a standalone Linux kernel module, made specifically
105+
for HPC and MPI libraries to enable high-performance single-copy
106+
message transfers. It is derived from the Cray XPMEM system.
107+
108+
Open MPI must be able to find the XPMEM header files in order to
109+
build support for XPMEM.
110+
111+
* Linux Cross-Memory Attach (CMA). This mechanism is built-in to
112+
modern versions of the Linux kernel. Although more performance
113+
than the two-copy shared memory transfer mechanism, CMA is the
114+
lowest performance of the single-copy mechanisms. However, CMA
115+
is likely the most widely available because it is enabled by
116+
default in several modern Linux distributions.
117+
118+
Open MPI must be built on a Linux system with a recent enough
119+
Glibc and kernel version in order to build support for Linux CMA.
120+
121+
Which mechanism is used at run time depends both on how Open MPI was
122+
built and how your system is configured. You can check to see which
123+
single-copy mechanisms Open MPI was built with via two mechanisms:
124+
125+
#. At the end of running ``configure``, Open MPI emits a list of
126+
transports for which it found relevant header files and libraries
127+
such that it will be able to build support for them. You might see
128+
lines like this, for example:
129+
130+
.. code-block:: text
131+
132+
Shared memory/copy in+copy out: yes
133+
Shared memory/Linux CMA: yes
134+
Shared memory/Linux KNEM: no
135+
Shared memory/XPMEM: no
136+
137+
The above output indicates that Open MPI will be built with 2-copy
138+
(as mentioned above, 2-copy is *always* available) and with Linux
139+
CMA support. KNEM and XPMEM support will *not* be built.
140+
141+
#. After Open MPI is installed, the ``ompi_info`` command can show
142+
which ``smsc`` (shared memory single copy) components are
143+
available:
144+
145+
.. code-block:: text
146+
147+
shell$ ompi_info | grep smsc
148+
MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.1.0)
149+
150+
This Open MPI installation only supports the Linux CMA single-copy
151+
mechanism.
152+
153+
.. note:: As implied by the SMSC component names, none of them are
154+
supported on macOS. macOS users will use the two-copy mechanism.
155+
156+
/////////////////////////////////////////////////////////////////////////
77157

78-
.. error:: TODO Is this correct?
158+
Shared Memory Mapping on the Filesystem
159+
---------------------------------------
79160

80-
The file will be in the OMPI session directory, which is typically
81-
something like ``/tmp/openmpi-sessions-USERNAME@HOSTNAME/*``.
82-
The file itself will have the name
83-
``shared_mem_pool.HOSTNAME``. For example, the full path could be
84-
``/tmp/openmpi-sessions-johndoe@node0_0/1543/1/shared_mem_pool.node0``.
161+
The default location of the file is in the ``/dev/shm directory``. If ``/dev/shm``
162+
does not exist on the system, the default location will be the OMPI session
163+
directory. The path is typically something like:
164+
``/dev/shm/sm_segment.nodename.user_id.job_id.my_node_rank``.
165+
For example, the full path could be: ``/dev/shm/sm_segment.x.1000.23c70000.0``.
85166

86-
.. error:: TODO The filename above will certainly be wrong.
167+
You can use the MCA parameter ``btl_sm_backing_directory`` to place the session
168+
directory in a non-default location.
87169

88-
To place the session directory in a non-default location, use the MCA parameter
89-
``orte_tmpdir_base``.
170+
.. note:: The session directory is defined in PMIx. You can
171+
use ``--pmixmca orte_tmpdir_base "/path/to/somewhere"`` to place the session
172+
directory in a non-default location.
90173

91-
.. error:: TODO The MCA param name above is definitely wrong.
174+
.. note:: Even when using single-copy methods like CMA, a shared memory file is still
175+
created for managing connection metadata.

0 commit comments

Comments
 (0)