Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation on the "Building MFEM" page for CMake install #291

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 96 additions & 0 deletions src/building.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,3 +239,99 @@ git clone https://github.com/spack/spack.git
cd spack
./bin/spack install -v mfem
```

## Building MFEM with CMake
To build a serial version of MFEM with CMake first create a build directory. For example, using a build directory named `build`:
```sh
mkdir build
cd build
```

Run the CMake configuration on the MFEM source directory.
```sh
cmake ..
```

Run the build command associated with the CMake configuration, specifying the number of parallel build tasks with the `-j` flag (4 tasks in this case).
```sh
cmake --build . -j 4
```
### Parallel build using CMake
To build a parallel version of MFEM with CMake first build METIS and Hypre as described above.
From the MFEM source directory, create a build directory. For example, using a build directory named `build`:
```sh
cd mfem-4.5
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't add versions to general instructions

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was something I had a question about while writing this, the Makefile section of these instructions assume mfem-4.5, should I stick to that convention or use "MFEM_DIR", and should the Makefile instructions do the same?

mkdir build
cd build
```

Run the CMake configuration on the MFEM source directory using the `MFEM_USE_MPI` CMake variable to enable MPI.
This will automatically search for the system MPI implementation, the METIS installation (in `<mfem-source-dir>/../metis-4.0`), and Hypre installation (in `<mfem-source-dir/../hypre`).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

```sh
cmake .. -DMFEM_USE_MPI=YES
```
Run the build command associated with the cmake configuration, specifying the number of parallel build tasks with the `-j` flag (4 tasks in this case).
```sh
cmake --build . -j 4
```

### Advanced configuration steps
To build with CUDA:
```sh
cmake .. -DMFEM_USE_CUDA=YES
```
To specify what CUDA architecture to target:
```sh
cmake .. -DCUDA_ARCH="sm_70"
```
The CUDA architecture is formatted as `sm_{CC}`, or just `{CC}`, where CC is the GPU compute capability of the target GPU without the decimal point. A list of Nvidia GPU compute capabilities can be found in [the Nividia developers documentation](https://developer.nvidia.com/cuda-gpus). Multiple CUDA architectures can be targeted with a comma or semicolon separated list.
```sh
cmake .. -DCUDA_ARCH="{ARCH1},{ARCH2},{ARCH3}"
```
or
```sh
cmake .. -DCUDA_ARCH="{ARCH1};{ARCH2};{ARCH3}"
```
Other accepted architecture identifies are `"all"` which targets all CUDA architectures,
`"all-major"` which targets all major versions `sm_{*0}`, and `"native"` which targets the visible GPUs on the system.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To specify what CUDA architecture to target:
```sh
cmake .. -DCUDA_ARCH="sm_70"
```
The CUDA architecture is formatted as `sm_{CC}`, where CC is the GPU compute capability of the target GPU without the decimal point. A list of Nvidia GPU compute capabilities can be found in [the Nividia developers documentation](https://developer.nvidia.com/cuda-gpus).

TODO: currently I think MFEM's CMake setup doesn't allow targeting multiple CUDA architectures at once. I think we should fix this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a CMake 3.18 target property CUDA_ARCHITECTURES that allows semicolon separated lists https://cmake.org/cmake/help/latest/prop_tgt/CUDA_ARCHITECTURES.html

Right now it looks like mfem manually sets the -arch flag. From the nvcc compiler documentation it seems like it might take comma separated lists if the format is the same as the --gpu-code flag. I don't have access to cuda enabled machines to try this out right now.

The all, all-major, and native options for the architecture may be worth mentioning.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I expanded support for all, all-major, and native as well as multiple specific cuda architectures in this PR: pr4561.

Supported formats now include:
-DCUDA_ARCH="all"
-DCUDA_ARCH="all-major"
-DCUDA_ARCH="native"
-DCUDA_ARCH="{ARCH1},{ARCH2},..."
-DCUDA_ARCH="{ARCH1};{ARCH2};..."
where ARCHN can be either just the CC number (70, 86, etc.), or optionally prefixed with sm_ (sm_70, sm_86, etc.).

This should work even for CMake older than 3.18 (the current CMakeLists minimum version is 3.8).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great, added some documentation for this!

To build with METIS 5, after following the instructions to build METIS 5 above:
```sh
cmake .. -DMFEM_USE_MPI=YES -DMFEM_USE_METIS_5=YES -DMETIS_DIR=../../metis-5.1.0
```
To build with HIP:
```sh
cmake .. -DMFEM_USE_HIP=YES
```

gabsillis marked this conversation as resolved.
Show resolved Hide resolved
To specify what HIP architecture(s) to target:
```sh
cmake .. -DHIP_ARCH="gfx942;gfx90a"
```
Multiple architectures can be targeted using a semi-colon separated list.
The HIP architecture for different GPU models can be found in [the LLVM documentation](https://llvm.org/docs/AMDGPUUsage.html#processors)
### Alternate build steps
Different targets can be built with the --target flag in the build step
```sh
cmake --build . -j 4 --target <target-name>
```
To build the examples use the `examples` target (the executables will be in the `build/examples` directory).
```sh
cmake --build . -j 4 --target examples
```

To quickly check if the code is succesfully built using example 1/1p use the `check` target.
```sh
cmake --build . -j 4 --target check
```

To build the miniapps use the `miniapps` target
```sh
cmake --build . -j 4 --target miniapps
```

To build everything use the `exec` target
```sh
cmake --build . -j 4 --target exec
```