diff --git a/docs/development.md b/docs/development.md index 2a0c97581ec7..1ae56d042cb4 100644 --- a/docs/development.md +++ b/docs/development.md @@ -93,103 +93,73 @@ sudo apt install clang ccache lld 1. Set up Developer PowerShell [for Visual Studio](https://learn.microsoft.com/en-us/visualstudio/ide/reference/command-prompt-powershell?view=vs-2022#start-in-visual-studio) 1. Ensure that the compiler and linker binaries are in the `PATH` variable. -#### Configure for Building... +#### Configure for Building + +1. [Activate the Python environment](#set-up-the-python-environment) +1. Choose command with relevant LLVM options: + 1. **If building "in-tree"**, run/append: + + ```shell + cmake -GNinja -Bbuild \ + `# Enables "--debug" and "--debug-only" flags for the "torch-mlir-opt" tool` \ + -DCMAKE_BUILD_TYPE=RelWithDebInfo \ + -DLLVM_ENABLE_ASSERTIONS=ON \ + -DPython3_FIND_VIRTUALENV=ONLY \ + -DMLIR_ENABLE_BINDINGS_PYTHON=ON \ + -DLLVM_TARGETS_TO_BUILD=host \ + \ + `# For building LLVM "in-tree"` \ + externals/llvm-project/llvm \ + -DLLVM_ENABLE_PROJECTS=mlir \ + -DLLVM_EXTERNAL_PROJECTS="torch-mlir" \ + -DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$PWD" + ``` + + - NOTE: uses external/llvm-project/llvm as the main build, so LLVM will be built in additional to torch-mlir and its sub-projects. + 1. **If using "out-of-tree" build**, run/append: + + ```shell + cmake -GNinja -Bbuild \ + `# Enables "--debug" and "--debug-only" flags for the "torch-mlir-opt" tool` \ + -DCMAKE_BUILD_TYPE=RelWithDebInfo \ + -DLLVM_ENABLE_ASSERTIONS=ON \ + -DPython3_FIND_VIRTUALENV=ONLY \ + -DMLIR_ENABLE_BINDINGS_PYTHON=ON \ + -DLLVM_TARGETS_TO_BUILD=host \ + \ + `# For building LLVM "out-of-tree"` \ + -DMLIR_DIR="$LLVM_INSTALL_DIR/lib/cmake/mlir/" \ + -DLLVM_DIR="$LLVM_INSTALL_DIR/lib/cmake/llvm/" + ``` + + - Be sure to have built LLVM with `-DLLVM_ENABLE_PROJECTS=mlir`. + - Be aware that the installed version of LLVM needs in general to match the committed version in `externals/llvm-project`. Using a different version may or may not work. + + - [About MLIR debugging](https://mlir.llvm.org/getting_started/Debugging/) +1. **If you anticipate needing to frequently rebuild LLVM**, append: -Two setups are possible to build: in-tree and out-of-tree. The in-tree setup is the most straightforward, as it will build LLVM dependencies as well. - -##### ...with LLVM "in-tree" using... - -The following commands generate configuration files to build the project *in-tree*, that is, using llvm/llvm-project as the main build. This will build LLVM as well as torch-mlir and its subprojects. - -###### ...Base + Optimization Options - -If you do anticipate needing to frequently rebuild LLVM "in-tree", run: - -```shell -cmake -GNinja -Bbuild \ - `# Enables "--debug" and "--debug-only" flags for the "torch-mlir-opt" tool` \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DLLVM_ENABLE_ASSERTIONS=ON \ - -DPython3_FIND_VIRTUALENV=ONLY \ - -DMLIR_ENABLE_BINDINGS_PYTHON=ON \ - -DLLVM_TARGETS_TO_BUILD=host \ - `# For building LLVM "in-tree"` \ - externals/llvm-project/llvm \ - -DLLVM_ENABLE_PROJECTS=mlir \ - -DLLVM_EXTERNAL_PROJECTS="torch-mlir" \ - -DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$PWD" \ - `# use clang`\ - -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \ - `# use ccache to cache build results` \ - -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ - `# use LLD to link in seconds, rather than minutes` \ - `# if using clang <= 13, replace --ld-path=ld.lld with -fuse-ld=lld` \ - -DCMAKE_EXE_LINKER_FLAGS_INIT="--ld-path=ld.lld" \ - -DCMAKE_MODULE_LINKER_FLAGS_INIT="--ld-path=ld.lld" \ - -DCMAKE_SHARED_LINKER_FLAGS_INIT="--ld-path=ld.lld" \ - `# Enabling libtorch binary cache instead of downloading the latest libtorch everytime.` \ - `# Testing against a mismatched version of libtorch may cause failures` \ - -DLIBTORCH_CACHE=ON \ - `# Enable an experimental path to build libtorch (and PyTorch wheels) from source,` \ - `# instead of downloading them` \ - -DLIBTORCH_SRC_BUILD=ON \ - `# Set the variant of libtorch to build / link against. (shared|static and optionally cxxabi11)` \ - -DLIBTORCH_VARIANT=shared -``` - -- This will build `libtorch` / `PyTorch` wheels from source and requires [the enablement mentioned earlier](#optional-enable-quicker-builds). -- If you encounter issues when you run this, try the [simplified build command](#base-options) instead. - -###### ...Base Options - -If you don't anticipate needing to frequently rebuild LLVM "in-tree", run: - -```shell -cmake -GNinja -Bbuild \ - `# Enables "--debug" and "--debug-only" flags for the "torch-mlir-opt" tool` \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DLLVM_ENABLE_ASSERTIONS=ON \ - -DPython3_FIND_VIRTUALENV=ONLY \ - -DMLIR_ENABLE_BINDINGS_PYTHON=ON \ - -DLLVM_TARGETS_TO_BUILD=host \ - `# For building LLVM "in-tree"` \ - externals/llvm-project/llvm \ - -DLLVM_ENABLE_PROJECTS=mlir \ - -DLLVM_EXTERNAL_PROJECTS="torch-mlir" \ - -DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$PWD" -``` - - -##### ...with LLVM "out-of-tree" - -If you have built llvm-project separately in the directory `$LLVM_INSTALL_DIR`, you can also build the project *out-of-tree* using the following command as template: -```shell -cmake -GNinja -Bbuild \ - `# Enables "--debug" and "--debug-only" flags for the "torch-mlir-opt" tool` \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DLLVM_ENABLE_ASSERTIONS=ON \ - -DPython3_FIND_VIRTUALENV=ONLY \ - -DMLIR_ENABLE_BINDINGS_PYTHON=ON \ - -DLLVM_TARGETS_TO_BUILD=host \ - `# For building LLVM "out-of-tree"` \ - -DMLIR_DIR="$LLVM_INSTALL_DIR/lib/cmake/mlir/" \ - -DLLVM_DIR="$LLVM_INSTALL_DIR/lib/cmake/llvm/" - . -``` -The same QoL CMake flags can be used to enable clang, ccache, and lld. Be sure to have built LLVM with `-DLLVM_ENABLE_PROJECTS=mlir`. - -Be aware that the installed version of LLVM needs in general to match the committed version in `externals/llvm-project`. Using a different version may or may not work. + ```shell + \ + `# use clang`\ + -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \ + `# use ccache to cache build results` \ + -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ + `# use LLD to link in seconds, rather than minutes` \ + -DCMAKE_LINKER_TYPE=lld + ``` -###### [About MLIR debugging](https://mlir.llvm.org/getting_started/Debugging/) + - This requires [the enablement mentioned earlier](#optional-enable-quicker-builds). + - If these options cause issues, just skip them for now. +1. **If you need to enable local end-to-end tests**, append: -##### Options to run end-to-end tests + ```shell + \ + -DTORCH_MLIR_ENABLE_PYTORCH_EXTENSIONS=ON \ + -DTORCH_MLIR_ENABLE_JIT_IR_IMPORTER=ON + ``` -Running the end-to-end execution tests locally requires enabling the native PyTorch extension features and the JIT IR importer, which depends on the -former and defaults to `ON` if not changed: -```shell - -DTORCH_MLIR_ENABLE_PYTORCH_EXTENSIONS=ON \ - -DTORCH_MLIR_ENABLE_JIT_IR_IMPORTER=ON \ -``` + - NOTE: The JIT IR importer depends on the native PyTorch extension features and defaults to `ON` if not changed. +1. Run assembled command (once you've appended the options pertaining to your workflow) #### Initiate Build