add guidelines for creating a test suite in WAMR

Author: lum1n0us

tests/unit/README.md

@@ -0,0 +1,194 @@
+# Guide to Creating a Test Suite for a New Feature in WAMR
+
+This guide provides instructions for contributors on how to create a test suite for a new feature in the WAMR project. Follow these steps to ensure consistency and maintainability across the test framework.
+
+---
+
+## General Guidelines
+
+- **Create a New Directory**:
+  Always create a dedicated directory for a new feature under the `tests/unit/` directory.
+
+  - Reuse existing test cases and patch them when possible to avoid redundancy.
+  - Name the directory in lowercase with words separated by hyphens (e.g., `new-feature`).
+
+- **Avoid Committing `.wasm` Files**:
+  Do not commit precompiled `.wasm` files. Instead:
+
+  - Generate `.wasm` files from `.wat` or `.c` source files.
+  - Use `ExternalProject` and the `wasi-sdk` toolchain to compile `.wasm` files during the build process.
+
+- **Keep Using `ctest` as the framework**:
+  Continue to use `ctest` for running the test cases, as it is already integrated into the existing test framework.
+
+---
+
+## Writing `CMakeLists.txt` for the Test Suite
+
+When creating a `CMakeLists.txt` file for your test suite, follow these best practices:
+
+1. **Do Not Fetch Googletest Again**:
+   The root `unit/CMakeLists.txt` already fetches Googletest. Avoid including or fetching it again in your test suite.
+
+2. **Find LLVM on Demand**:
+   If your test suite requires LLVM, use `find_package` to locate LLVM components as needed. Do not include LLVM globally unless required.
+
+3. **Include `unit_common.cmake`**:
+   Always include `../unit_common.cmake` in your `CMakeLists.txt` to avoid duplicating common configurations and utilities.
+
+   Example:
+
+   ```cmake
+   include("../unit_common.cmake")
+   ```
+
+4. **Use `WAMR_RUNTIME_LIB_SOURCE`**:
+   Replace long lists of runtime source files with the `WAMR_RUNTIME_LIB_SOURCE` variable to simplify your configuration.
+
+   Example:
+
+   ```cmake
+   target_sources(your_test_target PRIVATE ${WAMR_RUNTIME_LIB_SOURCE})
+   ```
+
+5. **Avoid Global Compilation Flags**:
+   Do not define global compilation flags in the `unit` directory. Each test case should specify its own compilation flags based on its unique requirements.
+
+---
+
+## Generating `.wasm` Files
+
+- **Compile `.wasm` Files Dynamically**:
+  Use `ExternalProject` in your `CMakeLists.txt` to compile `.wasm` files from `.wat` or `.c` source files.
+  - Use the `wasi-sdk` toolchain for `.c` or `.cc` source files.
+  - Example configuration:
+    ```cmake
+    ExternalProject_Add(
+        generate_wasm
+        SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/wasm-apps
+        BUILD_ALWAYS YES
+        CONFIGURE_COMMAND  ${CMAKE_COMMAND} -S ${CMAKE_CURRENT_SOURCE_DIR}/wasm-apps -B build
+                              -DWASI_SDK_PREFIX=${WASI_SDK_DIR}
+                              -DCMAKE_TOOLCHAIN_FILE=${WASISDK_TOOLCHAIN}
+        BUILD_COMMAND      ${CMAKE_COMMAND} --build build
+        INSTALL_COMMAND    ${CMAKE_COMMAND} --install build --prefix ${CMAKE_CURRENT_BINARY_DIR}/wasm-apps
+    )
+    ```
+- **Example for `wasm-apps` Directory**:
+  Place your source files in a `wasm-apps/` subdirectory within your test suite directory.
+
+  - Create a `CMakeLists.txt` in `wasm-apps/` to handle the compilation of these files.
+  - Example `CMakeLists.txt` for `wasm-apps/`:
+
+    ```cmake
+    cmake_minimum_required(VERSION 3.13)
+    project(wasm_apps)
+
+    add_executable(example example.c)
+    set_target_properties(example PROPERTIES SUFFIX .wasm)
+    install(TARGETS example DESTINATION .)
+    ```
+
+---
+
+## Compiling and Running Test Cases
+
+To compile and run the test cases, follow these steps:
+
+1. **Generate Build Files**:
+
+   ```bash
+   cmake -S . -B build
+   ```
+
+2. **Build the Test Suite**:
+
+   ```bash
+   cmake --build build
+   ```
+
+3. **Run the Tests**:
+
+   ```bash
+   ctest --test-dir build --output-on-failure
+   ```
+
+   This will compile and execute all test cases in the test suite, displaying detailed output for any failures.
+
+4. **List all Tests**:
+   To see all available test cases, use:
+
+   ```bash
+   ctest --test-dir build -N
+   ```
+
+5. **Run a Specific Test**:
+   To run a specific test case, use:
+   ```bash
+   ctest --test-dir build -R <test_name> --output-on-failure
+   ```
+
+---
+
+## Collecting Code Coverage Data
+
+To collect code coverage data using `lcov`, follow these steps:
+
+1. **Build with Coverage Flags**:
+   Ensure the test suite is built with coverage flags enabled:
+
+   ```bash
+   cmake -S . -B build -DCOLLECT_CODE_COVERAGE=1
+   cmake --build build
+   ```
+
+2. **Run the Tests**:
+   Execute the test cases as described above.
+
+3. **Generate Coverage Report**:
+   Use `lcov` to collect and generate the coverage report:
+
+   ```bash
+   lcov --capture --directory build --output-file coverage.all.info
+   lcov --extract coverage.all.info "*/core/iwasm/*" "*/core/shared/*" --output-file coverage.info
+   genhtml coverage.info --output-directory coverage-report
+   ```
+
+4. **View the Report**:
+   Open the `index.html` file in the `coverage-report` directory to view the coverage results in your browser.
+
+5. **Summary of Coverage**:
+   To get a summary of the coverage data, use:
+
+   ```bash
+   lcov --summary coverage.info
+   ```
+
+---
+
+## Example Directory Structure
+
+Here’s an example of how your test suite directory might look:
+
+```
+new-feature/
+├── CMakeLists.txt
+├── new_feature_test.cc
+├── wasm-apps/
+|   ├── CMakeLists.txt
+│   ├── example.c
+│   └── example.wat
+```
+
+---
+
+## Additional Notes
+
+- **Testing Framework**: Use Googletest for writing unit tests. Refer to existing test cases in the `tests/unit/` directory for examples.
+- **Documentation**: Add comments in your test code to explain the purpose of each test case.
+- **Edge Cases**: Ensure your test suite covers edge cases and potential failure scenarios.
+- **Reuse Utilities**: Leverage existing utilities in `common/` (e.g., `mock_allocator.h`, `test_helper.h`) to simplify your test code.
+
+---
+
+By following these guidelines, you can create a well-structured and maintainable test suite that integrates seamlessly with the WAMR testing framework.