Migrate documentation to the new documentation system

.github/workflows/5_builderpackage_plugins_onpush.yml

@@ -4,6 +4,8 @@ name: Build packages (on push)
 # - On push
 on:
   push:
+    paths:
+      - '**.java'
 
 jobs:
   call-build-workflow:

CHANGELOG.md

@@ -7,19 +7,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ## [Unreleased 5.0.x]
 
 ### Added
+- Add technical documentation [(#301)](https://github.com/wazuh/wazuh-indexer-plugins/pull/301)
 
 ### Dependencies
 
 ### Changed
-- Third-party integrations maintenance
-- Upgraded opensearch version to 2.19.1
+- Third-party integrations maintenance [(#299)](https://github.com/wazuh/wazuh-indexer-plugins/pull/299)
+- Upgrade to Opensearch 2.19.1 [(#304)](https://github.com/wazuh/wazuh-indexer-plugins/pull/304)
 
 ### Deprecated
 
 ### Removed
 
 ### Fixed
- -  Add a validation to the order of the action arguments to force action.name to come before action.arg so that they can always be validated.  (#260).
+- Add a validation to the order of the action arguments to force action.name to come before action.arg so that they can always be validated [(#260)](https://github.com/wazuh/wazuh-indexer-plugins/issues/260)
 
 ### Security
 

docs/README.md

@@ -19,6 +19,6 @@ To work with this documentation, you need **mdBook** installed. For installation
 
 - To serve the documentation locally for preview, run:
   ```bash
-  ./serve.sh
+  ./server.sh
   ```
-  The documentation will be available at [http://127.0.0.1:3000](http://127.0.0.1:3000).
+  The documentation will be available at [http://127.0.0.1:3000](http://127.0.0.1:3000).

docs/SUMMARY.md

@@ -14,32 +14,32 @@
 
 # Reference Manual
 
-- [Introduction](ref/README.md)
+- [Introduction]()
   - [Description](ref/description.md)
   - [Architecture](ref/architecture.md)
   - [Compatibility](ref/compatibility.md)
-- [Getting Started](ref/getting-started/README.md)
+- [Getting Started]()
   - [Requirements](ref/getting-started/requirements.md)
   - [Packages](ref/getting-started/packages.md)
   - [Installation](ref/getting-started/installation.md)
-- [Configuration](ref/configuration.md)
-  - [Environment variables](ref/getting-started/variables.md)
-  - [Command line options](ref/getting-started/cli-options.md)
+- [Configuration]()
+  - [Environment variables]()
+  - [Command line options]()
   - [Configuration files](ref/getting-started/config-files.md)
-- [Modules](ref/modules/README.md)
+- [Modules]()
   - [Setup](ref/modules/setup/README.md)
     - [Architecture](ref/modules/setup/architecture.md)
-    - [API Reference](ref/modules/setup/api-reference.md)
+    - [API Reference]()
   - [Command Manager](ref/modules/command-manager/README.md)
     - [Architecture](ref/modules/command-manager/architecture.md)
     - [API Reference](ref/modules/command-manager/api-reference.md)
   - [Content Manager](ref/modules/content-manager/README.md)
     - [Architecture](ref/modules/content-manager/architecture.md)
-    - [API Reference](ref/modules/content-manager/api-reference.md)
+    - [API Reference]()
 - [Upgrade](ref/upgrade.md)
 - [Uninstall](ref/uninstall.md)
 - [Back Up and Restore](ref/backup-restore.md)
-- [Security](ref/security.md)
-- [Performance](ref/performance.md)
-- [Glossary](ref/glossary.md)
+- [Security]()
+- [Performance]()
+- [Glossary]()
 

docs/book.toml

@@ -5,3 +5,13 @@ description = "The technical documentation for the Wazuh Indexer, composed by th
 language = "en"
 multilingual = false
 src = "."
+
+[preprocessor]
+
+[preprocessor.mermaid]
+command = "mdbook-mermaid"
+
+[output]
+
+[output.html]
+additional-js = ["mermaid.min.js", "mermaid-init.js"]

docs/dev/README.md

@@ -1 +1,5 @@
 # Development documentation
+
+Under this section, you will find the development documentation of Wazuh Indexer. This documentation contains instructions to compile, run, test and package the source code. Moreover, you will find instructions to set up a development environment in order to get started at developing the Wazuh Indexer.
+
+This documentation assumes basic knowledge of certain tools and technologies, such as Docker, Bash (Linux) or Git.

docs/dev/build-image.md

@@ -1 +1,69 @@
 # How to generate a container image
+
+This guide includes instructions to generate distribution packages locally using Docker.
+
+Wazuh Indexer supports any of these combinations:
+
+- distributions: `['tar', 'deb', 'rpm']`
+- architectures: `['x64', 'arm64']`
+
+Windows is currently not supported.
+
+> For more information navigate to the [compatibility section](/ref/compatibility.html).
+
+Before you get started, make sure to clean your environment by running `./gradlew clean` on the **root level** of the `wazuh-indexer` repository.
+
+## Pre-requisites
+
+The process to build packages requires Docker and Docker Compose.
+
+- [Install Docker](https://docs.docker.com/engine/install/)
+- [Install Docker Compose](https://docs.docker.com/compose/install/linux/)
+
+Your workstation must meet the minimum hardware requirements (the more resources the better ☺):
+
+   - 8 GB of RAM (minimum)
+   - 4 cores
+
+The tools and source code to generate a package of Wazuh Indexer are hosted in the [wazuh-indexer](https://github.com/wazuh/wazuh-indexer) repository, so clone it if you haven't done already.
+
+## Building `wazuh-indexer` Docker images
+
+The `wazuh-indexer/build-scripts/docker` folder contains the code to build Docker images. Below there is an example of the command needed to build the image. Set the build arguments and the image tag accordingly.
+
+The Docker image is built from a wazuh-indexer tarball (tar.gz), which must be present in the same folder as the Dockerfile in `wazuh-indexer/build-scripts/docker`.
+
+```bash
+docker build \
+   --build-arg="VERSION=5.0.0" \
+   --build-arg="INDEXER_TAR_NAME=wazuh-indexer_5.0.0-0_linux-x64.tar.gz" \
+   --tag=wazuh-indexer:5.0.0-0 \
+   --progress=plain \
+   --no-cache .
+```
+
+Then, start a container with:
+
+```bash
+docker run -p 9200:9200 -it --rm wazuh-indexer:5.0.0-0
+```
+
+The `build-and-push-docker-image.sh` script automates the process to build and push Wazuh Indexer Docker images to our repository in quay.io. The script takes serveral parameters. Use the `-h` option to display them.
+
+To push images, credentials must be set at environment level:
+
+- QUAY_USERNAME
+- QUAY_TOKEN
+
+```bash
+Usage: build-scripts/build-and-push-docker-image.sh [args]
+
+Arguments:
+-n NAME         [required] Tarball name.
+-r REVISION     [Optional] Revision qualifier, default is 0.
+-h help
+```
+
+The script will stop if the credentials are not set, or if any of the required parameters are not provided.
+
+This script is used in the `5_builderpackage_docker.yml` **GitHub Workflow**, which is used to automate the process even more. When possible, **prefer this method**.

docs/dev/build-packages.md

@@ -1 +1,60 @@
 # How to generate a package
+
+This guide includes instructions to generate distribution packages locally using Docker.
+
+Wazuh Indexer supports any of these combinations:
+
+- distributions: `['tar', 'deb', 'rpm']`
+- architectures: `['x64', 'arm64']`
+
+Windows is currently not supported.
+
+> For more information navigate to the [compatibility section](/ref/compatibility.html).
+
+Before you get started, make sure to clean your environment by running `./gradlew clean` on the **root level** of the `wazuh-indexer` repository.
+
+## Pre-requisites
+
+The process to build packages requires Docker and Docker Compose.
+
+- [Install Docker](https://docs.docker.com/engine/install/)
+- [Install Docker Compose](https://docs.docker.com/compose/install/linux/)
+
+Your workstation must meet the minimum hardware requirements (the more resources the better ☺):
+
+   - 8 GB of RAM (minimum)
+   - 4 cores
+
+The tools and source code to generate a package of Wazuh Indexer are hosted in the [wazuh-indexer](https://github.com/wazuh/wazuh-indexer) repository, so clone it if you haven't done already.
+
+## Building `wazuh-indexer` packages
+
+The Docker environment under `wazuh-indexer/build-scripts/builder` automates the build and assemble process for the Wazuh Indexer and its plugins, making it easy to create packages on any system.
+
+Use the `builder.sh` script to build a package.
+
+```bash
+./builder.sh -h
+Usage: ./builder.sh [args]
+
+Arguments:
+-p INDEXER_PLUGINS_BRANCH     [Optional] wazuh-indexer-plugins repo branch, default is 'main'.
+-r INDEXER_REPORTING_BRANCH   [Optional] wazuh-indexer-reporting repo branch, default is 'main'.
+-R REVISION                   [Optional] Package revision, default is '0'.
+-s STAGE                      [Optional] Staging build, default is 'false'.
+-d DISTRIBUTION               [Optional] Distribution, default is 'rpm'.
+-a ARCHITECTURE               [Optional] Architecture, default is 'x64'.
+-D      Destroy the docker environment
+-h      Print help
+```
+
+The example below it will generate a wazuh-indexer package for _Debian_ based systems, for the _x64_ architecture, using _1_ as revision number and using the production naming convention.
+
+```bash
+# Wihtin wazuh-indexer/build-scripts/builder
+bash builder.sh -d deb -a x64 -R 1 -s true
+```
+
+The resulting package will be stored at `wazuh-indexer/artifacts/dist`.
+
+> The `STAGE` option defines the naming of the package. When set to `false`, the package will be unequivocally named with the commits' SHA of the `wazuh-indexer`, `wazuh-indexer-plugins` and `wazuh-indexer-reporting` repositories, in that order. For example: `wazuh-indexer_5.0.0-0_x86_64_aff30960363-846f143-494d125.rpm`.

docs/dev/build-sources.md

@@ -1 +1,5 @@
 # How to build from sources
+
+Every Wazuh Indexer repository includes one or more Gradle projects with predefined tasks to run and build the source code.
+
+In this case, to build (compile and zip) a distribution of Wazuh Indexer, run the `./gradlew build` command at the root level of the repository. When completed, the distribution artifacts will be located in the `build/distributions` directory.

docs/dev/run-sources.md

@@ -1 +1,46 @@
 # How to run from sources
+
+Every Wazuh Indexer repository includes one or more Gradle projects with predefined tasks to run and build the source code.
+
+In this case, to run a Gradle project from source code, run the `./gradlew run` command.
+
+For Wazuh Indexer, additonal plugins may be installed by passing the `-PinstalledPlugins` flag:
+
+```bash
+./gradlew run -PinstalledPlugins="['plugin1', 'plugin2']"
+```
+
+The `./gradlew run` command will build and start the project, writing its log above Gradle's status message. A lot of stuff is logged on startup, specifically these lines tell you that OpenSearch is ready.
+
+```
+[2020-05-29T14:50:35,167][INFO ][o.e.h.AbstractHttpServerTransport] [runTask-0] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
+[2020-05-29T14:50:35,169][INFO ][o.e.n.Node               ] [runTask-0] started
+```
+
+It's typically easier to wait until the console stops scrolling, and then run `curl` in another window to check if OpenSearch instance is running.
+
+```bash
+curl localhost:9200
+
+{
+  "name" : "runTask-0",
+  "cluster_name" : "runTask",
+  "cluster_uuid" : "oX_S6cxGSgOr_mNnUxO6yQ",
+  "version" : {
+    "number" : "1.0.0-SNAPSHOT",
+    "build_type" : "tar",
+    "build_hash" : "0ba0e7cc26060f964fcbf6ee45bae53b3a9941d0",
+    "build_date" : "2021-04-16T19:45:44.248303Z",
+    "build_snapshot" : true,
+    "lucene_version" : "8.7.0",
+    "minimum_wire_compatibility_version" : "6.8.0",
+    "minimum_index_compatibility_version" : "6.0.0-beta1"
+  }
+}
+```
+
+Use `-Dtests.opensearch.` to pass additional settings to the running instance. For example, to enable OpenSearch to listen on an external IP address pass `-Dtests.opensearch.http.host`. Make sure your firewall or security policy allows external connections for this to work.
+
+```bash
+./gradlew run -Dtests.opensearch.http.host=0.0.0.0
+```

docs/dev/run-tests.md

@@ -1 +1,21 @@
 # How to run the tests
+
+This section explains how to run the Wazuh Indexer tests.
+
+### Full set of tests
+
+To execute all kind of tests, use the command `./gradlew check`. This command does not only run tests, but also tasks to check the quality of the code, such as documentation and linter checks.
+
+### Unit tests
+
+To run unit tests, use the `./gradlew test` command.
+
+### Integration tests
+
+To run integration tests, use the `./gradlew integTest` and the `./gradlew yamlresttest` commands.
+
+### Package testing
+
+For package testing, we conduct smoke tests on the packages using the [GitHub Actions Workflows](https://github.com/wazuh/wazuh-indexer/blob/main/.github/workflows/5_builderpackage_indexer.yml). These tests consist on installing the packages on a supported operating system. DEB packages are installed in the “Ubuntu 24.04” runner executing the workflow, while RPM packages are installed in a Red Hat 9 Docker container, as there is no RPM compatible runner available in GitHub Actions.
+
+As a last note, there is also a **Vagrantfile** and **testing scripts** in the [repository](https://github.com/wazuh/wazuh-indexer-plugins/tree/main/test-tools) to perform some tests on a real wazuh-indexer service running on a virtual machine. Refer to its README.md for more information about how to run these tests.

docs/dev/setup.md

@@ -1 +1,57 @@
 # Set up the development environment
+
+## 1. Git
+
+Install and configure Git (ssh keys, commits and tags signing, user and email).
+
+1. [Set your username](https://docs.github.com/en/get-started/getting-started-with-git/setting-your-username-in-git).
+1. [Set your email address](https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/setting-your-commit-email-address).
+1. Generate an [SSH key](https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key).
+1. Add the public key to your [GitHub account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account) for authentication and signing.
+1. [Configure Git to sign commits with your SSH key](https://docs.gitlab.com/ee/user/project/repository/signed_commits/ssh.html#configure-git-to-sign-commits-with-your-ssh-key).
+
+## 2. Repositories
+
+Before  you  start,  you  need  to  properly  configure  your  working repository to have *origin* and *upstream* remotes.
+
+* Clone the `wazuh-indexer` fork
+
+    ```bash
+    git clone [email protected]:wazuh-indexer.git
+    git remote add upstream [email protected]:opensearch-project/opensearch.git
+    ```
+
+* Clone the `wazuh-indexer-reporting` fork
+
+    ```bash
+    git clone [email protected]:wazuh/wazuh-indexer-reporting.git
+    git remote add upstream [email protected]:opensearch-project/reporting.git
+    ```
+
+* Clone the `wazuh-indexer-plugins` repository
+
+    ```bash
+    git clone [email protected]:wazuh/wazuh-indexer-plugins.git
+    ```
+
+## 3. IntelliJ IDEA
+
+Prepare your IDE:
+
+1. Install IDEA Community Edition as per the [official documentation](https://www.jetbrains.com/help/idea/installation-guide.html).
+1. Set a global SDK to Eclipse Temurin following  [this guide](https://www.jetbrains.com/help/idea/sdk.html#add_global_sdk).
+
+> You can find the JDK version to use under the `wazuh-indexer/gradle/libs.versions.toml` file. IntelliJ IDEA includes some JDKs by default. If you need to change it, or if you want to use a different distribution, follow the instructions on the next section.
+
+## 4. JDK (Optional)
+
+We use the Eclipse Temurin JDK. To use a different JDK installed on your machine, use `sudo update-alternatives --config java` to select the JDK of your preference.
+
+Set the **JAVA\_HOME** and **PATH** environment variables by adding these lines to your Shell RC file (`.bashrc`, `.zsrhrc`, …):
+
+```bash
+export JAVA_HOME=/usr/lib/jvm/temurin-21-jdk-amd64
+export PATH=$PATH:/usr/lib/jvm/temurin-21-jdk-amd64/bin
+```
+
+After that, restart your shell or run `source  ~/.zshrc` (or similar) to apply the changes. Finally, check Java is installed correctly by running `java --version`.

docs/ref/README.md

@@ -1 +1,3 @@
 # Reference manual
+
+This section contains the Wazuh Indexer reference manual, composed by installation, upgrade, removal, restore and configuration instructions, performance and security recommendations, architecture and API references for each of the modules composing the Wazuh Indexer.