docs: Update the contribution guidelines

README.adoc

@@ -1,14 +1,21 @@
-// Header of this document:
-//
-
 = Stackable Documentation
 :base-repo: https://github.com/stackabletech
 
 This is the main repository for the documentation of the Stackable platform.
-Have a look at the https://docs.stackable.tech/[live version].
+Have a look at the https://docs.stackable.tech/[live version] and the current https://docs.stackable.tech/home/nightly/[nightly live version].
 
 The documentation is built with https://antora.org[Antora]. This repository hosts the Antora playbook file as well as platform documentation. Other Stackable repos contain 'docs' directories which are pulled in by this repo.
 
+== Repository structure
+
+* The link:antora.yml[`antora.yml`] file defines the main `home` https://docs.antora.org/antora/latest/component-version/#docs-component[Component].
+* The various `*-playbook.yml` files are https://docs.antora.org/antora/latest/playbook/[Antora playbook files] used to build the docs, either locally or for production. The playbooks link to all the other repositories that contain content.
+* The `modules` directory contains the platform level documentation content.
+
+== Building locally
+
+**Dependencies**: `make`, `npm`.
+
 To build the site, pull submodules, install dependencies and run `make`:
 
 [source,console]
@@ -20,6 +27,19 @@ $ make
 
 NOTE: Antora caches the external content repos in the cache directory (configured to be `./cache`). It will _not_ automatically update those.  Use the `--fetch` flag (`make ANTORAFLAGS=--fetch`) to update all sources, or use `make clean` to delete the `cache` and `build` directory.
 
+== Production deployment
+
+The documentation is deployed by Netlify on a regular basis or when something is pushed to  the `main` branch.
+To trigger an out of band deployment, use the _Build and deploy production site_ GitHub action (internal contributors only).
+
+Regular (nightly) deployments  are run to pick up changes in the operator repositories. These repositories are not watched by Netlify and thus any changes to the documentation there would be ignored.
+
+=== Netlify configuration
+
+Netlify is configured with the link:netlify.toml[`netlify.toml`] file inside of the documentation repo. In there, the command `make netlify-build` is configured as the build command. Further documentation of the build process can then be found in the link:Makefile[`Makefile`].
+
+The build process creates a static site in `build/site` which is then published (as it is configured in the `netlify.toml`).
+
 == Development
 
 === Generating the documentation
@@ -40,10 +60,8 @@ The design & layout comes from our https://github.com/stackabletech/documentatio
 
 `LIVERELOAD=true gulp` may be used to recreate the built documentation after each edit. The 'live reload' feature does not work over gitpod currently due to https://github.com/schickling/gulp-webserver/pull/126
 
-=== Remote IDE
-
-A remote vscode instance can be used via https://gitpod.io/#/github.com/stackabletech/documentation
-
-== Tracking
+== More useful links
 
-The documentation UI includes a script for user tracking. The documentation can be built with or without it. Tracking enabled/disabled by setting `site.keys.enable_tracking` to `true` or `false`. The tracking code is in the UI repository in `src/partials/head-scripts.hbs`.
+* The https://github.com/stackabletech/documentation-ui[documentation-ui] repository.
+* The https://github.com/stackabletech/crddocs[crddocs] repository from which the https://crds.stackable.tech/[CRD reference] is generated.
+* The Stackable https://docs.stackable.tech/home/stable/contributor/[contributor's guide] containing more info on how to contribute to the documentation.

antora-playbook.yml

@@ -5,7 +5,7 @@ site:
   start_page: home::index.adoc
   robots: allow
   keys:
-    enable_tracking: true
+    enable_tracking: true # this key is used in the documentation-ui
 # URL config settings.
 # docs: https://docs.antora.org/antora/latest/playbook/configure-urls/
 urls:

local-antora-playbook.yml

@@ -5,7 +5,7 @@ site:
   start_page: home::index.adoc
   robots: allow
   keys:
-    enable_tracking: false
+    enable_tracking: false # this key is used in the documentation-ui
 urls:
   # This replaces the component version in the URL of the latest stable version with 'stable'
   # i.e. /commons-operator/stable/index.html instead of /commons-operator/0.3/index.html

modules/contributor/nav.adoc

@@ -1,12 +1,16 @@
 * xref:index.adoc[]
 ** xref:project-overview.adoc[]
-** xref:steps.adoc[]
-** xref:testing_on_kubernetes.adoc[]
-** xref:code-style-guide.adoc[]
-** xref:docs-style-guide.adoc[]
-** Implementation guidelines
+** xref:contributing-code.adoc[]
+*** xref:testing_on_kubernetes.adoc[]
+*** xref:code-style-guide.adoc[]
 *** xref:guidelines/logging.adoc[]
 *** xref:guidelines/service-discovery.adoc[]
 *** xref:guidelines/opa-configuration.adoc[]
 *** xref:guidelines/webhook-server.adoc[]
+** xref:docs/contributing-documentation.adoc[]
+*** xref:docs/overview.adoc[]
+*** xref:docs/style-guide.adoc[]
+*** xref:docs/backporting-changes.adoc[]
+*** xref:docs/releasing-a-new-version.adoc[]
+*** xref:docs/troubleshooting-antora.adoc[]
 include::partial$adr-nav.adoc[]

modules/contributor/pages/contributing-code.adoc

@@ -0,0 +1,98 @@
+= Contributing code
+
+:templating-repo: https://github.com/stackabletech/operator-templating
+:operator-repo: https://github.com/stackabletech/operator-rs
+:docker-repo: https://github.com/stackabletech/docker-images
+
+== Development Environment
+
+In order to contribute source code, you need an environment that is capable of running the following tools:
+
+* https://git-scm.com/[Git]
+* https://www.gnu.org/software/make/manual/make.html[make]
+* https://www.docker.com/[Docker]
+* https://kind.sigs.k8s.io/[Kind]
+* https://helm.sh/[Helm]
+* https://kuttl.dev/[Kuttl]
+* https://www.rust-lang.org/[Rust]
+* https://www.python.org/[Python]
+* https://jqlang.github.io/jq/[jq]
+* https://github.com/mikefarah/yq[yq]
+* https://tilt.dev/[Tilt]
+* https://pre-commit.com/[pre-commit] (optional)
+
+Depending on the repository, you might also need https://go.dev/[Go], https://www.java.com/en/[Java] or other programming language support.
+
+Almost all build scripts assume a Unix based environment (preferably Linux).
+
+=== IDEs and Editors
+
+Of course you are free to use whatever works for you best. No editor is perfect but we have positive experience with:
+
+* https://www.jetbrains.com/idea/[IntelliJ Idea] with the `Rust` plug-in
+* https://code.visualstudio.com/[VisualStudio Code] with the `rust-analyzer` extension
+
+For `VisualStudio Code` we also recommend the following extensions:
+
+* https://marketplace.visualstudio.com/items?itemName=tamasfe.even-better-toml[Even Better TOML]
+* https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb[CodeLLDB] (for debugging)
+* https://marketplace.visualstudio.com/items?itemName=usernamehw.errorlens[Error Lens] (inline error messages)
+* https://marketplace.visualstudio.com/items?itemName=asciidoctor.asciidoctor-vscode[AsciiDoc]
+* https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-pull-request-github[GitHub Pull requests and Issues]
+* https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens[GitLens]
+* https://marketplace.visualstudio.com/items?itemName=ms-python.python[Python]
+* https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker[Docker]
+
+== Steps
+
+. Make your desired changes in the according repository and test them manually. Ensure that the code compiles without
+  warnings (`cargo clippy --all-targets`) and that the code is formatted with `cargo fmt`. Also make sure that all
+  changes are made in accordance to the xref:code-style-guide.adoc[source code style guide].
+. If code was added or adapted then please create or adapt the unit tests in the same file as well as the integration
+  tests in the `tests` directory. Ensure that all unit tests run successfully `cargo test`) and all integration tests
+  run successfully (`./scripts/run-tests`). See also <<_changes_in_the_integration_tests>>.
+. Comment your code and check with `cargo doc --document-private-items` that there are no syntax errors.
+. The YAML schemas of the custom resource definitions (CRDs) are rebuilt when the project is compiled (see
+  `rust/operator-binary/build.rs` if changing an operator). These CRDs as well as the product configuration are also
+  required in the Helm chart and the Kubernetes manifest. To ensure that everything is in a consistent state, please
+  execute `make regenerate-charts`.
+. If it is useful for the users of the project to know about the change then it must be added to the changelog. For
+  instance, if only the dependencies in an operator are upgraded but nothing changes for the user then the upgrade
+  should not be added to the changelog. Conversely, if the dependencies in the {operator-repo}[operator framework] are
+  upgraded then changes are probably required in the operators (which are the clients of the framework) and therefore
+  the upgrade must be mentioned in the changelog. The changelog must be formatted according to
+  https://keepachangelog.com/en/1.1.0/[keep a changelog].
+
+== Changes in the integration tests
+
+. Most code changes should also be tested with integration tests. The tests for every operator can be found in the
+  operator repository in the `tests` directory.
+. Create or adapt the tests.
+  Try to mimic the style of the other tests.
+  They are written with https://kuttl.dev/[KUTTL] and our own templating tool https://github.com/stackabletech/beku.py[beku.py] to template tests and test multiple product versions at once.
+. Start a test cluster using https://kind.sigs.k8s.io/[kind].
+. Run your development version of the operator with `make run-dev` (see also xref:testing-on-kubernetes.adoc[] for more information on this).
+  This will deploy the operator directly into the cluster, also using part of the Helm Chart definition and therefore the RBAC rules.
+. When making changes to the Helm Chart, you should however test the Helm Chart explicitly.
+  First a Docker image of the operator must be built locally and uploaded to the kind cluster and then the Helm chart must be installed.
+  This can be achieved in the operator directory with the following commands:
++
+[source,bash]
+----
+docker build --file docker/Dockerfile --tag docker.stackable.tech/stackable/<operator>:<version>-dev .
+kind load docker-image docker.stackable.tech/stackable/<operator>:<version>-dev --name=integration-tests
+helm install <operator> deploy/helm/<operator>/
+----
+. Run the tests from the repository root with `./scripts/run-tests`.
+
+== Adding support for a new product version
+
+If a new version of a product was added then the following tasks must be performed:
+
+* Add the new version in the https://github.com/stackabletech/docker-images[docker-images] repository.
+* Update the operator to support the new version if necessary.
+* Update the examples in the operator to use the new version.
+* Update the integration tests.
+  The tests should cover the latest patch version of each supported versions.
+* Add the new version to the supported ones in the documentation of the operators (see
+  `docs/modules/\{product name\}/partials/supported-versions.adoc` in the operator repositories).

modules/contributor/pages/docs/backporting-changes.adoc

@@ -0,0 +1,22 @@
+= Backporting changes
+
+The documentation uses https://trunkbaseddevelopment.com/[trunk based development], so any new content or fixes should first be applied to the `main` branch and then ported to the release branches where the feature/fix also applies.
+
+== Prerequisites
+
+* Make sure your changes are committed to the `main` branch and you have all the latest changes checked out locally on your `main` branch.
+* Have the commit ID of the commit that you want to port to a release branch.
+  You can get the commit ID for example by looking at the log: `git log --oneline -n 5`.
+  The commit ID might look like this: `bc0b08e9`.
+
+== Steps
+
+. Switch to the release branch you want to backport to, for example: `git switch release/23.11`.
+  Make sure the release branch is up to date with the upstream (`git pull`).
+. Cherry-pick the commit with the ID you retrieved earlier: `git cherry-pick bc0b08e9`.
+  In most cases this will work without changes, sometimes you need to do conflict resolution, similar to how you would need to do it for a merge.
+. Push the new commit in the `release/23.11` branch upstream with `git push`.
+  That's it, you're done!
+
+The changes will become visible in the online documentation once the next build is triggered.
+You can either wait for the nightly build, or trigger a build yourself with the https://github.com/stackabletech/documentation/actions/workflows/deploy.yml[Build and deploy production site] GitHub action.

modules/contributor/pages/docs/contributing-documentation.adoc

@@ -0,0 +1,6 @@
+= Contributing to documentation
+:page-alias: contributing-documentation.adoc
+
+To understand the repositories involved in building the documentation, see xref:project-overview.adoc[]. For an overview of the content structure and the tools involved, see the  xref:docs/overview.adoc[].
+
+To contribute, follow the pull request flow outlined in the xref:index.adoc[].

modules/contributor/pages/docs/overview.adoc

@@ -0,0 +1,81 @@
+= Documentation overview
+:figure-caption!:
+:antora-docs: https://docs.antora.org/antora/latest/
+:antora-playbook: https://docs.antora.org/antora/latest/playbook/
+:netlify: https://www.netlify.com/
+:diataxis: https://diataxis.fr/
+:documentation: https://github.com/stackabletech/documentation
+
+We use {antora-docs}[Antora] to write our user facing documentation,
+{netlify}[Netlify] to host it and the {diataxis}[Diátaxis] framework as the guide for the structure of the content.
+The main repository for the documentation is the {documentation}[Documentation] repository and
+each operator and other tools have a `docs` directory from which content is pulled in.
+Have a look at the xref:project-overview.adoc[] to learn about the repositories that are involved in providing the documentation.
+
+== Content structure: Diátaxis
+
+The {diataxis}[Diátaxis] framework is a way to classify documentation content into four groups with distinct use cases.
+
+.Source: https://diataxis.fr/
+image::diataxis.png[]
+
+Documentation exists along two axis: _study_ or _work_; _pracital steps_ and _theory_.
+Practical learning happens in tutorials, and they are backed up by conceptual background information (theory).
+On the work oriented side, documentation should contain exhaustive reference information (every commandline flag, every yaml property, their type and whether they are optional or not etc.) as well as narrow, task oriented guides.
+
+At Stackable we have to document the platform as a whole, individual operators, products and command line tools like `stackablectl`.
+
+**Conceptual** information lives at the platform level in the xref:concepts:index.adoc[] section. Some operators have their own specific concepts such as xref:zookeeper:znodes.adoc[] for ZooKeeper, but most concepts apply to all operators (i.e. roles and role groups, resource management, config overrides).
+
+**Tutorials** (learning oriented guides) exist at the xref:tutorials:index.adoc[top level] but there are also the Getting Started guides for all the individual operators.
+Tutorials provide complete set up information and require very little to be there already in contrast to guides (work instead of study oriented) which are more focused on solving specific tasks.
+There are also the xref:demos:index.adoc[Demos] which are complete solutions that showcase how the platform integrates different products in a unified, reproducible and transparent way.
+
+**Guides** are also instructional, but focus a narrow topic, something that the user might want to solve in their particular setup.
+For example: "How do I set up Kerberos with HDFS?" or "How do I connect Superset to Druid?"
+The guide has to account for different use-cases (i.e. the user is using their own pre-existing ZooKeeper instead of our ZooKeeper). This is not the case for tutorials.
+Since this kind of information is typically product specific, it is located in the usage guide section of individual operators.
+
+**Reference** information for the Stackable platform entails all the settings and Options in our YAMLs, which we generate.
+The reference is found at https://crds.stackable.tech/ and generated from the https://github.com/stackabletech/crddocs[crddocs repository].
+
+=== Style guide
+
+The xref:docs/style-guide.adoc[] contains all the information about the writing style, formatting style, naming documents and more.
+
+== Technical bits: Antora, Netlify
+
+{antora-docs}[Antora] uses a {antora-playbook}[playbook] to build the documentation.
+It pulls information from all the individual operators, so their documentation can live in the same repository.
+Antora builds a static website which we serve over {netlify}[Netlify].
+The web template of the documentation is also custom made and is developed in the https://github.com/stackabletech/documentation-ui[documentation-ui] repository.
+
+For Antora, the https://antora.zulipchat.com/[Antora Zulip chatroom] is a good place to get help (besides the documentation)!
+
+Building the documentation and also the deployment process on Netlify are documented in the https://github.com/stackabletech/documentation/blob/main/README.adoc[README] file of the documentation repository.
+
+== Executable tutorials
+
+The getting started guides for each operator are backed by an executable script from which the documentation includes excerpts.
+This has the benefit that the guides are easily tested.
+When writing a new guide, please also write it in this way.
+Have a look at the existing getting started guides on how to do this.
+
+== Templating
+
+There is a templating mechanism in the docs.
+This has been introduced to template in mostly version numbers, so the updating doesn't have to be done by hand. 
+
+Every Operator repo has a script `scripts/docs_templating.sh` and a file with templating variables `docs/templating_vars.yaml`.
+The script applies the variables to all `.j2` files in the `docs` directory.
+
+This is used for getting started scripts, and in there only for versions of operators and for Helm Chart URLs.
+
+Without this templating mechanism, every release these values would need to be updated by hand (or possibly with a search and replace) with is error prone and time consuming.
+
+== Branch and version structure
+
+All Stackable repositories use https://trunkbaseddevelopment.com/[trunk based development], and so the documentation is also pulled from different release branches.
+This means that any changes from the `main` branch that should be published in other versions need to be cherry-picked over into that branch.
+
+Antora recommends using https://docs.antora.org/antora/latest/playbook/content-branches/[branches] to organize different versions of components.

modules/contributor/pages/docs/releasing-a-new-version.adoc

@@ -0,0 +1,13 @@
+= Releasing a new documentation version
+
+NOTE: This guide is directed at internal contributors, as an external contributor, you cannot release a new documentation version.
+
+Whenever there is a new Stackable Data Platform release, the documentation is also released with a new version.
+This process has been automated with scripts, which are found in the https://github.com/stackabletech/documentation/tree/main/scripts[`scripts`] directory of the documentation repository.
+
+The process consists of two steps: 
+
+. Making a new release branch (`make-release-branch.sh`)
+. Publishing the new version by modifying the playbooks (`publish-new-version.sh`)
+
+Consult the scripts for details about the required steps, as well as prerequisites.

modules/contributor/pages/docs-style-guide.adoc → modules/contributor/pages/docs/style-guide.adoc

@@ -1,5 +1,5 @@
 = Documentation style guide
-:page-aliases: style_guide.adoc, style-guide.adoc
+:page-aliases: style_guide.adoc, style-guide.adoc, docs-style-guide.adoc
 
 :asciidoc-recommended-practices: https://asciidoctor.org/docs/asciidoc-recommended-practices[AsciiDoc recommended practices]
 :kubernetes-style-guide: https://kubernetes.io/docs/contribute/style/style-guide/[Kubernetes style guide]