Skip to content

Commit f9a23e2

Browse files
fhennigrazvan
andauthored
docs: Update the contribution guidelines (#577)
* A bunch of changes * formatting fixes * Added some languages that are needed * Turn doc page into a stub to be written later. * ~ * ~ * a bunch of changes * docs: Add release guide * docs: Add backporting guide * docs: add troubleshooting page * docs: add note about executable tutorials * docs: add note about templating * docs: add note about signed commits * Update README.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update README.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update README.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/contributing-code.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/contributing-code.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/contributing-documentation.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * fixes from the review * formatting fixes * formatting fixes * formatting fixes * more mini README improvements * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * Update modules/contributor/pages/docs/overview.adoc Co-authored-by: Razvan-Daniel Mihai <[email protected]> * added a couple more tools to the list * clarified some aspects of integration test instructions * add 'add product image' step to the instructions for adding a new version --------- Co-authored-by: Razvan-Daniel Mihai <[email protected]>
1 parent b6b6c74 commit f9a23e2

19 files changed

+323
-208
lines changed

README.adoc

+28-10
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,21 @@
1-
// Header of this document:
2-
//
3-
41
= Stackable Documentation
52
:base-repo: https://github.com/stackabletech
63

74
This is the main repository for the documentation of the Stackable platform.
8-
Have a look at the https://docs.stackable.tech/[live version].
5+
Have a look at the https://docs.stackable.tech/[live version] and the current https://docs.stackable.tech/home/nightly/[nightly live version].
96

107
The documentation is built with https://antora.org[Antora]. This repository hosts the Antora playbook file as well as platform documentation. Other Stackable repos contain 'docs' directories which are pulled in by this repo.
118

9+
== Repository structure
10+
11+
* The link:antora.yml[`antora.yml`] file defines the main `home` https://docs.antora.org/antora/latest/component-version/#docs-component[Component].
12+
* The various `*-playbook.yml` files are https://docs.antora.org/antora/latest/playbook/[Antora playbook files] used to build the docs, either locally or for production. The playbooks link to all the other repositories that contain content.
13+
* The `modules` directory contains the platform level documentation content.
14+
15+
== Building locally
16+
17+
**Dependencies**: `make`, `npm`.
18+
1219
To build the site, pull submodules, install dependencies and run `make`:
1320

1421
[source,console]
@@ -20,6 +27,19 @@ $ make
2027

2128
NOTE: Antora caches the external content repos in the cache directory (configured to be `./cache`). It will _not_ automatically update those. Use the `--fetch` flag (`make ANTORAFLAGS=--fetch`) to update all sources, or use `make clean` to delete the `cache` and `build` directory.
2229

30+
== Production deployment
31+
32+
The documentation is deployed by Netlify on a regular basis or when something is pushed to the `main` branch.
33+
To trigger an out of band deployment, use the _Build and deploy production site_ GitHub action (internal contributors only).
34+
35+
Regular (nightly) deployments are run to pick up changes in the operator repositories. These repositories are not watched by Netlify and thus any changes to the documentation there would be ignored.
36+
37+
=== Netlify configuration
38+
39+
Netlify is configured with the link:netlify.toml[`netlify.toml`] file inside of the documentation repo. In there, the command `make netlify-build` is configured as the build command. Further documentation of the build process can then be found in the link:Makefile[`Makefile`].
40+
41+
The build process creates a static site in `build/site` which is then published (as it is configured in the `netlify.toml`).
42+
2343
== Development
2444

2545
=== Generating the documentation
@@ -40,10 +60,8 @@ The design & layout comes from our https://github.com/stackabletech/documentatio
4060

4161
`LIVERELOAD=true gulp` may be used to recreate the built documentation after each edit. The 'live reload' feature does not work over gitpod currently due to https://github.com/schickling/gulp-webserver/pull/126
4262

43-
=== Remote IDE
44-
45-
A remote vscode instance can be used via https://gitpod.io/#/github.com/stackabletech/documentation
46-
47-
== Tracking
63+
== More useful links
4864

49-
The documentation UI includes a script for user tracking. The documentation can be built with or without it. Tracking enabled/disabled by setting `site.keys.enable_tracking` to `true` or `false`. The tracking code is in the UI repository in `src/partials/head-scripts.hbs`.
65+
* The https://github.com/stackabletech/documentation-ui[documentation-ui] repository.
66+
* The https://github.com/stackabletech/crddocs[crddocs] repository from which the https://crds.stackable.tech/[CRD reference] is generated.
67+
* The Stackable https://docs.stackable.tech/home/stable/contributor/[contributor's guide] containing more info on how to contribute to the documentation.

antora-playbook.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ site:
55
start_page: home::index.adoc
66
robots: allow
77
keys:
8-
enable_tracking: true
8+
enable_tracking: true # this key is used in the documentation-ui
99
# URL config settings.
1010
# docs: https://docs.antora.org/antora/latest/playbook/configure-urls/
1111
urls:

local-antora-playbook.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ site:
55
start_page: home::index.adoc
66
robots: allow
77
keys:
8-
enable_tracking: false
8+
enable_tracking: false # this key is used in the documentation-ui
99
urls:
1010
# This replaces the component version in the URL of the latest stable version with 'stable'
1111
# i.e. /commons-operator/stable/index.html instead of /commons-operator/0.3/index.html
141 KB
Loading

modules/contributor/nav.adoc

+9-5
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,16 @@
11
* xref:index.adoc[]
22
** xref:project-overview.adoc[]
3-
** xref:steps.adoc[]
4-
** xref:testing_on_kubernetes.adoc[]
5-
** xref:code-style-guide.adoc[]
6-
** xref:docs-style-guide.adoc[]
7-
** Implementation guidelines
3+
** xref:contributing-code.adoc[]
4+
*** xref:testing_on_kubernetes.adoc[]
5+
*** xref:code-style-guide.adoc[]
86
*** xref:guidelines/logging.adoc[]
97
*** xref:guidelines/service-discovery.adoc[]
108
*** xref:guidelines/opa-configuration.adoc[]
119
*** xref:guidelines/webhook-server.adoc[]
10+
** xref:docs/contributing-documentation.adoc[]
11+
*** xref:docs/overview.adoc[]
12+
*** xref:docs/style-guide.adoc[]
13+
*** xref:docs/backporting-changes.adoc[]
14+
*** xref:docs/releasing-a-new-version.adoc[]
15+
*** xref:docs/troubleshooting-antora.adoc[]
1216
include::partial$adr-nav.adoc[]
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
= Contributing code
2+
3+
:templating-repo: https://github.com/stackabletech/operator-templating
4+
:operator-repo: https://github.com/stackabletech/operator-rs
5+
:docker-repo: https://github.com/stackabletech/docker-images
6+
7+
== Development Environment
8+
9+
In order to contribute source code, you need an environment that is capable of running the following tools:
10+
11+
* https://git-scm.com/[Git]
12+
* https://www.gnu.org/software/make/manual/make.html[make]
13+
* https://www.docker.com/[Docker]
14+
* https://kind.sigs.k8s.io/[Kind]
15+
* https://helm.sh/[Helm]
16+
* https://kuttl.dev/[Kuttl]
17+
* https://www.rust-lang.org/[Rust]
18+
* https://www.python.org/[Python]
19+
* https://jqlang.github.io/jq/[jq]
20+
* https://github.com/mikefarah/yq[yq]
21+
* https://tilt.dev/[Tilt]
22+
* https://pre-commit.com/[pre-commit] (optional)
23+
24+
Depending on the repository, you might also need https://go.dev/[Go], https://www.java.com/en/[Java] or other programming language support.
25+
26+
Almost all build scripts assume a Unix based environment (preferably Linux).
27+
28+
=== IDEs and Editors
29+
30+
Of course you are free to use whatever works for you best. No editor is perfect but we have positive experience with:
31+
32+
* https://www.jetbrains.com/idea/[IntelliJ Idea] with the `Rust` plug-in
33+
* https://code.visualstudio.com/[VisualStudio Code] with the `rust-analyzer` extension
34+
35+
For `VisualStudio Code` we also recommend the following extensions:
36+
37+
* https://marketplace.visualstudio.com/items?itemName=tamasfe.even-better-toml[Even Better TOML]
38+
* https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb[CodeLLDB] (for debugging)
39+
* https://marketplace.visualstudio.com/items?itemName=usernamehw.errorlens[Error Lens] (inline error messages)
40+
* https://marketplace.visualstudio.com/items?itemName=asciidoctor.asciidoctor-vscode[AsciiDoc]
41+
* https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-pull-request-github[GitHub Pull requests and Issues]
42+
* https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens[GitLens]
43+
* https://marketplace.visualstudio.com/items?itemName=ms-python.python[Python]
44+
* https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker[Docker]
45+
46+
== Steps
47+
48+
. Make your desired changes in the according repository and test them manually. Ensure that the code compiles without
49+
warnings (`cargo clippy --all-targets`) and that the code is formatted with `cargo fmt`. Also make sure that all
50+
changes are made in accordance to the xref:code-style-guide.adoc[source code style guide].
51+
. If code was added or adapted then please create or adapt the unit tests in the same file as well as the integration
52+
tests in the `tests` directory. Ensure that all unit tests run successfully `cargo test`) and all integration tests
53+
run successfully (`./scripts/run-tests`). See also <<_changes_in_the_integration_tests>>.
54+
. Comment your code and check with `cargo doc --document-private-items` that there are no syntax errors.
55+
. The YAML schemas of the custom resource definitions (CRDs) are rebuilt when the project is compiled (see
56+
`rust/operator-binary/build.rs` if changing an operator). These CRDs as well as the product configuration are also
57+
required in the Helm chart and the Kubernetes manifest. To ensure that everything is in a consistent state, please
58+
execute `make regenerate-charts`.
59+
. If it is useful for the users of the project to know about the change then it must be added to the changelog. For
60+
instance, if only the dependencies in an operator are upgraded but nothing changes for the user then the upgrade
61+
should not be added to the changelog. Conversely, if the dependencies in the {operator-repo}[operator framework] are
62+
upgraded then changes are probably required in the operators (which are the clients of the framework) and therefore
63+
the upgrade must be mentioned in the changelog. The changelog must be formatted according to
64+
https://keepachangelog.com/en/1.1.0/[keep a changelog].
65+
66+
== Changes in the integration tests
67+
68+
. Most code changes should also be tested with integration tests. The tests for every operator can be found in the
69+
operator repository in the `tests` directory.
70+
. Create or adapt the tests.
71+
Try to mimic the style of the other tests.
72+
They are written with https://kuttl.dev/[KUTTL] and our own templating tool https://github.com/stackabletech/beku.py[beku.py] to template tests and test multiple product versions at once.
73+
. Start a test cluster using https://kind.sigs.k8s.io/[kind].
74+
. Run your development version of the operator with `make run-dev` (see also xref:testing-on-kubernetes.adoc[] for more information on this).
75+
This will deploy the operator directly into the cluster, also using part of the Helm Chart definition and therefore the RBAC rules.
76+
. When making changes to the Helm Chart, you should however test the Helm Chart explicitly.
77+
First a Docker image of the operator must be built locally and uploaded to the kind cluster and then the Helm chart must be installed.
78+
This can be achieved in the operator directory with the following commands:
79+
+
80+
[source,bash]
81+
----
82+
docker build --file docker/Dockerfile --tag docker.stackable.tech/stackable/<operator>:<version>-dev .
83+
kind load docker-image docker.stackable.tech/stackable/<operator>:<version>-dev --name=integration-tests
84+
helm install <operator> deploy/helm/<operator>/
85+
----
86+
. Run the tests from the repository root with `./scripts/run-tests`.
87+
88+
== Adding support for a new product version
89+
90+
If a new version of a product was added then the following tasks must be performed:
91+
92+
* Add the new version in the https://github.com/stackabletech/docker-images[docker-images] repository.
93+
* Update the operator to support the new version if necessary.
94+
* Update the examples in the operator to use the new version.
95+
* Update the integration tests.
96+
The tests should cover the latest patch version of each supported versions.
97+
* Add the new version to the supported ones in the documentation of the operators (see
98+
`docs/modules/\{product name\}/partials/supported-versions.adoc` in the operator repositories).
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
= Backporting changes
2+
3+
The documentation uses https://trunkbaseddevelopment.com/[trunk based development], so any new content or fixes should first be applied to the `main` branch and then ported to the release branches where the feature/fix also applies.
4+
5+
== Prerequisites
6+
7+
* Make sure your changes are committed to the `main` branch and you have all the latest changes checked out locally on your `main` branch.
8+
* Have the commit ID of the commit that you want to port to a release branch.
9+
You can get the commit ID for example by looking at the log: `git log --oneline -n 5`.
10+
The commit ID might look like this: `bc0b08e9`.
11+
12+
== Steps
13+
14+
. Switch to the release branch you want to backport to, for example: `git switch release/23.11`.
15+
Make sure the release branch is up to date with the upstream (`git pull`).
16+
. Cherry-pick the commit with the ID you retrieved earlier: `git cherry-pick bc0b08e9`.
17+
In most cases this will work without changes, sometimes you need to do conflict resolution, similar to how you would need to do it for a merge.
18+
. Push the new commit in the `release/23.11` branch upstream with `git push`.
19+
That's it, you're done!
20+
21+
The changes will become visible in the online documentation once the next build is triggered.
22+
You can either wait for the nightly build, or trigger a build yourself with the https://github.com/stackabletech/documentation/actions/workflows/deploy.yml[Build and deploy production site] GitHub action.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
= Contributing to documentation
2+
:page-alias: contributing-documentation.adoc
3+
4+
To understand the repositories involved in building the documentation, see xref:project-overview.adoc[]. For an overview of the content structure and the tools involved, see the xref:docs/overview.adoc[].
5+
6+
To contribute, follow the pull request flow outlined in the xref:index.adoc[].
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
= Documentation overview
2+
:figure-caption!:
3+
:antora-docs: https://docs.antora.org/antora/latest/
4+
:antora-playbook: https://docs.antora.org/antora/latest/playbook/
5+
:netlify: https://www.netlify.com/
6+
:diataxis: https://diataxis.fr/
7+
:documentation: https://github.com/stackabletech/documentation
8+
9+
We use {antora-docs}[Antora] to write our user facing documentation,
10+
{netlify}[Netlify] to host it and the {diataxis}[Diátaxis] framework as the guide for the structure of the content.
11+
The main repository for the documentation is the {documentation}[Documentation] repository and
12+
each operator and other tools have a `docs` directory from which content is pulled in.
13+
Have a look at the xref:project-overview.adoc[] to learn about the repositories that are involved in providing the documentation.
14+
15+
== Content structure: Diátaxis
16+
17+
The {diataxis}[Diátaxis] framework is a way to classify documentation content into four groups with distinct use cases.
18+
19+
.Source: https://diataxis.fr/
20+
image::diataxis.png[]
21+
22+
Documentation exists along two axis: _study_ or _work_; _pracital steps_ and _theory_.
23+
Practical learning happens in tutorials, and they are backed up by conceptual background information (theory).
24+
On the work oriented side, documentation should contain exhaustive reference information (every commandline flag, every yaml property, their type and whether they are optional or not etc.) as well as narrow, task oriented guides.
25+
26+
At Stackable we have to document the platform as a whole, individual operators, products and command line tools like `stackablectl`.
27+
28+
**Conceptual** information lives at the platform level in the xref:concepts:index.adoc[] section. Some operators have their own specific concepts such as xref:zookeeper:znodes.adoc[] for ZooKeeper, but most concepts apply to all operators (i.e. roles and role groups, resource management, config overrides).
29+
30+
**Tutorials** (learning oriented guides) exist at the xref:tutorials:index.adoc[top level] but there are also the Getting Started guides for all the individual operators.
31+
Tutorials provide complete set up information and require very little to be there already in contrast to guides (work instead of study oriented) which are more focused on solving specific tasks.
32+
There are also the xref:demos:index.adoc[Demos] which are complete solutions that showcase how the platform integrates different products in a unified, reproducible and transparent way.
33+
34+
**Guides** are also instructional, but focus a narrow topic, something that the user might want to solve in their particular setup.
35+
For example: "How do I set up Kerberos with HDFS?" or "How do I connect Superset to Druid?"
36+
The guide has to account for different use-cases (i.e. the user is using their own pre-existing ZooKeeper instead of our ZooKeeper). This is not the case for tutorials.
37+
Since this kind of information is typically product specific, it is located in the usage guide section of individual operators.
38+
39+
**Reference** information for the Stackable platform entails all the settings and Options in our YAMLs, which we generate.
40+
The reference is found at https://crds.stackable.tech/ and generated from the https://github.com/stackabletech/crddocs[crddocs repository].
41+
42+
=== Style guide
43+
44+
The xref:docs/style-guide.adoc[] contains all the information about the writing style, formatting style, naming documents and more.
45+
46+
== Technical bits: Antora, Netlify
47+
48+
{antora-docs}[Antora] uses a {antora-playbook}[playbook] to build the documentation.
49+
It pulls information from all the individual operators, so their documentation can live in the same repository.
50+
Antora builds a static website which we serve over {netlify}[Netlify].
51+
The web template of the documentation is also custom made and is developed in the https://github.com/stackabletech/documentation-ui[documentation-ui] repository.
52+
53+
For Antora, the https://antora.zulipchat.com/[Antora Zulip chatroom] is a good place to get help (besides the documentation)!
54+
55+
Building the documentation and also the deployment process on Netlify are documented in the https://github.com/stackabletech/documentation/blob/main/README.adoc[README] file of the documentation repository.
56+
57+
== Executable tutorials
58+
59+
The getting started guides for each operator are backed by an executable script from which the documentation includes excerpts.
60+
This has the benefit that the guides are easily tested.
61+
When writing a new guide, please also write it in this way.
62+
Have a look at the existing getting started guides on how to do this.
63+
64+
== Templating
65+
66+
There is a templating mechanism in the docs.
67+
This has been introduced to template in mostly version numbers, so the updating doesn't have to be done by hand.
68+
69+
Every Operator repo has a script `scripts/docs_templating.sh` and a file with templating variables `docs/templating_vars.yaml`.
70+
The script applies the variables to all `.j2` files in the `docs` directory.
71+
72+
This is used for getting started scripts, and in there only for versions of operators and for Helm Chart URLs.
73+
74+
Without this templating mechanism, every release these values would need to be updated by hand (or possibly with a search and replace) with is error prone and time consuming.
75+
76+
== Branch and version structure
77+
78+
All Stackable repositories use https://trunkbaseddevelopment.com/[trunk based development], and so the documentation is also pulled from different release branches.
79+
This means that any changes from the `main` branch that should be published in other versions need to be cherry-picked over into that branch.
80+
81+
Antora recommends using https://docs.antora.org/antora/latest/playbook/content-branches/[branches] to organize different versions of components.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
= Releasing a new documentation version
2+
3+
NOTE: This guide is directed at internal contributors, as an external contributor, you cannot release a new documentation version.
4+
5+
Whenever there is a new Stackable Data Platform release, the documentation is also released with a new version.
6+
This process has been automated with scripts, which are found in the https://github.com/stackabletech/documentation/tree/main/scripts[`scripts`] directory of the documentation repository.
7+
8+
The process consists of two steps:
9+
10+
. Making a new release branch (`make-release-branch.sh`)
11+
. Publishing the new version by modifying the playbooks (`publish-new-version.sh`)
12+
13+
Consult the scripts for details about the required steps, as well as prerequisites.

modules/contributor/pages/docs-style-guide.adoc renamed to modules/contributor/pages/docs/style-guide.adoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
= Documentation style guide
2-
:page-aliases: style_guide.adoc, style-guide.adoc
2+
:page-aliases: style_guide.adoc, style-guide.adoc, docs-style-guide.adoc
33

44
:asciidoc-recommended-practices: https://asciidoctor.org/docs/asciidoc-recommended-practices[AsciiDoc recommended practices]
55
:kubernetes-style-guide: https://kubernetes.io/docs/contribute/style/style-guide/[Kubernetes style guide]

0 commit comments

Comments
 (0)