- Developing Kiali Operator
- How To Change Resources or Configuration
- Developer Checklists When Changing Resources or Configuration
- Maintain Backward Compatibility
- Run Molecule Tests
- Are You Altering a Kiali Operator Resource?
- Are You Altering a Kiali Server Resource?
- Are You Altering a Kiali Server Role Permission that Applies to Resources in All Accessible Namespaces?
- Are You Altering a Kiali Server Role Permission that Applies to Resources Only in the Control Plane Namespace?
- Are You Altering a Kiali Server Configuration Setting?
- Are You Altering a Kiali Monitoring Dashboard Template?
- Do You Need To Backport Changes To Support Older Versions?
- Are You Adding Support For a New Ansible Role Version?
- Are You Removing Support For an Old Ansible Role Version?
- Developer Checklists When Changing Resources or Configuration
The Kiali Operator source lives in its own git repository, but because it is necessary for Kiali developers to use it, the Kiali Operator repository should be accessible in the Kiali git repo via a softlink in the "operator" directory or just cloned right inside the "operator" directory.
For example:
# Clone Kiali Operator in a local git repository location
cd $HOME/source
git clone git@github.com:kiali/kiali-operator.git
# Go to your existing Kiali local git repository
GOPATH=<your GO path to the Kiali project - something like $HOME/source/kiali>
cd $GOPATH/src/github.com/kiali/kiali
# Provide a link to your Kiali Operator local git repository
ln -s $HOME/source/kiali-operator operator
Note
|
If you want to clone directly into your kiali/kiali operator directory, you run something like: git clone git@github.com:kiali/kiali-operator.git $GOPATH/src/github.com/kiali/kiali/operator . However, if you plan on making code changes to the operator, it is highly recommended you use the softlink approach mentioned above. See the WARNING below for an explanation for this recommendation.
|
Warning
|
If you git cloned the operator repo directly into your kiali/kiali git repo’s operator/ directory, and you need to checkout an older branch of your kiali git repo, then you must make sure you push any operator changes to your remote fork before you checkout that older kiali repo branch, otherwise you will lose those operator changes. This is because the operator directory will get replaced with the old kiali repo branch’s original operator content. When you then checkout the master branch again, your operator directory will be deleted. At this point, you will need to either create your softlink again or re-clone the git repo. It is for this reason that it is recommended that the operator directory be a softlink that points to an external directory where you have git cloned the operator repo (with a softlink, your operator code changes remain intact - you merely have to recreate the softlink again).
|
This now makes it easy for Kiali developers to use the operator when working directly in the Kiali git repo. The Kiali git repo has all the necessary make
targets to build, deploy, and test the Kiali Operator along with Kiali itself. This is very similiar to how things worked prior to moving the operator code to a separate git repo.
The Kiali Operator git repo itself has a very small Makefile whose sole job is to build and push the Kiali Operator container image. This Makefile is used mainly for CI and release automation; developers typically will not need to use it.
Assuming you set up your local git repositories as explained in the above section Setting up your local git repositories, then you can develop Kiali Operator features and bug fixes while working in the local Kiali repo directory.
If, however, a developer needs to make changes to the operator, because the Kiali Operator code really lives in a separate repo, the developer will need to create PRs to the Kiali Operator git repo (just like any other git repo you want to commit to). Simply use the same basic workflow and commands that you are used to using already when creating PRs - just remember you need to create a PR from within the Kiali local repo (if changes were made to it) and you need to create another PR from within the "operator" local repo (if changes were made to the operator code).
The Kiali git repo has a Makefile and a set of Makefile targets that are used in conjunction with the Kiali Operator so you can install and run the operator built from your local git clones. They use the Kiali Operator helm chart to install the operator, so you must clone the helm-chart git repo in the proper place as well.
To install the operator that contains your local changes, run make operator-create
. If you are using minikube, you will need to set the environment variable CLUSTER_TYPE=minikube
and you need to also set MINIKUBE_PROFILE
if your minikube profile name is not minikube
. At this point you can create your own Kiali CR to trigger the operator to install Kiali, or use make kiali-create
to install a sample Kiali CR for you. Similarly, make ossmconsole-create
will create an OSSMConsole CR to install the OSSMC OpenShift Console plugin.
To remove the operator (and any CRs that may have also been created), run make operator-delete
.
If you have a cluster with OLM installed (e.g. OpenShift), you can test the Kiali OLM metadata by installing the Kiali operator using OLM via make.
If you do not have OLM installed, you can easily install it. First determine what OLM release you want to install. Then simply run the command curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${OLM_VERSION}/install.sh | bash -s ${OLM_VERSION}
. If you are running minikube, all you need to do is execute the command hack/k8s-minikube.sh olm
and the hack script will install the latest OLM for you.
Once OLM is installed, you can install the Kiali Operator (and all the required resources such as the OLM catalog source and subscription) via make olm-operator-create
.
Once the operator is running, you can install a Kiali CR normally. One way to do this is to first create a kiali-operator namespace and run the kiali-create
make target: kubectl create ns kiali-operator && make kiali-create
. Similarly with the ossmconsole-create
target for installing OSSMC.
To remove the Kiali operator and all the other OLM resources that came with it (including the CRs if you created some via the kiali-create
target or the ossmconsole-create
target), run make olm-operator-delete
.
Note
|
By default, you will test with the latest version of the "kiali-ossm" metadata. You can specify a different version by setting the BUNDLE_VERSION env var on the make command line. If the OC env var points to an oc binary, the metadata from the kiali-community will be used; otherwise, the metadata from the kiali-upstream will be used. If you want to do this, you also must set the env var OLM_BUNDLE_PACKAGE to kiali on the make command line.
|
Sometimes you made simple changes to the operator playbook and roles that can be quickly tested by running the Ansible playbook on your local machine. This saves time because it doesn’t require you to install the operator directly in your cluster. To do this, you must have Ansible installed (e.g. ansible-playbook
must be in your $PATH). You can run the operator playbook via make run-operator-playbook-kiali
(to test the Kiali playbooks) or make run-operator-playbook-ossmconsole
(to test the OSSMC playbooks). run-operator-playbook-kiali
will run both the kiali-deploy and kiali-remove playbooks so the operator will install and then immediately uninstall a Kiali Server. run-operator-playbook-ossmconsole
will run both the ossmconsole-deploy and ossmconsole-remove playbooks so the operator will install and then immediately uninstall a OSSMC plugin. The operator playbooks are configured via the files found in the dev-playbook-config directory.
In order to use these "run-operator-playbook" targets, you must have Python3 in your PATH (having an alias is not enough).
You also must make sure your local Python and Ansible environment matches as closely as possible to the environment of the Kiali operator. To find out the different versions of software within the Kiali operator image, run the following:
-
To get the Python version:
podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest python3 --version
-
To get the Python libraries (
kubernetes
is an important one to look at):podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest pip3 list
-
To get the Ansible version:
podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest ansible --version
To get started, try to run python -m pip install --user --upgrade -r operator/molecule/requirements.txt
to get Python3 libraries installed. If you need to, you can install specific versions of the libraries like this: python -m pip install --user kubernetes==12.0.1
.
You also must get the required Ansible collections installed. To find out the different versions of Ansible collections within the Kiali operator image, run the following:
podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest ansible-galaxy collection list
Note
|
Older versions of the operator base image did not support the collection list option to ansible-galaxy . For those older images, you can run this script to determine the versions of the collections installed in the image: for m in $(podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest find .ansible/collections/ansible_collections -name MANIFEST.json); do manifest=$(echo -n ${m} | tr --delete '\r'); echo -n "${manifest}-→"; podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest cat "${manifest}" | jq .collection_info.version; done
|
To install these, run ansible-galaxy collection install -r operator/requirements.yml --force-with-deps
The previous section tells you how to run the Ansible playbook directly on your local machine. If you want to run the actual operator on your local machine in the same manner in which it runs inside the cluster (that is, within the ansible-operator
shell process) then use the make run-operator
make target. This will run the ansible-operator executable and point it to the operator’s Ansible playbooks and roles. This operator will watch for Kiali CRs and OSSMConsole CRs in the cluster - when it sees one, it will process it just as if it was running in the cluster. This will allow you to test the Ansible operator infrastructure as well as the operator’s Ansible playbooks themselves.
If the ansible.cfg has callbacks_enabled = profile_tasks
specified (or the deprecated callback_whitelist
for older Ansible installations), a profiler is run and its report is dumped at the end of each reconciliation run. You can tell the operator to use such a configuration by setting the ANSIBLE_CONFIG
environment variable to /opt/ansible/ansible-profiler.cfg
.
The profiler report is usually good enough to start tracking down performance issues. But it may be difficult to see performance issues over tasks that loop because the profiler report will show duplicate entries for each loop invocation. In that case, copy the profiler report to a file and pipe it to a script with the following content to provide a more cumulative report:
awk -F~ ' ##Setting field separator as tilde here.
{
val=$2; ##Creating a variable named val whose value is 2nd field of current line.
$2="@"; ##Setting value of 2nd field as @ here to keep all lines same(to create index for array a).
a[$0]+=val ##Creating an array named a whose index is the current line and its value is the new sum
}
!b[$0]++{ ##Checking if array b, whose index is the current line, has a value of NULL; if so do following.
c[++count]=$0} ##Creating an array named c whose index is variable count increasing value with 1 and value is line.
END{ ##Starting END block of awk code here.
for(i=1;i<=count;i++){ ##Starting a for loop whose value starts from 1 to till value of count variable.
sub("@",a[c[i]],c[i]); ##Substituting @ in value of array c(which is actually lines value) with value of SUMMED $2.
print c[i]} ##Printing newly value of array c where $2 is now replaced with its actual value.
}' OFS=\~ <(cat - | sed 's/\(.*\) -\+ \(.*\)s/\1~\2/') | sort -n -t '~' -k 2 -r | column -s~ -t
Note
|
This section documents how the Kiali Operator git repo was originally created. This only had to be performed one time and one time only. Developers and consumers of Kiali and Kiali Operator do not need to know any of this, so you can skip this section. This is here just for historical purposes. |
The new Kiali Operator git repo was created with a empty history. Once a repo was created, it then needed to be populated with the original operator code from the Kiali git repo while retaining all history. Here is how this was done:
-
Go to a local git repository of the Kiali repo.
-
Create a new branch to be used to populate the new Kiali Operator git repo:
git checkout -b split-operator-master origin/master
-
Extract only the operator files retaining all history:
git filter-branch --subdirectory-filter operator
-
Push the operator files to the Kiali Operator git repo’s master branch:
git push --force -u git@github.com:kiali/kiali-operator.git split-operator-master:master
Analogous steps were done for the other branches v1.0
and v1.12
.
The operator
directory in the Kiali git repository is no longer used - it thus is removed and ignored, leaving room for developers to link in their working copy of their local Kiali Operator git repository.
-
Go to a local git repository of the Kiali repo.
-
Create a branch to create a PR with:
git checkout -b remove-operator origin/master
-
Delete the
operator
directory and ignore it to get it out of the way permanently:rm -rf operator && echo operator >> .gitignore
-
Commit the change:
git commit -am "Remove and ignore the operator directory"
-
Push and create a PR from this remove-operator branch, then merge the PR to fully remove the operator from master.
If you need to alter Kubernetes resources for either the Kiali Server or Operator, or if you need to add, remove, or modify a configuration setting, here are the places you need to check and possibly edit. You must make changes to all of these in order to support the different install mechanisms.
The Operator’s own resources are deployed either via OLM or the Operator Helm Chart. You must make changes to support them all. Today there are three sets of OLM metadata maintained in the kiali-operator repository - we call them the "golden copies" because they are the source of truth for the OperatorHub.io upstream operator, community operator, and the Red Hat OSSM Operator. The Operator Helm Chart templates are here.
The Kiali server resources are deployed by the Kiali Operator Ansible roles or by the Kiali Server Helm Chart. There are generic Kubernetes templates as well as OpenShift templates for the default (i.e. current) release as well as select previous releases.
The configuration settings for the Kiali CR and Kiali ConfigMap are documented in the kiali_cr.yaml example file and are defined in the Kiali Operator Ansible roles and in the Helm Charts, described below in more detail.
The default values for all configuration settings are found in the Kiali Operator defaults/main.yml file (there are analogous defaults/main.yml
files for the select previous releases) and in the Kiali Operator Helm Chart and Kiali Server Helm Chart.
If new Kiali Server resources are added or removed, you must make appropriate changes to the Kiali Operator Remove Role as well.
Below are checklists that developers should consult when making various kinds of changes.
Note
|
Remember to maintain backward compatiblity for all supported versions. Do not make any changes if those changes would break the older supported versions. For example, if you are removing permissions from the operator roles/rolebindings within the golden copies, and if those permissions are required by the older supported versions such as v1.24 , then you must not make those changes. Only when those older versions are no longer supported can you make those changes. In this case, create a github issue as a reminder to make the changes at the appropriate time in the future when those versions are no longer supported.
|
Warning
|
This may not work if you have SELinux enabled. If you get permission errors, disable SELinux via setenforce 0 and try again.
|
When making any kind of change, you should also make the necessary changes to the Molecule tests to ensure they still pass. You can easily run the Molecule tests on your local box using Minikube by running the ci-minikube-molecule-tests.sh hack script. If you want to run just a few or one Molecule test rather than the whole suite, you can run something like this (omit CLUSTER_TYPE
if running the test on OpenShift; you can omit MINIKUBE_PROFILE
if your profile is the default of minikube
).
CLUSTER_TYPE=minikube MINIKUBE_PROFILE=ci MOLECULE_USE_DEV_IMAGES=true MOLECULE_SCENARIO="roles-test" make clean build test cluster-push molecule-test
where roles-test
is the Molecule test(s) you want to run. The value of MOLECULE_SCENARIO
is a space-separated list of Molecule tests - the full list of Molecule tests are found here. There are other environment variables you can set (such as MOLECULE_DEBUG
for turning on debug messages) - see the make/Makefile.molecule.mk file in the kiali/kiali repo.
Note
|
If you have never run the Molecule tests before, the Molecule container image will be built for you automatically. If you need to rebuild the image for some reason, run FORCE_MOLECULE_BUILD=true make molecule-build .
|
-
❏ Update the golden copy of the kiali-upstream CSV metadata (* see note above)
-
❏ Update the golden copy of the kiali-community CSV metadata (* see note above)
-
❏ Update the golden copy of the kiali-ossm CSV metadata (* see note above)
-
❏ Update the Operator Helm Chart templates
-
❏ Update the Operator Kubernetes templates
-
❏ Update the Operator OpenShift templates
-
❏ Check if you need to modify the Operator remove role
-
❏ Update the Server Helm Chart templates
Are You Altering a Kiali Server Role Permission that Applies to Resources in All Accessible Namespaces?
-
❏ Update the golden copy of the kiali-upstream CSV metadata (* see note above)
-
❏ Update the golden copy of the kiali-community CSV metadata (* see note above)
-
❏ Update the golden copy of the kiali-ossm CSV metadata (* see note above)
-
❏ Update the Operator Kubernetes role.yaml
-
❏ Update the Operator Kubernetes role-viewer.yaml
-
❏ Update the Operator OpenShift role.yaml
-
❏ Update the Operator OpenShift role-viewer.yaml
-
❏ Update the Operator Helm Chart clusterrole.yaml
-
❏ Update the Server Helm Chart role.yaml
-
❏ Update the Server Helm Chart role-viewer.yaml
Are You Altering a Kiali Server Role Permission that Applies to Resources Only in the Control Plane Namespace?
-
❏ Update the golden copy of the kiali-upstream CSV metadata (* see note above)
-
❏ Update the golden copy of the kiali-community CSV metadata (* see note above)
-
❏ Update the golden copy of the kiali-ossm CSV metadata (* see note above)
-
❏ Update the Operator Helm Chart clusterrole.yaml
-
❏ Set the default value in Operator default/main.yaml
-
❏ If adding a new top-level group or setting, add it to Operator vars/main.yaml
-
❏ Document the new value in the CRD schema (and put the default values in the example CR). Run
make validate-cr
in your local kiali-operator repo to ensure your changes are valid. -
❏ Only if appropriate (usually it is not) set the value in the
cr.spec
section of Operator Helm Chart values.yaml -
❏ Set the default value in the Server Helm Chart values.yaml
-
❏ Ensure each group of settings are alphabetically sorted in all the files where the new setting was added.
-
❏ If appropriate, add some tests to the Molecule config-values-test
-
❏ Modify the built-in templates used by the Server
-
❏ Add the new role directory by copying the default role and naming the new role with the form
vX.Y
. -
❏ Add the new RELATED_IMAGE reference for the version being added to the the CSV
-
❏ Add the new version to kiali-default-supported-images.yml
-
❏ Add the new version to ossmconsole-default-supported-images.yml
-
❏ Delete the role directory (one of these)
-
❏ Remove the RELATED_IMAGE reference for the version being removed from the CSV
-
❏ Remove the version from kiali-default-supported-images.yml
-
❏ Remove the version from ossmconsole-default-supported-images.yml