Old helm chart releases are missing

[no-12]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

With the release of the awx-operator helm chart 0.29.0 all old releases are missing from https://ansible.github.io/awx-operator/index.yaml.

AWX Operator version

0.29.0

AWX version

any

Kubernetes platform

kubernetes

Kubernetes/Platform version

any

Modifications

no

Steps to reproduce

helm repo add awx-operator https://ansible.github.io/awx-operator/
helm install my-awx-operator awx-operator/awx-operator --version 0.28.0

Expected results

helm should install version 0.28.0 of awx-operator

Actual results

Error: INSTALLATION FAILED: failed to download "awx-operator/awx-operator" at version "0.28.0"

Additional information

No response

Operator Logs

No response

[PaulVerhoeven1]

Can confirm that this is an issue, i only see version 0.29.0. Also this was also an issue in 0.28.0. Maybe this is an issue since the beginning (when we started to support Helm charts) ?

[djyasin]

@rooftopcellist would you mind reading over this issue and letting us know what you think?

[janorn]

I have been playing around with this. It seems to be some kind of issue with chart-releaser. It will replace all entries in the index.yaml file with the one just been packed in the Makefile. If one downloads all releases and places them in the same directory when running cr index you get a complete index.yaml.

I have done a workaround to resolve this for myself at https://janorn.github.io/helmrepo-awx/

https://github.com/janorn/helmrepo-awx/blob/1b7e19f88c902632d721e9112d8137ceee8018ce/index.yaml

[janorn]

Perhaps it could be as simple as to copy in the index.yaml file into ./charts during the buildprocess?!

awx-operator/Makefile

Lines 352 to 375 in 1892b8f

	.PHONY: helm-package
	helm-package: cr helm-chart
	@echo "== CHART RELEASER (package) =="
	$(CR) package ./charts/awx-operator
	# The actual release happens in ansible/helm-release.yml
	# until https://github.com/helm/chart-releaser/issues/122 happens
	.PHONY: helm-index
	helm-index: cr helm-chart
	@echo "== CHART RELEASER (httpsorigin) =="
	git remote add httpsorigin "https://github.com/$(CHART_OWNER)/$(CHART_REPO).git"
	git fetch httpsorigin
	@echo "== CHART RELEASER (index) =="
	$(CR) index \
	--owner "$(CHART_OWNER)" \
	--git-repo "$(CHART_REPO)" \
	--token "$(CR_TOKEN)" \
	--pages-branch "$(CHART_BRANCH)" \
	--index-path "./charts/$(CHART_INDEX)" \
	--charts-repo "https://$(CHART_OWNER).github.io/$(CHART_REPO)/$(CHART_INDEX)" \
	--remote httpsorigin \
	--release-name-template="{{ .Version }}" \
	--push

[janorn]

Add this perhaps between package and index:

        @echo "== CHART FETCH previous index =="
        git show $(CHART_BRANCH):$(CHART_INDEX) > ./charts/$(CHART_INDEX)

Have not been able to test this properly though.

[janorn]

Perhaps @miles-w-3 can comment on this?

[miles-w-3]

I can take a more in-depth look into how chart-releaser works, but at a surface level I'd agree that it's an issue with the index yaml. I'll follow up once I've done some more research

[janorn]

Looking at chart-releaser this issues stands out as interesting:
helm/chart-releaser#133

[janorn]

This didn't seem to work?! New index.yaml still only has one release. Where could one find the output from the release run?

[janorn]

Looks like this is missplaced. I guess I need help placing this properly in the Makefile.

awx-operator/Makefile

Lines 357 to 358 in 17eb9cf

	# List all tags oldest to newest.
	TAGS := $(shell git tag -l --sort=creatordate)

It only seem to pickup one tag.

 "# Download all old releases", "cd .cr-release-packages;\\", "for tag in 0.30.0; do\\", 

[lerminou]

Hi, can you re-open this issue please ? the index.yaml is still containing the latest version @rooftopcellist @miles-w-3

[janorn]

There is already a PR to fix this #1075

[miles-w-3]

@janorn or @rooftopcellist in case you want to pick up on this, it does seem possible that we could use chart releaser to make our releases way easier, it will just take some work to figure out the following things:

• the native chart releaser role requires the chart to already be built and present in the chart directory, which means we'd need to build the chart as a part of the release tag branch. Might be better to have the release branch build the chart in some other directory, and build the chart there during release.
• This means back-porting the previous releases to work automatically will be difficult, we would have to branch off of all the releases and build the chart.
  I had partial success with one build, the nice thing is that it automatically hooks in with the releaser workflow to create a chart release, though we'd want to rename the tag https://github.com/miles-w-3/awx-operator/releases/tag/awx-operator-0.23.0-99-g99ee596

There's just a lot of overhead involved in testing this on a fork, it would be nice if someone had the time to automate the creation of those requisite branches in a forked repo, and we could try using the simple chart releaser role to have it generate all of the builds. I might try to do a PoC with a dummy chart when I have the time to nail down the complete process of using it, since it's not that straightforwartd given the old tags. This may be a lot of short term work, but it would pay off in the long-term if we can just use the simple role.

[miles-w-3]

Updating to reinforce how easy this is once it's working. I have a sandbox repo with a dummy helm chart, all it took was me adding the action which calls the cr, and every time I make a change to the master branch and run the pipeline, it automatically updates the index file. The great thing is that it pulls straight from the releases, so you don't have to manage the tars. I would highly recommend adopting this.

❯ helm search repo sandbox
NAME         	CHART VERSION	APP VERSION	DESCRIPTION
sandbox/basic	0.2.0        	1.17.0     	A Helm chart for Kubernetes

[janorn]

Just curious what happens if you have a release without a chart? awx-operator has a lot releases without any charts.

[miles-w-3]

Well, any change to the operator resources result in a change to the helm chart when it is built. The key problem is that chart-releaser relies on the chart to be built in order to compare the "diff". One option would be to build the chart and commit it to the commit tags, but that's not ideal because you have more drift from the main branch. Another option to consider would be to spin off the helm chart to its own repo, with a "depend-bot" which builds, bumps, and commits changes based on the operator whenever they are tagged. I think this would be interesting to discuss because, unlike other helm apps, the helm chart is not core to the development of the operator and is put together as an afterthought.