[ALL CHARTS] publish Prometheus community charts to OCI registry

[scottrigby]

Is your feature request related to a problem ?

This January Helm completed and released full OCI support, moving OCI from experimental to a fully supported feature. Since then users have been able to install Helm charts completely from OCI rather than only from Helm HTTP repos.

However, Prometheus community Helm chart users are not able to pull or install Prometheus charts using Helm's OCI registry support, because they are not yet published to a registry.

Additionally, OCI method can reduce memory usage for users installing charts using an agent in the cloud rather than running the Helm client on their local machine, such as Flux. This is especially true of Helm repos with many charts, such as this Prometheus community charts repo, which contains over 30 charts. In a similarly large Helm repo, Bitnami has recently removed all Helm chart package versions older than six months from their repository because the index file has become too large to support all past versions.

Describe the solution you'd like.

Publish all existing Prometheus community chart package versions to an OCI registry.

• GitHub Packages on ghcr.io would be the easiest way to set this up
• Community Flux charts publish to ghcr.io. We have automation set up for this, which we can emulate
• As an admin of this Prometheus community charts repo, and also as a Helm and Flux maintainer, I can help set this up here as well
• The additional OCI support here will not affect the existing HTTP Helm repo support users currently depend on. It will just give those and other users another option to use OCI as well or as a replacement

Describe alternatives you've considered.

We could publish to some other OCI registry, or not publish at all.
If we don't at all, I think the most likely scenario is someone else will fork and set it up on their own.
I believe it will be better to support this here, so all packages are connected to one Git repo.

Additional context.

I don't imagine any objection here. Mainly opening this issue to track the work and for transparency. Will keep status updated here 💖

[monotek]

No objections as long as we run both in parallel at least for a while :)

We migth use something like: https://github.com/appany/helm-oci-chart-releaser
Just need to check how we can add all exisitng charts, but should be scriptable.
Not sure if the old release timestamp can be used though.

[scottrigby]

@monotek 👋

No objections as long as we run both in parallel at least for a while :)

Agreed 👍

We migth use something like: https://github.com/appany/helm-oci-chart-releaser

had a look at that action. It requires adding config for each chart, which I think is not ideal. But even less ideal is it requires updating action config every time you update a chart version - individually inputting each new chart version of each chart in the workflow config. I think that won't fit the needs of this repo.

Just need to check how we can add all existing charts, but should be scriptable.

Yes adding all existing Prometheus community Helm package versions can definitely be scripted. We did a similar thing for the now archived helm/stable and helm/incubator repos using a tool I wrote (https://github.com/scottrigby/helm-adopt-package-history). That tool currently only supports HTTP Helm repos, but I think we can pretty easily add OCI push support as well. Or really whatever tool works in a foolproof and verifiable way.

Not sure if the old release timestamp can be used though.

This can be done for images with OCI metadata. Will have to look into the best way to do this with OCI artifacts according to distribution spec. Let's just make sure this is considered before copying previous chart package versions to the OCI registry.

[monotek]

had a look at that action. It requires adding config for each chart, which I think is not ideal. But even less ideal is it requires updating action config every time you update a chart version - individually inputting each new chart version of each chart in the workflow config. I think that won't fit the needs of this repo.

My favoutrite solution would also be that the ct / chart releaser action would be extendet to support oci releases too.
Not sure if it's worth to wait. There are at least issues available discussing it (helm/chart-releaser-action#107 & helm/chart-releaser#183).

[jkroepke]

Does someone not that all current releases are failing to ghcr.io with 403 forbidden?

[onedr0p]

Yea, it looks like the new versions of OCI charts have stopped being released.

[monotek]

@scottrigby
i rather would remove oci releases now as it does not work reliable.

[geowalrus4gh]

In this world of OCI registries(for Helm charts too), we are eagerly waiting for this issue to be closed.

[jkroepke]

Bitnami recently changes from the legacy repo format to OCI repository. See: https://blog.bitnami.com/2023/04/httpsblog.bitnami.com202304bitnami-helm-charts-now-oci.html

Since we are using chart-releaser for releasing charts, we should not expect that chart-releaser gets OCI support soon.

The FluxCD Maintainers are using a shell script for years to publish the charts on ghcr.io. See: https://github.com/fluxcd-community/helm-charts/pull/94/files

Is that something what we can look into it? @monotek @scottrigby

[monotek]

If we find a reliable way to use OCI releases i'm open to activate it again:
https://github.com/prometheus-community/helm-charts/blob/main/.github/workflows/release.yaml#L44C7-L70
I would keep it deactivated if CI is failing constantly beacause of it again.

[jkroepke]

What are the issues? Sadly, the build logs from #2719 are not available.

Can we separate OCI Push and cosign? For example adding cosign after having stable OCI pushes?

Would it be possible to add continue-on-error to observe the errors while not blocking the CI?

[onedr0p]

@monotek maybe this could be reactivated again since the logs are gone?

[jkroepke]

And put the continue-on-error on the action that we are able to observe the error while not blocking us.

[monotek]

pr welcome

[jkroepke]

@monotek #3433 opened

[monotek]

Push failed with 403 :/
https://github.com/prometheus-community/helm-charts/actions/runs/5088616788/jobs/9145245233

[jkroepke]

@monotek

Can you check, if github actions is allowed to publish into the packages? You have to go to https://github.com/prometheus-community/helm-charts/pkgs/container/charts%2Fprometheus-rabbitmq-exporter and then Packages settings on the right:

On my repository, the permissions was granted automatically for actions, because the actions initially create the package on ghcr. This behavior is documented here. Could it be possible that some of GHCR repositories are initially created manually for testing reasons? In that case, it could be possible that actions doesn't habe any access.

Is it possible to nuke all the current package chart repositories and let them create by actions only? That should just works, if I'm reading the docs correctly.

[monotek]

Hmm... most of the packages were missing the setting.
Did it manually now.
Hopefully it will work automatically for new charts.

Edit:
Seems so :)
https://github.com/prometheus-community/helm-charts/actions/runs/5090521788/jobs/9149445718

[jkroepke]

Lets observse this for three weeks. If we dont have any issues, i would like to change the level from warning to error.

[monotek]

Meanwhile i'll write a script which pushes all existing charts to ghcr....

[onedr0p]

@monotek if/when you do that I can out using them. No rush, I doubt it's going to be a fun task to work on :)

[monotek]

It's already uploading....

[jkroepke]

Please validate, in case you create new charts, check the settings that I mentions.

[monotek]

prometheus-consul-exporter package seems to have some issue i can't fix.
The "Package settings" button is missing so i guess it's some permission issue.

Upload also fails.

Error: unexpected status from POST request to https://ghcr.io/v2/prometheus-community/charts/prometheus-consul-exporter/blobs/uploads/: 403 Forbidden

@scottrigby
Could you please have a look?

https://github.com/prometheus-community/helm-charts/pkgs/container/charts%2Fprometheus-consul-exporter

[onedr0p]

Would it be possible to delete that one package and try it again?

[monotek]

I can't. Need organisation admin like @scottrigby to do it.

[jkroepke]

@monotek No clue, but scottrigby seems not like an org admin to me.

Looking at https://github.com/orgs/prometheus-community/people?query=role%3Aowner @SuperQ can help us here.

[monotek]

True, sorry.
I thought as he started to work on it initialy he had the permissions too :)

[jkroepke]

@monotek what did you think about switching the Pipeline from Warning to Error now? As next step, chaging the README.md of all charts to use the oci endpoint instead the classical one?

[monotek]

Not sure if it should be the default.
Maybe add both ways?

OCI of prometheus-consul-exporter is still broken.

[onedr0p]

Is there anyway (slack, discord, IRC etc..) to directly reach out to @SuperQ to get him notified of the issue with that chart?

[SuperQ]

Sorry, I've been busy and not had time to look at this issue. I'll read over it today and see what I can do.

[SuperQ]

Ok, I think I fixed the prometheus-consul-exporter package access/permissions. Let me know if that fixes the issue and if there are any other packages to fix.

[monotek]

@SuperQ
thanks a lot, it works :)
consul exporter packages are available now in the oci registry too.

@jkroepke
we could move forward to adjust the charts readmes....

[jkroepke]

Can it all in one PR or each chart separtly?

[monotek]

The action will not work for multiple charts, so it has to be doen separately.

But it might be enoguht to just add it to the repo readme for now:

https://github.com/prometheus-community/helm-charts#readme