🔊 New container images security assessments

[carrodher]

On December 10th, 2024, we released a minor version for every Helm chart in the Bitnami catalog. This minor version introduced a new security mechanism to detect the usage of non-standard containers while installing or upgrading Helm chart releases.

Replacing the original containers that are shipped with the charts will result in installation errors like the one below:

⚠ ERROR: Original containers have been substituted for unrecognized ones. Deploying this chart with non-standard containers is likely to cause degraded security and performance, broken chart features, and missing environment variables.

Unrecognized images:

- [REGISTRY]/[REPOSITORY]:[TAG]
(...)

FAQ

Why did Bitnami introduce this mechanism?

There are several reasons to introduce this change:

1. Bitnami charts are designed, tested, and validated on multiple platforms using a specific set of Bitnami and Tanzu Application Catalog containers. Therefore, Bitnami cannot guarantee the same quality standards levels when the original container images are replaced.
2. Non-standard container images are likely to cause degraded security and performance, broken chart features, and missing environment variables.
3. Incapacity to provide efficient support for non-standard container images given the Bitnami support team's limited bandwidth.

Can I use different image tags?

Yes, replacing default image tags is considered a valid use case. For instance, a user might be interested in using the Bitnami PostgreSQL chart using Bitnami PostgreSQL 15.x container images (note the chart uses 16.x images by default).

That said, it is expected that you see some warnings on the chart notes given that the specific chart & image combination wasn't tested and validated by Bitnami on any platform.

Can I skip the security verification?

Yes, we introduced a new global parameter on every chart that can be used for that purpose, see:

Name	Description	Value
global.security.allowInsecureImages	By default, this chart verifies that the original container images that were designed, tested, and validated are used. This option makes the chart skip the verification step and proceed.	false

At your own risk, you bypass the verification specifying the parameter adding the --set global.security.allowInsecureImages=true argument to helm install or helm upgrade commands or, alternatively, adding the block below to your values YAML file:

global: 
  security: 
    allowInsecureImages: true

Should I expect any issues upgrading my existing chart releases?

No issue is expected if you upgrade your existing releases using original container images. If you replace them, you'll have to skip the security verification.

Should I expect errors consuming Bitnami charts from mirror registries?

Yes. It is expected that you face installation errors while consuming Bitnami chart from mirror registries given the security mechanism will detect original images are not used. To continue using mirror registries, you'll have to skip the security verification.

I built my custom image extending a Bitnami container image, should I expect errors?

Yes. Custom images, even if they're extensions of original Bitnami container images, would be detected by the security mechanism and it is expected to face installation errors. To use your custom image, you'll have to skip the security verification at your own risk.

[perfectra1n]

Bitnami's policy for versioning Helm charts is to release a major version for anything that could break the service upgrade

A configuration change in the application (e.g., configuration parameter) that breaks compatibility with the previously deployed configuration.

This change breaks the service upgrade, as the the Helm chart's templates no longer complete evaluation when using values that previously compatible with the deployed configuration. This isn't a minor version update of the Helm chart.

[ggogel]

We made this change concurrently with the release of our commercial product, Bitnami Premium.

It appears that this breaking change was introduced to align with, and potentially promote, your new commercial offering rather than serving a purely technical or user-driven purpose.

We have seen multiple instances where users substituting non-Bitnami or modified images experienced serious security problems

Claiming that this change enhances security is not entirely accurate. Using the default image does not inherently guarantee the integrity of the deployed image, as true verification can only be achieved by validating its hash. Likewise, using a different image does not necessarily result in a less secure deployment. I therefore see the name of this parameter as counterintuitive and misleading.

[viceice]

what about the official public.ecr.aws/bitnami/* images? they should also be accepted. 🤔

[seagyn]

If I enable allowInsecureImages for a pull-through cache, does that imply Bitnami images are inherently insecure?

[abhilash-ghosh]

we released a minor version for every Helm chart in the Bitnami catalog.

I'd suggest to follow the sem version policy strictly. If this is a minor version release, the default value of global.security.allowInsecureImages is set to true, isn't it?

[guyguy333]

Helper checks image URL, breaking all mirrors and pull-through caches, even if used image is the official one.
Instead, helper should checks SHA256 of deployed image. Issue is not origin of image but its content.

[Dreathorian]

maybe instead of declaring everything except bitnami as insecure maybe let us decide which sources are trustworthy?
single floodgate on/off mechanisms like global.security.allowInsecureImages are always bad solutions and the opposite of secure, because they are mandatory to use now, please just gives us control of some granularity (deno does the same btw)

[SF97]

There seems to be no interest from Bitnami to change anything... This is obviously a push for Bitnami premium. It's obviously fine that you want to profit from your work, but stating that non-bitnami repositories are, by default, unsafe and a security flaw is not only wrong but ethically questionable. With no changes to this policy don't expect me to buy anything from Bitnami. I know I'm just a drop in the bucket, but since our words are not being listened to, maybe our money will.

[cradules]

This is the first step in monetizing. We already have examples of this strategy. Open source has started to be more like a temporary mask until they hook you, and after that, you are forced to pay, already hooked.

I have nothing against making money, but don't use the principle of "open source" and then you force users to use their card.

More and more I want to shift to the old school where open source was driven 100% by communities and not companies. I see big companies getting into the open source (and I was happy about it) but I was so naive! I think they want to control the open source tro' all kinds of "best practices and security"!

Risks will always be. Every time you step out of your house you increase the risk of happening something to you. This means I don't need to step out of my house ?

Shame on you!

No worries I will build my own charts, I don't need yours! Like this, I will keep my freedom!

[portswigger-tim]

It's all okay guys... just add --set global.security.allowInsecureImages=true to your Helm deployment pipelines 🤣 - no need for all the rage.

[cglacet]

I don't quite understand, why is bitnami marking its own images as unsafe to use?

I was using oci://registry-1.docker.io/bitnamicharts/postgresql but apparently its unsafe now because it uses docker.io/bitnami/bitnami-shell:10-debian-10-r431 internally. I have no idea what that means.

I only use this for local dev so its not too much of a problem, but its weird.

[efbicief]

The supply chain integrity check only affects an optional property of the chart for users who are overwriting the path of the images. It does not affect the previously deployed application during the upgrade; the error is thrown when evaluating the template.

I work in an environment where all images, including in this case official Bitnami images, get pulled through a proxy. This change has resulted in applications which break when they get moved or recreated (in my environment, these operations result in the templates getting re-evaluated). This is not acceptable for a "minor" change (clearly violates semver - MINOR version when you add functionality in a backward compatible manner, this breaks config which previously worked so is not backward compatible), and really reduces my trust in Bitnami.

If this supply chain integrity check were to be implemented properly (by evaluating the pulled image, eg. comparing a hash of the image to what is expected), rather than just checking where the image gets pulled from, this would have caused no problems.
As other commenters have said, I understand that Bitnami wants to monetise through Bitnami Premium, but please don't needlessly break my deployments next time you want to push your product.

[davgia]

This is even more relevant now that docker-hub has changed policies. For unauthenticated users there will be a limit of 10 pull/h. Both Bitnami's images and charts are hosted on docker-hub now. So a pull-through cache is practically a must. This means that I will need to set global.security.allowInsecureImages everywhere probably defeating the whole purpose of it.

[davgia]

I just find out something that invalidates my previous statement. bitnami and bitnamicharts are a DockerHub Verified Publishers. This means that DockerHub pull rate limiting is not applied for their images and charts. So it's a problem no more (at least for now). That said, I will go ahead and setup a pull-through cache with Harbor anyway just to be safe if docker changes its policy in the future.

[stefanprodan]

Funny how Bitnami's own images are considered insecure

ERROR: Original containers have been substituted for unrecognized ones. Deploying this chart with non-standard containers is likely to cause degraded security and performance, broken chart features, and missing environment variables.

Unrecognized images:
- public.ecr.aws/bitnami/metrics-server:0.7.2-debian-12-r18

[jessesimpson36]

One side-effect of this that I don't think has been considered are usecases where the bitnami charts are installed as subcharts to a bigger chart. If global.security.allowInsecureImages is enabled, it doesn't just affect bitnami charts, it would affect all charts installed that listen on that option. Though I admit, I do not know of a chart other than bitnami's that listens for the global.security.allowInsecureImages option.

I personally would prefer if this were a setting in the chart-level such as allowInsecureImages, or if this was in a subsection of global such as global.bitnami.security.allowInsecureImages. (because other companies charts would not add options under global.bitnami)

[yardenshoham]

What's the point of this check if I can mirror the registry in the container runtime configuration?

[franklouwers]

I just find out something that invalidates my previous statement. bitnami and bitnamicharts are a DockerHub Verified Publishers. This means that DockerHub pull rate limiting is not applied for their images and charts. So it's a problem no more (at least for now). That said, I will go ahead and setup a pull-through cache with Harbor anyway just to be safe if docker changes its policy in the future.

@davgia I don't find any mention of that on https://www.docker.com/blog/revisiting-docker-hub-policies-prioritizing-developer-experience/ but also seems to contradict the following quote from #30853

Beginning on December 16th, 2024, all free Bitnami Helm charts and containers will be subject to standard pull-rate limits and pull caps in Docker Hub.

[aschleifer]

Also just because you can pull them without rate limting from docker hub doesn't mean you want to. There are security concerns out there which make you want to remove public "random" internet access from your cluster nodes. in that case you want to have PTC through a "trusted" registry (e.g. AWS ECR). in that case everything is pulled through a PTC but still the official image.

[RobKenis]

Can this be adapted to global.security.allowRegistries: []? We pull all images through our own registry for multiple reasons, so this change feels a little counter intuitive.

[claytonchew]

Might be worth adding an exception for "official" mirrors – such as https://gallery.ecr.aws/bitnami/.

Also, with many mirrors out there, there's even no mention of what source is considered the "official" images anywhere?

[franklouwers]

@claytonchew it seems that mirror isn't even "official" anymore. The blogpost announcing it's official is gone, no more mention of it on the website etc