Any way to build Docker images with containerMode=kubernetes?

[elcidowneador]

Hello, i'm trying to build Docker images using docker/setup-buildx-action@v2 with kubernetes driver and also docker/build-push-action@v4 action, but i'm getting this error:

Error: buildx failed with: ERROR: error for bootstrap "builder-40aacad7-1de4-4b62-9dfd-275a9139c7690": deployments.apps "builder-40aacad7-1de4-4b62-9dfd-275a9139c7690" is forbidden: User "system:serviceaccount:gha-runner:gha-runner-default-gha-rs-kube-mode" cannot get resource "deployments" in API group "apps" in the namespace "gha-runner"

This is happening because the role created by default don't add deployments permissions, but does not allow customizations neither.

Someone was able to achieve without forking the chart?

[omri-shilton]

we would also like to know if thats possible

[maljolani]

do you have a solution for it

[alextweneymackenzie-ibi]

I create my own service account, role, and role binding, then set template.spec.serviceAccountName. This lets me create containers using Kubernetes mode.

[diegoara96]

Hi @elcidowneador , you can't use the docker daemon in kubernetes mode. I use kaniko to generate my images and it works fine, it is intended for this same purpose, to run without daemon. You have several tutorials on the internet on how to create your actions with this. https://github.com/GoogleContainerTools/kaniko

[moulougeta]

@diegoara96 i'm interested in this approach. Have you by any change also used this approach to build multi-arch builds? If so, how did you handle it?

[oradwell]

we use buildkit daemon on the Kubernetes cluster and the workflow pods connect to it to build containers. you're also able to use the docker actions with it.

[moulougeta]

@oradwell This is interesting. Could you please elaborate how you scale buildkit to handle increasing load? Plus have you encountered any issues as, if I understood correctly, multiple jobs with use the same buildkit service?

[oradwell]

Initially it was set up as a cluster-wide deployment which didn't scale well, as you'd expect. The current implementation has buildkit as a sidecar container in the workflow pod. Each workflow has its own buildkit daemon

[moulougeta]

Ah I see. We currently have buildkit as a sidecar but still in dind mode. I haven't thought about having buildkit as a sidecar to the workflow pod. Would love to hear how you achieved that.

[oradwell]

You should be able to adapt the buildkit deployment resource to a workflow pod Hooktemplate definition and define as a sidecar

https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#configuring-hook-extensions

https://github.com/actions/runner-container-hooks/blob/main/examples/extension.yaml