Unauthenticated request error installing feature from private oci registry

[derektamsen]

I am having an issue installing a devcontainer feature in a coder workspace. The feature image is pushed to a private gcp artifact registry repository. The identity of the kubernetes service account is configured per the kaniko docs for workload identity authentication and the gcp service account has permission to pull images from the repo.

This is the log from coder (some lines redacted):

🐳 Queued pod from ReplicaSet: coder-8ad41073-62a9-46f8-ba22-dbb0765d701d-5f6db78945
🐳 Created pod: coder-8ad41073-62a9-46f8-ba22-dbb0765d701d-5f6db78945-wc884
Created pod: coder-8ad41073-62a9-46f8-ba22-dbb0765d701d-5f6db78945-wc884
Successfully assigned coder/coder-8ad41073-62a9-46f8-ba22-dbb0765d701d-5f6db78945-wc884 to <redacted>
AttachVolume.Attach succeeded for volume "pvc-528e230d-66bd-42ef-b5f2-0dc38a1f51bb" 
Pulling image "ghcr.io/coder/envbuilder:latest"
Successfully pulled image "ghcr.io/coder/envbuilder:latest" in 213ms (213ms including waiting). Image size: 32816815 bytes.
Created container: dev
Started container dev
envbuilder v1.1.0+a3a6a83 - Build development environments from repositories in a container
Using base64 encoded Docker config
Set DOCKER_CONFIG to /.envbuilder/.docker
Wrote Docker config JSON to /.envbuilder/.docker/config.json
#1: 📦 Cloning https://<redacted>.git to /workspaces/project...
#1: 🔒 Using HTTP basic authentication!
#1: Parsed Git URL as "https://<redacted>.git"
#1: 📦 The repository already exists! [2.577789ms]
No Dockerfile specified, looking for a devcontainer.json...
Building in Devcontainer mode using /.devcontainer/devcontainer.json
Restored DOCKER_CONFIG to 
error: compile devcontainer.json: extract feature us-docker.pkg.dev/redacted/directory/someprivatefeature:1: fetch feature image us-docker.pkg.dev/redacted/directory/someprivatefeature:1: GET https://us-docker.pkg.dev/v2/token?scope=repository%redacted%directory%someprivatefeature%3Apull&service=: DENIED: Unauthenticated request. Unauthenticated requests do not have permission "artifactregistry.repositories.downloadArtifacts" on resource "projects/redacted-project/locations/us/repositories/redacted-repo" (or it may not exist)

My .devcontainer/devcontainer.json contains the following (updated to reflect the redacted log):

{
	"features": {
		"ghcr.io/devcontainers/features/github-cli:1": {
			"installDirectlyFromGitHubRelease": true,
			"version": "latest"
		},
		"ghcr.io/devcontainers/features/terraform:1": {
			"installTerraformDocs": true,
			"version": "1.11.4",
			"tflint": "latest",
			"terragrunt": "latest"
		},
		"ghcr.io/devcontainers/features/aws-cli:1": {
			"version": "latest"
		},
		"us-docker.pkg.dev/redacted/directory/someprivatefeature:1": {}
	},
	"image": "mcr.microsoft.com/devcontainers/base:bookworm",
	"name": "<redacted>",
	"runArgs": ["--env-file", ".devcontainer/devcontainer.env"]
}

The docker config file that is passed to .devcontainer/devcontainer.json contains the necessary credHelpers to support workload identity authentication to us-docker.pkg.dev.

{
  "auths": {},
  "credHelpers": {
    "gcr.io": "gcr",
    "us-central1-docker.pkg.dev": "gcr",
    "us-docker.pkg.dev": "gcr",
    "us.gcr.io": "gcr"
  }
}

envbuilder version: envbuilder v1.1.0+a3a6a83

[derektamsen]

Digging into this issue more I suspect the primary issue is that the feature image pull request from envbuilder does not use any authentication. The relevant portion:

DENIED: Unauthenticated request. Unauthenticated requests do not have permission

Tracing this back from envbuilder -> kaniko -> go-containerregistry I believe this means authentication is not being passed at all to the request to pull the image from the registry.

[derektamsen]

I made an attempt in #458 to add support for pulling feature images from private registries.