skopeo inspect, on tag list failure, does not loudly highlight mirrors are not used for that

[nvsmirnov]

Summary

skopeo inspect bypasses docker.io location remapping (and mirror configuration too), contacts registry-1.docker.io directly instead.

My site uses combination of unqualified-search-registries and docker.io mirroring/remapping (air-gapped setup).

With this setup, skopeo inspect works only if I pass full image name with my registry (mirror) address.

If I pass unqualified image name, it successfully fetches the manifest from configured mirror or remapping, but then attempts a direct connection to registry-1.docker.io anyway. In example below, it gets 407 from proxy (which is exactly how it should be in my setup - no direct requests to the internet).

This causes skopeo inspect to fail, even though all required image data is available via the internal registry.

buildah pull (for example) works correctly with the same configuration. docker inspect works too of course.

---

Versions

skopeo version 1.21.0-dev

Reproduced on 1.22.2 commit: c766fdc4c1ff9525fa6a38f860430f10646124b3 as well.

---

Configuration

/etc/containers/registries.conf:

unqualified-search-registries = ["nexus-lan.hq.my.corp:8001"]

[[registry]]
location = "nexus-lan.hq.my.corp:8001"
insecure = true

[[registry]]
prefix = "docker.io"
# location = "docker.io"
location = "nexus-lan.hq.my.corp:8001"
insecure = true

Also tried with this with no success (buildah works with both):

  [[registry.mirror]]
  prefix = "docker.io"
  location = "nexus-lan.hq.my.corp:8001"
  insecure = true

Note on unqualified-search-registries: Yes, configuring an internal registry as the unqualified search registry is generally not ok, but it is permitted by the spec and documented as valid. The problem reproduces with fully-qualified docker.io/library/alpine:3.23.4 references as well.

---

Steps to reproduce

skopeo inspect --debug docker://alpine:3.23.4

---

Expected behavior

skopeo inspect returns the image metadata. All data is fetched from location/mirror for docker.io (nexus-lan.hq.my.corp:8001 in example), just as buildah pull does.

---

Actual behavior

skopeo inspect fetches the manifest and config blob from Nexus successfully, then starts a second registry lookup for docker.io/library/alpine that ignores the location remapping (or mirror config) and contacts registry-1.docker.io directly:

FATA[0000] Error determining repository tags: pinging container registry registry-1.docker.io: StatusCode: 407, "..."

(here 407 is from proxy server while trying to access docker.io directly)

---

Debug output with annotations

root@73f83223b30c:/# skopeo inspect --debug docker://alpine:3.23.4

DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"

# Phase 1: image data fetch - location remapping works correctly
DEBU[0000] Trying to access "nexus-lan.hq.my.corp:8001/library/alpine:3.23.4"
...
DEBU[0000] GET http://nexus-lan.hq.my.corp:8001/v2/
DEBU[0000] Ping http://nexus-lan.hq.my.corp:8001/v2/ status 401
DEBU[0000] GET http://nexus-lan.hq.my.corp:8001/v2/token?scope=repository%3Alibrary%2Falpine%3Apull&service=...
DEBU[0000] Increasing token expiration to: 60 seconds
DEBU[0000] GET http://nexus-lan.hq.my.corp:8001/v2/library/alpine/manifests/3.23.4
DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json"
DEBU[0000] GET http://nexus-lan.hq.my.corp:8001/v2/library/alpine/manifests/sha256:4d889c14e7d5a73929ab00be2ef8ff22437e7cbc545931e52554a7b00e123d8b
DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0000] Downloading /v2/library/alpine/blobs/sha256:3cb067eab609612d81b4d82ff8ad71d73482bb3059a87b642d7e14f0ed659cde
DEBU[0000] GET http://nexus-lan.hq.my.corp:8001/v2/library/alpine/blobs/sha256:3cb067eab609612d81b4d82ff8ad71d73482bb3059a87b642d7e14f0ed659cde

# Phase 2: new lookup, location remapping/mirror config NOT applied
DEBU[0000] Using registries.d directory /etc/containers/registries.d      # config re-read
DEBU[0000] No credentials matching docker.io/library/alpine found in ...  # canonical name used, not nexus-lan
DEBU[0000]  No signature storage configuration found for docker.io/library/alpine:3.23.4, ...
DEBU[0000] GET https://registry-1.docker.io/v2/                           # goes directly to Docker Hub
DEBU[0000] Ping https://registry-1.docker.io/v2/ err ... Proxy Authentication Required
DEBU[0000] GET http://registry-1.docker.io/v2/
DEBU[0000] Ping http://registry-1.docker.io/v2/ status 407                # 407 is EXPECTED: direct internet is blocked
FATA[0000] Error determining repository tags: pinging container registry registry-1.docker.io: StatusCode: 407, "..."

---

Proof that all image data is accessible via the internal registry

The config blob referenced in the manifest is served correctly by Nexus:

root@73f83223b30c:/# curl http://nexus-lan.hq.my.corp:8001/v2/library/alpine/blobs/sha256:3cb067eab609612d81b4d82ff8ad71d73482bb3059a87b642d7e14f0ed659cde | jq .
{
  "architecture": "amd64",
  "config": {
    "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ],
    "Cmd": [ "/bin/sh" ],
    "WorkingDir": "/"
  },
  "created": "2026-04-15T20:01:40.139676757Z",
  ...
}

---

buildah pull - works

buildah pull uses the same registries.conf and successfully pulls through Nexus without ever contacting registry-1.docker.io:

root@73f83223b30c:/# buildah pull alpine:3.23.4
Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull nexus-lan.hq.my.corp:8001/alpine:3.23.4...
Getting image source signatures
Copying blob 6a0ac1617861 done   |
Copying config 3cb067eab6 done   |
Writing manifest to image destination
3cb067eab609612d81b4d82ff8ad71d73482bb3059a87b642d7e14f0ed659cde

---

Workaround

# Use explicit registry URL - location remapping not needed, tags fetched from Nexus
skopeo inspect docker://nexus-lan.hq.my.corp:8001/alpine:3.23.4 | jq .Digest
"sha256:5b10f432ef3da1b8d4c7eb6c487f2f5a8f096bc91145e68878dd4a5019afde11"

[mtrmac]

Thanks for reaching out. The access that is failing is to list tags in a repo, and that is not subject to mirroring.

You can use skopeo inspect --no-tags …; that is a fairly good idea in most cases.

[nvsmirnov]

@mtrmac ok, thank you!

Although it would be nice to have some message along with the error - in case when there is a mirror defined, or registry location redefined.

Or better just when it got manifest successfully and then failed to get tags, to print "you can use --no-tags ..."

[nvsmirnov]

... although, I compared the results of inspect (.RepoTags) - they are identical for cases: when I run inspect directly on docker.io and on my Nexus proxy-repo (specifying full repo name). So, it seems like at least for case when location is redefined for docker.io, it is reasonable to use it instead of docker.io directly.

[mtrmac]

… It already says “Error determining repository tags”. Now, admittedly, I completely overlooked that part myself, as well.

As for warning only when a mirror is defined … it’s a thought; it would mean somewhat breaking the abstractions we built (which users don’t care about), but it would also be noise for users who already know the mirror configuration does not apply in this situation.

[mtrmac]

So, it seems like at least for case when location is redefined for docker.io, it is reasonable to use it instead of docker.io directly.

That’s completely dependent on what the owners of the source + mirror repositories chose to store in those repositories . Nothing guarantees that the two are the same. I understand partial mirrors, where the tag sets differ, are pretty common.

[nvsmirnov]

That’s completely dependent on what the owners of the source

Yes of course. But if someone is redefining actual location of docker.io - he probably thinks that all requests to docker.io must go to that location, at least for me it sounds logical. And of course the owner of this set-up must understand the consequences of redefining registry location.

Really, I'm not insisting. If my thoughts sounds just not right, it's ok to close the issue.

[mtrmac]

I think changing the tag lookup to use mirrors would be very surprising, and should not happen.

On improving the error message, I do see how the current state can be confusing, but I’m not sure what can be done at a reasonable cost.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.