Can i convert proxyfied registries ?

[ptempier]

I mean, i tried with estargz, it didn't work, but i was somehow expecting it to work.
Not sure if
it's not supported at all and i am doing something wrong.
Maybe i need a different setup, like an actual proxy project and an accelerated copy of that project.
It's not working and its not supposed to work.
It doesn't work right now, but maybe it will in a future release.

[Desiki-high]

Hi @ptempier. Could you please give more information to help reproduce your issue? Such as your setup steps and what kind of proxyfied registries are used. Acceleration-Service can work well with estargz in the latest harbor.

[ptempier]

Hello

I am not using the latest version, but v2.9.0-6d1ad65c, so maybe that's it.
I tried looking at the changlog but couldn't find anything related to acceld and proxy.
Was the support added in 2.10 ?

harbor-acceld is v0.2.13

The test was doing a pull from a proxified github registry.
The robot could successfully authenticate to pull the image
the accelertor could transform to estargz
but i was getting an nginx permission error (403?) when pushing, it was not appearing in the harbor logs.

Creating an estargz image via kaniko, pushing it in a different projet
then pulling it with a properly configured containerd works as expected.

[Desiki-high]

Can you provide the acceld work log? @ptempier

[Desiki-high]

Actually harbor v2.9 is supported too.

[Desiki-high]

Can you provide the acceld work log? @ptempier

If you can provide the harbor log, it will be better.

[ptempier]

The time it almost worked , i had this error :

error msg="convert in worker: push image: unexpected status from POST request to https://anonyme/v2/hub.docker.com/library/mysql/blobs/uploads/: 403 Forbidden"

Just tried a docker pull anonyme/hub.docker.com/library/postgres
It went through the proxy cache as per habor log

harbor#proxy-cache-service hub.docker.com/library/postgres:sha256:3a27b8f06bc0cc0b76ab124b8c48bf3177703aedbd9cc28fcebc0e312bcb8c7a artifact create 1/9/24, 12:51 PM
anonymous hub.docker.com/library/postgres:sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982 artifact pull 1/9/24, 12:45 PM
harbor#proxy-cache-service hub.docker.com/library/postgres:sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982 artifact create 1/9/24, 12:45 PM

It went through the webhook

10703 WEBHOOK Success Artifact pushed xxxx 1/9/24, 12:51 PM 1/9/24, 12:51 PM
10702 WEBHOOK Success Artifact pushed xxx 1/9/24, 12:45 PM 1/9/24, 12:45 PM

But then in acceld failed with :

time="2024-01-09T11:45:17.232130695Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/postgres@sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982"

time="2024-01-08T15:40:46.676196062Z" level=info msg="Version: v0.2.13 b82ccee1e73845741b033f8b04ab418fad5b84ef.20240108.0649\n"
time="2024-01-08T15:40:46.698308247Z" level=info msg="[API] HTTP server started on 0.0.0.0:2077"
time="2024-01-08T15:50:26.552634578Z" level=info msg="received webhook request from 172.19.0.1:52386" module=api
time="2024-01-08T15:50:26.581593392Z" level=info msg="POST /api/v1/conversions 200 28.919643ms 602>5bytes 172.19.0.1" module=api
time="2024-01-08T15:50:26.606109487Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/rockylinux@sha256:8b5296204bad12e84837c7b4c8b2cdff45bcc92ee5ffaaa3e86ba683f2384b14"
time="2024-01-08T15:50:31.651439807Z" level=info msg="received webhook request from 172.19.0.1:52392" module=api
time="2024-01-08T15:50:31.659627035Z" level=info msg="POST /api/v1/conversions 200 8.161151ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T15:50:31.668051229Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:28a16e31b140d750048cd5fadcaed22ac08d0eeb18567f79f822aee1f237b43c"
time="2024-01-08T15:53:45.357608101Z" level=info msg="received webhook request from 172.19.0.1:57896" module=api
time="2024-01-08T15:53:45.366814898Z" level=info msg="POST /api/v1/conversions 200 9.191926ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T15:53:45.373215343Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/nginx@sha256:5be1749f6a023b14ef069f2bbe1afd9a39295694a490963e527a848e4bc4d442"
time="2024-01-08T15:56:51.580870348Z" level=info msg="received webhook request from 172.19.0.1:47690" module=api
time="2024-01-08T15:56:51.587785057Z" level=info msg="POST /api/v1/conversions 200 6.842051ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T15:56:51.634318232Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:6f453b1c7bcbc42b8e3d7949d8dfa28a70f8bf86dff277f5909c5e714ee5153a"
time="2024-01-08T16:05:49.7946499Z" level=info msg="received webhook request from 127.0.0.1:34366" module=api
time="2024-01-08T16:05:49.80163495Z" level=info msg="POST /api/v1/conversions?sync=false 200 7.002226ms 135>5bytes 127.0.0.1" module=api
time="2024-01-08T16:05:49.802579063Z" level=info msg="pulling image anonyme/hub.docker.com/library/mysql:latest" module=converter
time="2024-01-08T16:05:52.887395575Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0977332ebd0a237ff1a892785275d5af61c13b24cd15cbe0ed7cadb7b5b68102" mediatype=application/vnd.in-toto+json size=34939
time="2024-01-08T16:05:52.888131555Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:dfe71fd1f4151b9017635d498f57dfb6d6b932f909c21a79cf1186851ec3317a" mediatype=application/vnd.in-toto+json size=11571712
time="2024-01-08T16:05:53.009889983Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0b8a63c307592189c7a4ea188ee3b5e76eba99edc475a34c635157d5e893e536" mediatype=application/vnd.in-toto+json size=34907
time="2024-01-08T16:05:53.009965038Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:8de9290b82bb48a07e68ea96148c288ffa09729055d5c506b7690cf42f948baf" mediatype=application/vnd.in-toto+json size=11573114
time="2024-01-08T16:06:05.439993598Z" level=info msg="pulled image anonyme/hub.docker.com/library/mysql:latest , elapse 15.607270101s" module=converter
time="2024-01-08T16:06:05.440048067Z" level=info msg="converting image anonyme/hub.docker.com/library/mysql:latest" module=converter
time="2024-01-08T16:06:13.090493323Z" level=info msg="received webhook request from 172.19.0.1:51998" module=api
time="2024-01-08T16:06:13.09860536Z" level=info msg="received webhook request from 172.19.0.1:51990" module=api
time="2024-01-08T16:06:13.11688972Z" level=info msg="POST /api/v1/conversions 200 26.377558ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T16:06:13.166796733Z" level=info msg="POST /api/v1/conversions 200 47.731353ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T16:06:13.186198622Z" level=info msg="received webhook request from 172.19.0.1:52006" module=api
time="2024-01-08T16:06:13.232626827Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:b8bfa6bfda24cf129ce2a20ea3fe679ad377840c0c16d7937887233485b3f170"
time="2024-01-08T16:06:13.279097726Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:483bf8eb111365bf322a25443d3f96ae0d80829c60f00fb329d8a0de0f21c6e7"
time="2024-01-08T16:06:13.318753293Z" level=info msg="POST /api/v1/conversions 200 132.527925ms 587>5bytes 172.19.0.1" module=api
time="2024-01-08T16:06:13.341869273Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/mysql@sha256:e870e58e0e1f937652982f99cddff85ab2076d217db08732856d22eb334e9e2a"
time="2024-01-08T16:08:48.548941466Z" level=info msg="converted image anonyme/hub.docker.com/library/mysql:latest-esgz , elapse 2m43.080799299s" module=converter
time="2024-01-08T16:08:48.549005034Z" level=info msg="pushing image anonyme/hub.docker.com/library/mysql:latest-esgz" module=converter
time="2024-01-08T16:08:48.587453917Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0b8a63c307592189c7a4ea188ee3b5e76eba99edc475a34c635157d5e893e536" mediatype=application/vnd.in-toto+json size=34907
time="2024-01-08T16:08:48.58782373Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:8de9290b82bb48a07e68ea96148c288ffa09729055d5c506b7690cf42f948baf" mediatype=application/vnd.in-toto+json size=11573114
time="2024-01-08T16:08:48.597293349Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:0977332ebd0a237ff1a892785275d5af61c13b24cd15cbe0ed7cadb7b5b68102" mediatype=application/vnd.in-toto+json size=34939
time="2024-01-08T16:08:48.597781217Z" level=warning msg="reference for unknown type: application/vnd.in-toto+json" digest="sha256:dfe71fd1f4151b9017635d498f57dfb6d6b932f909c21a79cf1186851ec3317a" mediatype=application/vnd.in-toto+json size=11571712
time="2024-01-08T16:08:48.968420375Z" level=error msg="convert in worker: push image: unexpected status from POST request to https://anonyme/v2/hub.docker.com/library/mysql/blobs/uploads/: 403 Forbidden"
time="2024-01-08T16:40:47.091302038Z" level=info msg="garbage collect, elapse 8.45699ms"
time="2024-01-08T17:33:13.824434821Z" level=info msg="received webhook request from 172.19.0.1:60738" module=api
time="2024-01-08T17:33:13.914808701Z" level=info msg="POST /api/v1/conversions 200 90.366352ms 617>5bytes 172.19.0.1" module=api
time="2024-01-08T17:33:13.921133726Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/sameersbn/apt-cacher-ng@sha256:58e74113cfac7e593201444648c105351cbfce7538bfb36dcafdc9479b2aefcc"
time="2024-01-08T17:39:58.249120243Z" level=info msg="received webhook request from 172.19.0.1:45660" module=api
time="2024-01-08T17:39:58.299701694Z" level=info msg="POST /api/v1/conversions 200 50.5531ms 608>5bytes 172.19.0.1" module=api
time="2024-01-08T17:39:58.30843498Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/gitlab/gitlab-runner@sha256:7a267b16df7d05786fa7d76758e5a0dcc34dc6318902fbfa29aaad7ad3cbe1be"
time="2024-01-09T11:45:17.207698977Z" level=info msg="received webhook request from 172.19.0.1:45606" module=api
time="2024-01-09T11:45:17.223161837Z" level=info msg="POST /api/v1/conversions 200 15.467165ms 596>5bytes 172.19.0.1" module=api
time="2024-01-09T11:45:17.232130695Z" level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/postgres@sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982"

[ptempier]

I am thinking maybe i need to add some dependencies to acceld ?
Right now it run inside a docker made from rockylinux:9,
At this moment it's only raw rockylinux, 0 packages added, and accelld copied into /root/

[ptempier]

I added some docker tools inside the container, but it doesn't change anything.
level=error msg="convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/alpinelinux/rsyncd@sha256:6f8b68b4b15a8e6b0abfb7db0e2a765849c77a6104ac248810ff9a9fb97996fb"

The Dockerfile i use :

FROM  anonyme/hub.docker.com/library/rockylinux:9

RUN echo "proxy=http://anonyme:3142" >> /etc/yum.conf
RUN dnf -y install dnf-plugins-core && dnf config-manager --set-disabled '*' \
    && dnf config-manager --add-repo 'http://anonyme/repository/rocky/9/AppStream/$basearch/os/' \
    && dnf config-manager --add-repo 'http://anonyme/repository/rocky/9/BaseOS/$basearch/os/'


#add the docker part
RUN dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
RUN dnf -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin


WORKDIR "/root"


RUN dnf install -y wget procps
RUN wget "https://raw.githubusercontent.com/goharbor/acceleration-service/main/misc/config/config.estargz.yaml"

RUN wget "https://github.com/goharbor/acceleration-service/releases/download/v0.2.13/harbor-acceld-v0.2.13-linux-amd64.tgz" && tar -xvzf  ./harbor-acceld-v0.2.13-linux-amd64.tgz
RUN ls -lha
RUN chmod ugo+x ./harbor-acceld/acceld ./harbor-acceld/accelctl


RUN  sed -i 's|hub.harbor.com|anonyme|' ./config.estargz.yaml && \
 sed -i 's|auth_header: header|auth_header: anonyme|' ./config.estargz.yaml && \
 sed -i 's|# auth: YTpiCg==|auth: anonyme|' ./config.estargz.yaml


CMD "./harbor-acceld/acceld --config ./config.estargz.yaml"

[imeoer]

@ptempier You seem to be pushing an image with the name format example.com/namespace/repo@sha256:xxx, how about trying to use the name format example.com/namespace/repo:tag instead?

[ptempier]

@imeoer nope i am doing :

docker pull anonyme/hub.docker.com/library/postgres
anonyme is the harbor registry server
hub.docker.com is the project set as proxy to docker hub
library is the default project space in docker hub
postgres is the image name
my understanding is that docker pull will automatically append the latest is no tag is specfied

So for me it s a very "normal" image pull with a very commonly used registry.

But acceld when receiving such a request respond with :
convert in worker: create target reference by rule: unsupported digested image reference: anonyme/hub.docker.com/library/postgres@sha256:695f076d0483b2169551cf5ae0d8056410d46265fd5ae16d48bb1d86516ff982"

[imeoer]

@ptempier Okay, it seems that the request sent to acceld by harbor webhook contains anonyme/hub.docker.com/library/postgres@xxx image name format that acceld can't handle, which is supposed to be an issue with acceld.

Could you try this?

accelctl convert --config ./config.yaml anonyme/hub.docker.com/library/postgres:latest

[Desiki-high]

@ptempier Okay, it seems that the request sent to acceld by harbor webhook contains anonyme/hub.docker.com/library/postgres@xxx image name format that acceld can't handle, which is supposed to be an issue with acceld.

Could you try this?

accelctl convert --config ./config.yaml anonyme/hub.docker.com/library/postgres:latest

Support image@sha265 can be a new feature? @imeoer .

[imeoer]

Support image@sha265 can be a new feature? @imeoer .

@Desiki-high Yes, it should also support the repo@sha256:xxx image name, but a map rule is needed to transform it to the converted image name, possibly repo@sha256:yyy, however, this makes it inconvenient for users to find the converted image.

[ptempier]

Maybe you need a fix on the harbor side to send the proper image name ?
The client send the query to harbor asking for image:tag
and its harbor who decide that it needs to pull blob in format repo@sha256:xxx
then send this to acceld

So maybe a webhook of acceld type is needed.

[Desiki-high]

Maybe you need a fix on the harbor side to send the proper image name ?

The client send the query to harbor asking for image:tag

and its harbor who decide that it needs to pull blob in format repo@sha256:xxx

then send this to acceld

So maybe a webhook of acceld type is needed.

Thanks for your advice! I will ping you if any process.

[Desiki-high]

@ptempier Could you please help to check acceld build from https://github.com/Desiki-high/acceleration-service/tree/feat/image-rule-sha256

[ptempier]

@Desiki-high
Thanks, it seems now it can build the estargz
but it still can 't push even if it has the permissions.

The robot has the permissions and is authenticated
but it get an error from nginx.
And that error won't show in the habor logs.

• From acceld side :
  2024-01-12T12:19:19.412076000Z time="2024-01-12T12:19:19.411711121Z" level=info msg="converted image anonymous/hub.docker.com/library/nginx:latest-esgz , elapse 29.408130674s" module=converter
  2024-01-12T12:19:19.412786000Z time="2024-01-12T12:19:19.411766868Z" level=info msg="pushing image anonymous/hub.docker.com/library/nginx:latest-esgz" module=converter
  2024-01-12T12:19:19.614473000Z time="2024-01-12T12:19:19.614081943Z" level=error msg="convert in worker: push image: unexpected status from POST request to https://anonymous/v2/hub.docker.com/library/nginx/blobs/uploads/: 403 Forbidden"

• From nginx side:
  2024-01-12T13:24:45.948904063+01:00 172.18.0.1 - "POST /v2/hub.docker.com/library/node/blobs/uploads/ HTTP/1.1" 403 108 "-" "containerd/1.7.11+unknown" 0.017 0.017 .

Permissions (Y = ticked)
Select all
List Repository Y
Pull Repository Y
Push Repository Y
Delete Repository
Read Artifact
List Artifact Y
Delete Artifact
Create Artifact label
Delete Artifact label
Create Tag Y
Delete Tag
List Tag Y
Create Scan
Stop Scan

[Desiki-high]

@ptempier I will take a look later.

[Desiki-high]

It looks like a permission error. You can try to confirm your auth with provider.source.xxxxxx.auth in your config yaml.

[ptempier]

Hello

The auth is fine, the robot appears authenticated when pulling.
I double checked inside the container, the token is properly replaced and with the correct indentation.

[Desiki-high]

Hello

The auth is fine, the robot appears authenticated when pulling. I double checked inside the container, the token is properly replaced and with the correct indentation.

I test the new acceld in my harbor registry, it works well with the image like repo@sha256xxx. Any good suggestions? @imeoer

[ptempier]

upgraded habor
gave all the permissions to the robot
refreshed the token
tried witht he habor super admin
removed the localhost unauthenticated connection
... still not working

Something strange is the unauthenticated pull in the logs, sometime it s authenticated.
Mayeb the code path is wrong and when it tries to push its unauthenticated, i dont know.
Couldnt find in the doc if i could get more verbose logs from acceld

[ptempier]

i did re-read the doc, and it says to use a system robot account and not a project robot account, but i get the same issue.

what tickle me here is the call to ldap.go, when its a robot account, but maybe it just search there after checking its not a robot.

2024-01-15T17:49:56Z [ERROR] [/server/middleware/security/robot.go:58][requestID="f567fb86-c90b-42cc-a6f2-d3d1a6599419"]: failed to authenticate robot account: robot$hub.docker.com-acceld-estargz
2024-01-15T17:49:56Z [WARNING] [/core/auth/ldap/ldap.go:73]: Not found an entry.
2024-01-15T17:49:56Z [WARNING] [/core/auth/authenticator.go:158]: Login failed, locking robot$hub.docker.com-acceld-estargz, and sleep for 1.5s
2024-01-15T17:49:57Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="172.21.0.1" requestID="f567fb86-c90b-42cc-a6f2-d3d1a6599419" user agent="containerd/1.7.11+unknown"]: failed to authenticate user:robot$hub.docker.com-acceld-estargz, error:Failed to authenticate user, due to error 'Not found an entry'
2024-01-15T17:49:58Z [ERROR] [/server/middleware/security/robot.go:58][requestID="43821675-1d74-49eb-bd28-f946e12a4025"]: failed to authenticate robot account: robot$hub.docker.com-acceld-estargz
2024-01-15T17:49:58Z [WARNING] [/core/auth/ldap/ldap.go:73]: Not found an entry.
2024-01-15T17:49:58Z [WARNING] [/core/auth/authenticator.go:158]: Login failed, locking robot$hub.docker.com-acceld-estargz, and sleep for 1.5s
2024-01-15T17:49:58Z [ERROR] [/server/middleware/security/robot.go:58][requestID="ea4db1fa-6856-4f25-91a0-40ca1469520d"]: failed to authenticate robot account: robot$hub.docker.com-acceld-estargz
2024-01-15T17:49:58Z [ERROR] [/server/middleware/security/robot.go:58][requestID="6452d1d5-21cf-45e4-bc78-3d7121325723"]: failed to authenticate robot account: robot$hub.docker.com-acceld-estargz
... it does about 100 tries

[oliverbaehler]

@ptempier Thanks for opening the issue, i came across around the same behavior. Mind sharing your config.yaml? But if I understand correctly it should not be possible to push anything to a project with docker proxy behind (That's also stated in the harbor doc). You also get a permission denied when trying to push anything :

The push refers to repository [harbor/dockerhub/test]
81150088de4c: Preparing
c909727f9cc1: Preparing
4f4ce317c6bb: Preparing
denied: denied: can not push artifact to a proxy project: dockerhub

/dockerhub points to hub.docker.com. Since your logs indicate you only have a 403 on image pushes, i suspect that's what's happening.

I suspect you would need to consider pushing to a different project, which is not a docker proxy. If i am not completely misunderstanding this thread

[ptempier]

@oliverbaehler That's the question i asked first, but apparently its supposed to work.
@Desiki-high says its working for him, but maybe its not well tested and works on in the specific setup of some testbed.

[Desiki-high]

@oliverbaehler That's the question i asked first, but apparently its supposed to work.

@Desiki-high says its working for him, but maybe its not well tested and works on in the specific setup of some testbed.

Sorry, my mistake. I did the test without setting up the proxy. I just fix the issue with image format.

[oliverbaehler]

@Desiki-high The question is, if we could figure out, if a image is from a docker proxy registry. If so, we need to push it to a different project (idk if we could do any rules).
@ptempier What you are trying is currently not supported. Although I was also thinking about that same use case. But you would need as mentioned to push the tags to a different registry and then work with multiple mirrors per registry.

Or the upstream projects release estargz or nydus images..