Error using docker-container (remote) builder context. How is connecting to a remote $DOCKER_HOST supported?

[fenollp]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I would like to create and connect to a docker-container builder through SSH (with custom port) or TCP. I use a builder in the hope that it'll negotiate the connection only once, keeping it up until I remove it.

I created a builder with this command (ssh tstr works on my machine and using this Docker daemon via $DOCKER_HOST=ssh://tstr works):

λ docker buildx create --name=tstr --driver=docker-container ssh://tstr --bootstrap --driver-opt=image=docker.io/moby/buildkit:buildx-stable-1@sha256:5d410bbb6d22b01fcaead1345936c5e0a0963eb6c3b190e38694e015467640fe

Expected behaviour

λ DOCKER_CONTEXT=default DOCKER_BUILDKIT=1 docker build -o=/usr/local/bin --platform=local --build-arg PREBUILT=1 https://github.com/FuzzyMonkeyCo/monkey.git 
[+] Building 2.3s (3/3) FINISHED                                                                                                                                               docker:default
 => [internal] load git source https://github.com/FuzzyMonkeyCo/monkey.git                                                                                                               1.9s
 => CANCELED resolve image config for docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                               0.3s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                         0.0s 
ERROR: failed to solve: Canceled: context canceled                                                                                                                                            

Actual behaviour

λ DOCKER_CONTEXT=tstr DOCKER_BUILDKIT=1 docker build -o=/usr/local/bin --platform=local --build-arg PREBUILT=1 https://github.com/FuzzyMonkeyCo/monkey.git 
Failed to initialize: unable to resolve docker endpoint: context "tstr": context not found: open /home/pete/.docker/contexts/meta/e9f2766a13abc9322b028c93c5a9949fc9f49b2f99d69f1ef26feadf0c86a9c3/meta.json: no such file or directory

λ DOCKER_CONTEXT=tstr docker images
Failed to initialize: unable to resolve docker endpoint: context "tstr": context not found: open /home/pete/.docker/contexts/meta/e9f2766a13abc9322b028c93c5a9949fc9f49b2f99d69f1ef26feadf0c86a9c3/meta.json: no such file or directory

Buildx version

github.com/docker/buildx v0.15.1 1c1dbb2

Docker info

Client: Docker Engine - Community
 Version:    27.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.25.0
    Path:     /home/pete/.docker/cli-plugins/docker-compose

Server:
 Containers: 7
  Running: 0
  Paused: 0
  Stopped: 7
 Images: 1715
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-36-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.98GiB
 Name: solaralchemist
 ID: 4ff1ee7f-a3ff-4df0-ad6e-9d0162ddbda5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: fenollp
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE                 DRIVER/ENDPOINT                   STATUS     BUILDKIT   PLATFORMS
tstr                      docker-container                                        
 \_ tstr0                  \_ ssh://tstr                    running    v0.14.1    linux/amd64, linux/amd64/v2, linux/386
unruffled_ishizaka        docker-container                                        
 \_ unruffled_ishizaka0    \_ unix:///var/run/docker.sock   inactive              
default*                  docker                                                  
 \_ default                \_ default                       running    v0.14.1    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

Configuration

Build logs

No response

Additional info

No response

[fenollp]

Oh I believe I conflated the two concepts of builders & contexts.

From this I'm interested in the part that says

This lets you switch the daemon your docker CLI connects to.

How can one use a context to maintain a connection to a remote docker-container-driver builder?

[tonistiigi]

If you use DOCKER_CONTEXT=tstr docker build that means tstr default builder for your context (unless you used buildx use --global ), not the builder instance.

You probably meant BUILDX_BUILDER=tstr.

If you do have docker context created already on ssh endpoint then you don't need to set the ssh address again on buildx create, you can use the context name or let it use the active context.

[fenollp]

So how does one achieve a persistent ssh connection to a remote docker-container builder? A conn that stays alive in between multiple docker buildx build calls, without having to re-establish the con.
Thanks!

[fenollp]

#1293 and your advice helped me understand that my command should be

docker buildx create --bootstrap --name oomphy ssh://oomphy --driver=docker-container --driver-opt=image=docker.io/moby/buildkit:buildx-stable-1@sha256:5d410bbb6d22b01fcaead1345936c5e0a0963eb6c3b190e38694e015467640fe

Then I make my build calls with $BUILDX_BUILDER=oomphy expecting them to connect straight up to my remote host, yet my code consistently waits around 2 seconds for the first logs to appear. I see this with other networked machines as well.

I also observe some wait times reaching 1s with BUILDX_BUILDER=default and more rarely without any builder, I guess that's more explainable by that I execv buildx on each call.

---

Is there any way for me to set up some kind of proxy that maintains a conn with my remote builder and basically multiplex it through the local UD socket to my local build calls? Can the docker daemon be configured to provide this?
Or do you maybe have some code to show?
And no, there's no way around it, my build calls cannot all come at once.

[tonistiigi]

So how does one achieve a persistent ssh connection to a remote docker-container builder? A conn that stays alive in between multiple docker buildx build calls, without having to re-establish the con.

If you wish for your ssh connection to not drop after buildx/docker exist then that is controlled by your SSH config.

Usually smth like

  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlMaster auto

First google link: https://www.cyberciti.biz/faq/linux-unix-reuse-openssh-connection/