Support for SFTP/SCP?

[mvoelske]

AsyncSSH seems to already support this on the protocol level. Perhaps all it takes is an SFTPServer subclass that overrides all the file operations as appropriate kubectl exec calls into the user pod? That would probably be quite slow, with every operation spawning a new process, but it might do in a pinch.

[yuvipanda]

This would be awesome as a basic start! However, a lot of those operations seem weird / hard to do with kubectl exec - for example, 'read' and 'write'. We'll have to approximate these using bash...

Another alternative I've been thinking of is:

1. Use kubectl exec to start an sshd process on the server, secured to be accessible by just that kubectl exec. It should only allow sftp access, and only on localhost.
2. Proxy all SFTP requests through to this sshd instance.

What do you think about this? It requires sshd is installed in the container, but could probably get away with asking people to do that now.

We could also use another mechanism (rather than sshd) to provide these capabilities. If it is a single binary, we could also just copy it into the container ourselves first before starting it, thus removing need for any package to be pre-installed in the container.

However, if you think we can get away with just using kubectl exec without a persistent process on the other side, that's a great great start.

[mvoelske]

Proxying to an OpenSSH would be much better than a bunch of shellscripts, of course. Come to think of it:

It requires sshd is installed in the container, but could probably get away with asking people to do that now

We could even avoid depending on OpenSSH in the user image if we inject a sidecar container into the pod spec, with the same VolumeMounts as the original container. It might be a bit confusing to see a different root FS via SFTP versus in the shell, but touching any files outside those volumes makes no sense in this setting anyway.

[yuvipanda]

It might be a bit confusing to see a different root FS via SFTP versus in the shell, but touching any files outside those volumes makes no sense in this setting anyway.

I was thinking about this, but then realized an important use case is to copy out files you have in non-persistent storage. For example, a file in /tmp. Or a big output file you have in /srv that can't fit your persistent storage device...

[mvoelske]

Good point, I hadn't thought of that. Copying a binary into the user container might be the most flexible solution then. Apparently OpenSSH's sftp-server already is a standalone binary that communicates over stdin/stdout. We wouldn't even need the full sshd, then. I haven't checked yet if it's easily possible to make a statically linked build; that would be desirable when we deploy it into an unknown container.

[yuvipanda]

Way, awesome. I see in the page you linked 'sftp-server is not intended to be called directly, but from sshd(8) using the Subsystem option.'. Very curious to see if we can get a static build going!

[yuvipanda]

We can prob probably still start off with full sshd, and make that implementation pluggable. What do you think?

[mvoelske]

I've played around with OpenSSH a bit: it's possible to get a static sftp-server binary out of it, without needing to modify the code at all. The libc nssswitch stuff was a bit problematic, but linking against dietlibc instead solves that. The shellscript in this repo produces a small static binary that works in every image I've thrown at it so far. The only requirement seems to be that /etc/passwd has an entry for the uid that the sftp-server process runs as.

The next needed piece would be a component that runs in the kubessh pod, and talks to the OpenSSH sftp-server running in the user pod over stdin/stdout. I think the easiest way to accomplish that would be if we could somehow pass through all SFTP related packets from the client side without even decoding them -- as far as I can gather, they should already be in the correct wire protocol that sftp-server understands. In asyncssh, this seems to be the class where SFTP packets initially get decoded; maybe we can override it to pass them through instead. Failing that, another slightly messier option would be to let asyncssh decode the packet, and provide an SFTPServer implementation that uses asyncssh's SFTP client code to re-encode the packets and pass them along.

Running a full sshd on the other end would be my least favored option, because we'd have to trick that sshd into authenticating us as the user (e.g. by injecting a password or public key into the pod?), and then I think we'd still need the hybrid SFTPServer/Client above on top of that.

[yuvipanda]

woop, if you can get this to work with a statically linked sftp-server, that would be amazing! \o/

[mvoelske]

It took some doing to patch this into asyncssh, but I now have a working implementation for a local sftp-server subprocess via Popen. All that's missing now is the kubernetes wiring to run in the user pod instead.