crates.io releases?

[RalfJung]

Looks like the latest josh release is not yet at https://crates.io/crates/josh. Are new releases usually put there?

When using josh as a docker image it cannot access my SSH keys for pushing, so I plan to change our instructions to install josh via cargo install instead, but currently we have to do that from the git repo since the crates.io version is outdated. Hence the question.

[vlad-ivanov-name]

Support for SSH access using local keys will likely be removed soon. Currently SSH fetch is supported with SSH agent forwarding in the latest docker image, but not push yet.

[christian-schilling]

About the initial question: We currently don't release to crates.io anymore and I don't think it really makes sense for the proxy.
We might start to release the CLI only tools (josh-filter and a reworked version of git-sync) at some point but that is not what you are looking for.
So for now I'd recommend to either keep using the docker image or use a git url for cargo. Your SSH issue should be resolved when using the docker image as soon as push over SSH with agent forwarding is implemented.

[RalfJung]

Support for SSH access using local keys will likely be removed soon.

Oh, that would be a bummer. :/

[vlad-ivanov-name]

The main reason for that is that it’s insecure, and that SSH agent forwarding fully covers all potential use cases. You can even try it with the latest container by overriding GIT_SSH_COMMAND to enable ssh ForwardAgent option

[RalfJung]

I don't see how it is insecure for josh to just run as my user ID and do a git push the same way that I do by hand. Docker makes things complicated, sure, but that's why I asked for crates.io releases. ;) A git URL will also work for us, though the version number we put in there will probably be outdated fairly quickly.

[vlad-ivanov-name]

It’s insecure because when deployed, it lets users to essentially bypass authentication by using the key on the server.

as for docker requirement, unfortunately it is there to stay. We can’t even guarantee josh will always be fully functional when run as a single process - simply because it needs additional services to work, hence the multi-service container. Docker is a more or less industry standard, so I can’t really see how it makes things complicated.

one thing you could try is just build a container from a given commit sha locally and then run it

[RalfJung]

Ah sure, on a server it shouldn't use the local keys. In the docker container there won't be any local keys anyway.

When running it locally however, it using my keys seems fine? Though maybe I should use josh-filter instead -- we went for the proxy as that's what the documentation mostly talks about. Does josh-filter have support for caching?

Docker is a more or less industry standard, so I can’t really see how it makes things complicated.

🤷 I still find docker pretty complicated and struggle each time I have to use it (which is rare). It's whole set of custom terminology, tons of implicit global state, and fairly poor UI don't help. I totally believe that if you use it all the time, it all makes sense and you know how to circumnavigate its rough edges, but for occasional use I find it fairly annoying.

[vlad-ivanov-name]

When running it locally however, it using my keys seems fine? Though maybe I should use josh-filter instead -- we went for the proxy as that's what the documentation mostly talks about. Does josh-filter have support for caching?

Yeah locally using locally available keys makes more sense. And you’re right that this might not be the best use of a proxy. I don’t think josh-filter has cache but maybe @christian-schilling can correct me

[vlad-ivanov-name]

Would it help if there was a script or something in the repo for building and running release containers? Currently we have one for test containers but not for release

[RalfJung]

I mean, the one-liner is not so bad:

docker run -it -p 8000:8000 -e JOSH_REMOTE=https://github.com -e JOSH_EXTRA_OPTS=--no-background -v josh-vol:/data/git joshproject/josh-proxy:r22.09.27

The problem is all the "things around" the container, which will be hard to put into a script -- things like #1089. (It took me quite a while to figure out that adding -it makes Ctrl-C work, only to then have it broken again a week later by the next update. I guess we can remove -it now, not sure.) Or things like #1088; it again took me some time to figure out how to delete that docker volume to fix this problem (docker refused to delete volumes that are still referenced somewhere, but provided 0 help to identify and delete those references -- listing a bunch of randomly generated UIDs without context is not what I would call helpful). Basically one cannot have things run in docker and pretend it's still a regular Linux process, one has to actually know docker to manage this -- which makes sense I guess, though I have much less trouble with things like flatpak that have less "home-grown infrastructure" around them that one has to learn.

And then there is the authentication problem, as we discussed in #1089. Though for a docker container I run locally I'd be fine giving it access to my ssh-agent (however that would work).

[christian-schilling]

It looks to me that the whole "service in a container" thing is really not that ideal for your use case. It's not really what the proxy is intended to be used for. Having a CLI that just does the filtering without the whole container/network/auth stuff would be much easier to handle.
josh-filter is supposed to be the tool for such a use case. I'm not 100% that it covers everything you need in the "reverse" direction, but if it doesn't that is something we should fix.
Of course josh-filter also does caching just like josh-proxy does. The only thing to be aware of is that you should not use git gc or git prune on such a repo, but this is also something that should be fixed. (as in: Properly handle cases where objects are referenced by the "cache" but missing in the ODB by re-creating them when needed)
josh-filter also makes much more sense to be released to crates.io.

[RalfJung]

I don't see a way to set the cache to be used by josh-filter though?

Even for the "fetch" side, I assume we'd have to arrange a bunch of things ourselves, like getting the upstream rustc commits into a local repo, and then getting the transformed commits into the Miri repo... the proxy interface is taking care of these things quite nicely.

[RalfJung]

The fact that the version on crates.io is ancient is a big footgun: people run cargo install josh and then spend hours debugging issues that have long been fixed. If you do not intend to update the crates.io version, it would be much appreciated if you could yank all the releases that have been made, to reduce the chance that people accidentally use an outdated broken version.

For anyone else reading along, we use the following command for install josh for Miri and it works great:

cargo +stable install josh-proxy --git https://github.com/josh-project/josh --tag r24.10.04

(You may want to bump the tag in the future, when there is a new release.)

[RalfJung]

Correction -- the outdated versions did not just cause hours of debugging, but permanently unfixable issues since broken history has been pushed into a repo where force pushes are an absolute no-go. :(

@christian-schilling it would really be better to jank the old broken versions. Or, even better, to release fixed versions on crates.io. :)