Support server-side dynamically generated tarballs

[daewok]

I'd love to make a ql distribution for my colleagues and myself that fetches tarballs directly from our source control website (such as gitlab or github) based on git SHA-1 IDs. This almost works out of the box, but there are a few show stoppers that need to be addressed. I was wondering if the improvements below would be welcome additions to the quicklisp-client.

1. Allow ? to appear in release URLs. Gitlab generates tarballs from URLs of the form: /GROUP/PROJECT/repository/archive.tar.gz?ref=REF-ID. Currently, it seems the ? throws the client for a loop. When unescaped in releases.txt, probe-file treats it as a wildcard and when it's escaped, it's escaped in the URL when fetching which is not correct.

2. Allow URL filenames to be different from the name of the tarball on the file system. Both github and gitlab generate tarballs from URLs with filenames that are independent of the repo name (REF-ID.tar.gz for github and archive.tar.gz?ref=REF-ID for gitlab). However, they both send back the Content-Disposition header with an appropriate name. I'd like to make the client look for this header and if it's provided, use the provided filename for storing the tarball on the file system.

3. Allow users to set headers for the fetch requests. (Not really directly related to server-side generated tarballs, but useful in many situations where you also want to use generated tarballs). We need this to authenticate to our gitlab install (and I believe it could be used for private github repos as well). I have a proof of concept for this that stores headers using config-value for the path fetch-headers/HOSTNAME/HEADER-NAME/value. I imagine a full-featured version would also allow for command instead of value to tell the client to run a command to get the header value in case users don't want to store private tokens in plain text on the file system. And the presence of secure would prevent the header from being sent over unencrypted connections.

[daewok]

Side note: Apparently it's only SBCL and ECL that have issues with ?. CCL, ABCL, and Allegro don't seem to treat ? as a wildcard character (can't test on clisp).

[quicklisp]

Thanks for taking the time to send feedback. I like these ideas.

For 1 and 2, I think it might work to use the "prefix" value from releases.txt as the base name for the tarball - does that make sense?

For 3, I like the idea!

So there may be a potential future issue, though. The planned scheme for download validation involves managing a list of download tarball digests, and signing that list with PGP. Is it feasible to include digests with your planned idea?

[quicklisp]

Or maybe I need to make the validation more pluggable, so that digests+PGP are the default option, but a custom dist can handle things differently.

[daewok]

Cool. For number 3 I'll clean up my implementation and send a PR.

As I've described it, it would be possible to include a digest since something has to generate the distribution with the SHA-1s in it and that tool could also fetch the tarballs to generate the digest and sign it. But I'd definitely argue for a more pluggable validation scheme. I'd love to also be able to make "live" distributions that fetch based on branch names and it's not realistic to create a digest in that scenario.

Similarly, using the prefix to create the tarball name locally would work here, but fall flat in the live distribution idea unless there's also an option we could set on distributions to always fetch tarballs on update even if they exist locally.

It's also worth noting that handling "?" is going to block on https://gitlab.com/embeddable-common-lisp/ecl/issues/394 for ECL. It's recognized as a bug in ECL and not a feature, but let me know if you'd want to wait on merging it to the client until ECL is fixed.

[daewok]

D'oh. Actually I think there are ways to work around that ECL bug since the local file names likely won't have a "?" in them. So ignore that last bit.

[daewok]

And the live distribution would be a fairly big change that I imagine would likely live better in some extension to quicklisp, I just wanted to mention some things that the core client could do to help support it.