Gemstash server integration

[micheelengronne]

What would you like Renovate to be able to do?

Renovate should be able to fetch gems in a private Gemstash server. It should be able to handle credentials for this server.

AFAIK, the url /api/v1/gems seems implemented in Gemstash but only for pushing gems https://github.com/rubygems/gemstash/blob/9d0e45d80e70f7e2d3987e5e1cdef68cd1119a91/lib/gemstash/web.rb#L48

It seems that urls used to fetch gems are /api/v1/dependencies. That's what fetched when I use bundle lock -V.

For instance, here fetching ffi and nokogiri gems:

HTTP GET https://username:password@INTERNALDOMAIN/private/api/v1/dependencies
HTTP 200 OK https://username:password@INTERNALDOMAIN/private/api/v1/dependencies
Fetching gem metadata from https://INTERNALDOMAIN/private/
Query List: ["ffi", "nokogiri"]
Query Gemcutter Dependency Endpoint API: ffi,nokogiri
HTTP GET https://username:password@INTERNALDOMAIN/private/api/v1/dependencies?gems=ffi%2Cnokogiri
HTTP 200 OK https://username:password@INTERNALDOMAIN/private/api/v1/dependencies?gems=ffi%2Cnokogiri

[rarkins]

Can we locate any documentation for this API? We need to (a) support it and (b) have a way to know how/when to switch to it

[micheelengronne]

Here is the doc from the user POV for setting auth https://github.com/rubygems/gemstash/blob/master/docs/gemstash-private-gems.7.md

The files handling auth and fetch are:

https://github.com/rubygems/gemstash/blob/master/lib/gemstash/api_key_authorization.rb

https://github.com/rubygems/gemstash/blob/master/lib/gemstash/authorization.rb

https://github.com/rubygems/gemstash/blob/master/lib/gemstash/gem_fetcher.rb

But I don't know if they documented an API. I think we should empirically test with a Gemstash docker container.

[micheelengronne]

I opened an issue on their side rubygems/gemstash#272

[rarkins]

If it works fine with Bundler then I'd hope there's an answer out there already, we just haven't found it yet. Thanks for your help

[micheelengronne]

I investigated further. Gemstash gives the api/v1/dependencies.json endpoint.

It is used that way (example with the ffi gem):

api/v1/dependencies.json?gems=ffi

and gives this result:

[{"name":"ffi","number":"1.9.11","platform":"x86-mingw32","dependencies":[]},{"name":"ffi","number":"0.4.0","platform":"java","dependencies":[]},{"name":"ffi","number":"1.0.11","platform":"java","dependencies":[]},{"name":"ffi","number":"1.2.0.pre6","platform":"ruby","dependencies":[]},{"name":"ffi","number":"1.2.0.pre5","platform":"ruby","dependencies":[]},{"name":"ffi","number":"1.7.0.dev","platform":"ruby","dependencies":[]},{"name":"ffi","number":"1.9.14","platform":"x64-mingw32","dependencies":[]},{"name":"ffi","number":"1.9.8","platform":"java","dependencies":[]},{"name":"ffi","number":"1.4.0","platform":"x86-mingw32","dependencies":[]},{"name":"ffi","number":"1.9.12","platform":"x64-mingw32","dependencies":[]},{"name":"ffi","number":"1.8.1","platform":"java","dependencies":[]},{"name":"ffi","number":"1.2.0.dev","platform":"ruby","dependencies":[]}...

etc...

It looks similar to the result of api/v1/gems (that works with https://rubygems.org) but gives less informations (no created_at, rubygems_version, or ruby_version).

But, we only need number and platform and it gives that.

So there is smthg to do here :

renovate/lib/datasource/rubygems/get.ts

Line 56 in b45502c

const releases = versions.map(

And here:

renovate/lib/datasource/rubygems/get.ts

Line 10 in b45502c

const INFO_PATH = '/api/v1/gems';

BTW, this endpoint also exist in rubygems. That would make the manager simpler and more reliable by removing:

https://github.com/renovatebot/renovate/blob/master/lib/datasource/rubygems/releases.ts#L10

and the file:

https://github.com/renovatebot/renovate/blob/b45502cd28dee7b583dd2f00da76a3807304d92f/lib/datasource/rubygems/get-rubygems-org.ts

[rarkins]

Thanks for the analysis. FYI rubygems.org will block any IP that uses the regular API more than one query per second, so it is definitely not possible to replace that functionality.

I think we want to default to the API that gives us the. Eat metadata and then fall back to the less informative one.

[rarkins]

Also note that gemfury uses the same bulk approach as rubygems

[micheelengronne]

Ok, as bundler uses the api/v1/dependencies the implementation of it will be the most compatible as any gem repository will at least implement this endpoint. Otherwise, they would not be compatible with bundler (smthg impossible in the ruby world).

[rarkins]

Bundler itself does not use the Rubygems API for most queries due to the load. It uses the append-only flat versions file that Renovate now uses too. Hopefully other servers support api v1 though