Adding arbitrary git remotes as code hosts

[imax9000]

I'm looking at the https://docs.sourcebot.dev/docs/connections/overview and don't see how to add a repository that I can fetch with git clone, but that's not hosted on one of the listed code host kinds.

Is there a way to add any repo that you can clone? If not, there probably should be an option to do that.

[msukkari]

Hey Max, we had this functionality in v2 but no longer support it in v3. If you could describe your use case I can see if we can add it back

[imax9000]

Umm, the use case is pretty obvious: adding any git repo regardless of how it's hosted, e.g.:

• https://git.kernel.org/
• https://go.googlesource.com/go
• https://savannah.gnu.org/
• https://cgit.freedesktop.org/ (deprecated in favour of a Gitlab instance, but that wasn't always the case)
• Any of the repos people self-host with Forgejo
• Any of the repos people self-host with a plain HTTP server
• Any of the repos hosted on https://tangled.sh
• Repos on my local machine, some of which aren't published anywhere

At various points in time I had all of the above (except maybe for https://tangled.sh) in my old Sourcegraph instances.

[brendan-kellam]

Will look into this. The approach I'm thinking is this: #250 (comment)

[imax9000]

Thanks! Yeah, that would probably get me covered. The ability to recursively index all repos under a given local dir would be nice to have, but I can live with just explicitly enumerating the ones I want.

Another thing that would be nice to have: AFAIR, Sourcegraph allowed to specify multiple repos for a single Git host, and would just append provided repo paths to the host URL to obtain the full address. That would group together the relevant repos, instead of having an entirely independent connection for each one. But I don't know if that has any meaning to Sourcebot and may be not worth the effort.

[brendan-kellam]

Yea I'm planning on supporting the local recursive case w/ glob patterns for the first version. This won't be supported for remote paths ofc.

I'm not familiar with that Sourcegraph feature, but it kind of sounds to me like a "templating" system that could be useful for when you have many remote repositories. Something like:

"my-connection": {
    "type": "git",
    "url": "https://github.com/sourcebot-dev/{{repo}}",
    "variables": {
        // each of these is substituted into {{repo}} to get
        // full remote path.
        "repo": [
            "sourcebot",
            "zoekt"
            // ... etc ...
        ]
    }
}

I'd say this one is out of scope for now, but curious to get your thoughts.

[imax9000]

Here are some examples I had in my Sourcegraph:

{
  "url": "git://git.kernel.org/pub/scm/",
  "repos": ["libs/klibc/klibc"]
}

{
  "url": "https://go.googlesource.com",
  "repos": [
    "go",
    "time",
  ]
}

As you can see, there isn't much to it: just a base URL and a list of suffixes to point to individual repos. I think stuff like fetch/update rate limits applies on per-code host basis, so grouping together repos also helps to not put excess strain on those hosts.

Your example does achieve what I have in mind, but I don't think a full-blown templating is even needed here. Having more than one variable doesn't make much sense (what should Sourcebot do then? Compute Cartesian product?), so it would be less confusing and error-prone if the configuration structure wouldn't allow adding more variables.