Implement rate-limit when pulling code from Git host

[gaeljw]

Related to #317

Hello folks,

We've got a Sourcebot instance with hundreds of projects (771 currently). All of these are on a single connection to a on-prem GitLab instance.

We noticed that during startup (and maybe also during the scheduled indexation?) Sourcebot can put some pressure on our GitLab instance to the point that the GitLab instance starts being irresponsive.

Hence, I'm wondering if Sourcebot could have a rate-limit feature when calling GitLab so that GitLab is not overloaded. A simple environment variable that sets a hard limit without trying to be smart could be enough.