Sporadic error when polling for npm package data

[tom--pollard]

msg="Error fetching packages" error="Polling for package @deskpro/react-content-editor returned error: context deadline exceeded (Client.Timeout or context cancellation while reading body)" feed=npm
https://registry.npmjs.org/@deskpro/react-content-editor

I've also logged it happening against https://registry.npmjs.org/@chainsafe/lodestar-utils & https://registry.npmjs.org/@duetds/css

As #123 introduced per package error handling the rest of the packages for the session as processed, in the past this would have resulted in a hard error & further data loss. The exact error isn't returned to the caller when triggered via the httpserver instead it's covered by the generic pollErr text.

[tom--pollard]

Interestingly if I set any of the listed packages as 'critical' npm packages, the timeout error isn't returned. I've only seen the error happen on the 'firehose' when the runtime is over 10s, which highlights it could be down to the global httpClient pointer and concurrency on deferring the body closures.

[Qinusty]

This could potentially be caused by rate limiting on requests as npm slows the rate of replies either intentionally or simply due to the increased load.

[maxfisher-g]

Errors in log still occurring

[calebbrown]

I think blindly generating one goroutine per npm package is a bit naive.

Instead, the packages should be pushed to a pool of workers that limit the amount of concurrency (e.g. instead of 200 reqs all trying to happen at once, only 10 can happen simultaneously.

[calebbrown]

The errors are largely gone with the above changes.

The remaining timeouts appear to be limited to packages with ~1000 or more versions. I suspect increasing the timeout further (e.g. 60s) will likely prevent these issues.

[maxfisher-g]

Closing as remaining timeout issues will be dealt with by #337