Codecov upload reports system is down

[jablonskipj]

While uploading reports to codecov it throws 503. Unfortunate situation since our CI is dependant on that software :(

[ogrisel]

We experienced many such errors since a couple of days. Generally a 502 error followed by 503 errors upon retry.

We assumed that those were caused by:

• https://status.codecov.com/incidents/5zxghg54j48t

and the subsequent:

• https://status.codecov.com/incidents/z3lnvr075nll

Today (Feb. 7th) the status page reports "All Systems Operational" but we still experienced the same error again (albeit at a lower frequency for now), for instance:

• https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=63618&view=logs&jobId=78a0bf4f-79e5-5387-94ec-13e67d216d6e&j=78a0bf4f-79e5-5387-94ec-13e67d216d6e&t=678c4faf-2449-5d3a-6566-e9ce3b54df82

[2024-02-07T10:17:10.706Z] ['verbose'] Start of uploader: 1707301030706...
[2024-02-07T10:17:10.710Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.7.1
[2024-02-07T10:17:10.710Z] ['info'] => Project root located at: /home/vsts/work/1/s
[2024-02-07T10:17:10.716Z] ['info'] ->  Token found by arguments
[2024-02-07T10:17:10.718Z] ['verbose'] Start of network processing...
[2024-02-07T10:17:10.718Z] ['verbose'] Searching for files in /home/vsts/work/1/s
[2024-02-07T10:17:10.732Z] ['verbose'] coveragepy is not installed
[2024-02-07T10:17:10.732Z] ['info'] Searching for coverage files...
[2024-02-07T10:17:10.838Z] ['verbose'] Preparing to clean the following coverage paths: coverage.xml,coverage.xml
[2024-02-07T10:17:10.838Z] ['info'] => Found 1 possible coverage files:
  coverage.xml
[2024-02-07T10:17:10.839Z] ['verbose'] End of network processing
[2024-02-07T10:17:10.839Z] ['info'] Processing /home/vsts/work/1/s/coverage.xml...
[2024-02-07T10:17:10.910Z] ['info'] Detected Azure Pipelines as the CI provider.
[2024-02-07T10:17:10.910Z] ['verbose'] -> Using the following env variables:
[2024-02-07T10:17:10.910Z] ['verbose']      BUILD_BUILDID: 63618
[2024-02-07T10:17:10.910Z] ['verbose']      BUILD_BUILDNUMBER: 20240207.17
[2024-02-07T10:17:10.910Z] ['verbose']      BUILD_SOURCEBRANCH: refs/pull/28378/merge
[2024-02-07T10:17:10.910Z] ['verbose']      BUILD_SOURCEVERSION: 6ea8e97c7cc8f9f44d3adaf186bbd81d57df1a41
[2024-02-07T10:17:10.910Z] ['verbose']      SYSTEM_PULLREQUEST_PULLREQUESTID: 1715243267
[2024-02-07T10:17:11.024Z] ['verbose'] The error stack is: Error: Error uploading to https://codecov.io/: Error: There was an error fetching the storage URL during POST: 503 - upstream connect error or disconnect/reset before headers. reset reason: connection failure
    at main (/snapshot/repo/dist/src/index.js)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-02-07T10:17:11.025Z] ['verbose'] End of uploader: 319 milliseconds

[...] followed by retries that all fail the same way.

Note that there are several codecov steps on the same CI build (different jobs) that did succeed almost concurrently to this error.

We suspected that our token was getting rate limited because of the retry mechanism. We rotated CODECOV_TOKEN once yesterday (or the day before) and it seems to have helped a bit but I am not so sure since it's failing again.

[ogrisel]

Thanks. Is this new planned maintenance related to this problem?

• https://status.codecov.com/incidents/hsc74mjj8jtk

[drazisil-codecov]

I think so. We have added a bunch of alerts to tell us when things go crazy again. Fingers crossed, I don't think it's happened today, has it?

[lesteve]

It is still happening from time to time for example 3 codecov upload errors in build log and below.

According to the log this happened today (February 8) around 6pm (I am guessing UTC) 2024-02-08T18:18:02.655Z

[drazisil-codecov]

Hi @lesteve I think that was during our maintenance window. https://status.codecov.com/incidents/hsc74mjj8jtk

[lesteve]

OK it seems to happen less frequently today it seems.

I guess we will report here, if we see the issue cropping up again, thanks for your answer!

[aj-codecov]

No activity in months, closing this out as any new or similar issues are likely different.