Cache error while uploading object: RequestTimeTooSkewed

[kelson42]

From https://farm.openzim.org/pipeline/845ea49e-0d80-4d92-8127-1afce890c276/debug

[kelson42]

@audiodude @benoit74 Do I understand properly: object storage refuses to work properly because clock (mwoffliner4) is not in sync (with server)?

[benoit74]

Re-occurence of #2092

I was not convinced at all by the solution implemented in #2093 (adding more sockets is just pushing the limit a bit further, not solving the real problem). This seems to indicate the problem is indeed not solved and we need to rearchitecture the software.

[kelson42]

@audiodude Any idea? The error is systematic, at least for Wikivoyage. Maybe related to #2117 ?

[audiodude]

I imagine @benoit74 is correct, this is another instance of requests waiting in a queue for much too long before actually being resolved. I'm not convinced it's the exact same code path on the mwoffliner side, but the result is the same, it's the smithy library, which is used by aws-sdk, which is failing. And yes, if bumping the socket number had been an adequate fix, we wouldn't be running into this again.

As far as #2117, it could in fact be that using the wrong upload method is causing this issue to be exacerbated (possibly by causing many many more superfluous requests), and that if we made the switch the error message suggests we could alleviate the problem.

Unfortunately, this only happens on zimfarm so it's hard to develop against/test.

[benoit74]

Unfortunately, this only happens on zimfarm so it's hard to develop against/test.

Probably that using an S3 optimization URL locally would make this happen also locally

[audiodude]

Unfortunately, this only happens on zimfarm so it's hard to develop against/test.

Probably that using an S3 optimization URL locally would make this happen also locally

Right but in the past, I couldn't reproduce the RequestTimeTooSkewed error.

[kelson42]

@audiodude Ok, but you don't have attempt to scrape the wikivoyage twice like the recipe does!? Do you?

[audiodude]

@kelson42 All I know is that I ran this command:

npm run build && npm run mwoffliner -- --webp --mwUrl="https://en.wikivoyage.org" --format="nopic:nopic" --format="novid:maxi" --osTmpDir="/dev/shm" --adminEmail="[email protected]" --customMainPage="Wikivoyage:Offline_reader_Expedition/Home_page" --customZimTitle="Wikivoyage" --outputDirectory="output" --customZimDescription="The collaborative travel guide" --optimisationCacheUrl="https://s3.us-west-1.wasabisys.com/?keyId=I9BI3Q2WJ228SZMTZIMJ&secretAccessKey=Ot8ldPPetvDEX5i7nCCuA2e5oYqa2prsmXGL0Fnb&bucketName=org-kiwix-dev-mwoffliner" --publisher="openZIM" --verbose

and it completed successfully

[benoit74]

Never publish secrets like a secreteAccessKey to github ... I disabled the access key and generated a new one which I sent through Slack

[audiodude]

Oh wow, sorry, I didn't realize I had done that. I just copy pasted the command line, I forgot it had the secret embedded in it. Thanks, sorry!