Commit ab35362
committed
Stagger the scheduled runs so the scraper repos never overlap
Every scraper repo ran its daily and Monday checks at 06:00 UTC. On Monday that
is the full README matrix in all four at once, up to about sixteen jobs against
one account, which is the burst that returned "Your system is sending too many
of this type of request" during the LinkedIn Python build. Valid inputs fail
with that error, so the runs would go red with nothing wrong in the code.
The Monday runs now start at 06:00, 06:20, 06:40 and 07:00 across
instagram-scraper-node, linkedin-scraper-node, linkedin-scraper-python and
amazon-scraper-node. This one runs at 06:20 UTC, daily and Monday alike.
The Monday run is recognised by comparing github.event.schedule to the cron
string, in the run name and in the plan job. Both comparisons are updated with
the cron line. Changing only the cron would have turned Monday into a plain
daily check with nothing failing to say so.1 parent 0bb2403 commit ab35362
1 file changed
Lines changed: 9 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
30 | | - | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
| |||
255 | 258 | | |
256 | 259 | | |
257 | 260 | | |
258 | | - | |
| 261 | + | |
259 | 262 | | |
260 | 263 | | |
261 | 264 | | |
| |||
0 commit comments