Skip to content

Commit ab35362

Browse files
committed
Stagger the scheduled runs so the scraper repos never overlap
Every scraper repo ran its daily and Monday checks at 06:00 UTC. On Monday that is the full README matrix in all four at once, up to about sixteen jobs against one account, which is the burst that returned "Your system is sending too many of this type of request" during the LinkedIn Python build. Valid inputs fail with that error, so the runs would go red with nothing wrong in the code. The Monday runs now start at 06:00, 06:20, 06:40 and 07:00 across instagram-scraper-node, linkedin-scraper-node, linkedin-scraper-python and amazon-scraper-node. This one runs at 06:20 UTC, daily and Monday alike. The Monday run is recognised by comparing github.event.schedule to the cron string, in the run name and in the plan job. Both comparisons are updated with the cron line. Changing only the cron would have turned Monday into a plain daily check with nothing failing to say so.
1 parent 0bb2403 commit ab35362

1 file changed

Lines changed: 9 additions & 6 deletions

File tree

‎.github/workflows/live.yml‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@
66
# credit and about a minute, so a broken commit shows up at once.
77
# A newer push cancels a run still going for an older commit.
88
# by hand, from the Actions tab, to scrape whatever accounts you choose
9-
# every day at 06:00 UTC except Monday: one profile, the async
9+
# every day at 06:20 UTC except Monday: one profile, the async
1010
# path, and the field table, about two credits
11-
# every Monday at 06:00 UTC: the same, plus every code block in the README,
11+
# every Monday at 06:20 UTC: the same, plus every code block in the README,
1212
# about 35 credits and 30 minutes
1313
#
1414
# A slow day at the API shows up as a red run. Rerun it before reading more
@@ -20,14 +20,17 @@
2020
# the main repository commits; a copy of the repository just runs the checks.
2121
name: Live check
2222

23-
run-name: "${{ github.event.schedule == '0 6 * * 1' && 'Weekly check: does every README block still work?' || github.event_name == 'schedule' && 'Daily check: is anything broken?' || github.event_name == 'push' && format('Push check: {0}', github.event.head_commit.message) || format('Scrape {0}', inputs.profiles) }}"
23+
run-name: "${{ github.event.schedule == '20 6 * * 1' && 'Weekly check: does every README block still work?' || github.event_name == 'schedule' && 'Daily check: is anything broken?' || github.event_name == 'push' && format('Push check: {0}', github.event.head_commit.message) || format('Scrape {0}', inputs.profiles) }}"
2424

2525
on:
2626
push:
2727
branches: [main]
2828
schedule:
29-
- cron: "0 6 * * 0,2-6" # daily smoke, every day except Monday
30-
- cron: "0 6 * * 1" # Monday: the whole README
29+
- cron: "20 6 * * 0,2-6" # daily smoke, every day except Monday
30+
- cron: "20 6 * * 1" # Monday: the whole README. Staggered across the
31+
# scraper repos (06:00, 06:20, 06:40, 07:00) so four full matrices never
32+
# hit one account at once, which is what trips "too many requests".
33+
# The two `== '20 6 * * 1'` checks below must match this line exactly.
3134
workflow_dispatch:
3235
inputs:
3336
profiles:
@@ -255,7 +258,7 @@ jobs:
255258
# instead of the sum of all of them.
256259
plan:
257260
name: List the README blocks
258-
if: github.event.schedule == '0 6 * * 1' || inputs.validate == true
261+
if: github.event.schedule == '20 6 * * 1' || inputs.validate == true
259262
runs-on: ubuntu-latest
260263
timeout-minutes: 5
261264
outputs:

0 commit comments

Comments
 (0)