feat: the APScheduler jobs (scrape every 8h, discover every 24h) run with no distributed lock — if docker-compose scale is used or the scheduler container restarts mid-cycle, duplicate jobs fire simultaneously and create duplicate database records

[prince-pokharna]

🐛 Problem Statement

Arachnode's scheduler/tasks.py uses APScheduler to trigger:

• Scrape cycle: every 8 hours
• Contact discovery: every 24 hours (offset by 4 hours)

The README notes:

"(offset by 4 hours to avoid deadlock throttling)"

This manual offset is a fragile workaround. If:

• The scheduler container restarts mid-cycle
• A user accidentally runs two scheduler instances (e.g., during docker compose up --scale scheduler=2)
• The container restarts exactly when a job is due

...multiple instances of the same job fire simultaneously. Both call the same POST /api/scrape → aggregator endpoint → PostgreSQL write, resulting in duplicate job records even with the deduplication logic.

💡 Proposed Fix

Use Redis-based distributed locking via redlock-py to ensure only one instance of each scheduled job runs at a time:

# scheduler/tasks.py

from redis import Redis
from redlock import Redlock

redis_client = Redis(host=os.environ.get("REDIS_HOST", "redis"), port=6379)
dlm = Redlock([redis_client])

SCRAPE_LOCK_TTL = 8 * 60 * 60 * 1000  # 8 hours in ms (matches scrape interval)

async def run_scrape_cycle():
    lock = dlm.lock("scheduler:scrape_cycle", SCRAPE_LOCK_TTL)
    if not lock:
        print("[Scheduler] Scrape cycle already running — skipping duplicate")
        return

    try:
        print("[Scheduler] Acquired scrape lock — starting cycle")
        async with httpx.AsyncClient() as client:
            await client.post(f"{GATEWAY_URL}/api/scrape", json={
                "role": os.environ.get("JOBSEEKER_ROLE", "Backend Engineer"),
                "platforms": ["wellfound", "yc", "remotive"],
            }, timeout=300.0)
    finally:
        dlm.unlock(lock)
        print("[Scheduler] Released scrape lock")

Also add a SCHEDULER_INSTANCE_ID environment variable so logs from multiple instances can be distinguished:

INSTANCE_ID = os.environ.get("SCHEDULER_INSTANCE_ID", "scheduler-0")
print(f"[{INSTANCE_ID}] Starting scheduler...")

📁 Files to Modify

File	Change
scheduler/tasks.py	Wrap each job with Redis distributed lock
scheduler/requirements.txt	Add redlock-py
docker-compose.yml	Add SCHEDULER_INSTANCE_ID env var
README.md	Document that running multiple scheduler replicas is now safe

Suggested labels: bug, scheduler, reliability

I would like to work on this. Could you please assign it to me?

[prince-pokharna]

Hey @sharmavaibhav31 ,
Please assign it to me under GSSoC 2026.
/claim /assign /claim-assignment