distributed-range-downloader

Distributed multi-device HTTP range downloader (early prototype).

Requirements

• Node.js (>= 20)
• npm

Install dependencies:

npm install

Phase 0 (single machine): probe + concurrent range download

Probe a URL for real range support (requires 206 for Range: bytes=0-0):

npm run phase0 -- probe <url>

Download a file by splitting into byte ranges and merging locally:

npm run phase0 -- download <url> --out /tmp/out.bin --concurrency 4 --chunkSize 1048576

Local origin server (range-capable fixture)

This is a local HTTP server that serves a deterministic binary fixture at /file and supports Range.

npm run origin -- --host 0.0.0.0 --port 7331 --size 67108864

• Health: http://<host>:7331/health
• File: http://<host>:7331/file

Phase 1 (prototype MVP): coordinator + workers (HTTP control plane)

Phase 1 uses:

• Control plane (HTTP): workers hello + requestWork
• Data plane (HTTP): workers upload chunk bytes to coordinator

Quickstart (recommended): phase1 script

Host (Computer A): start coordinator (and optional local origin fixture).

• Download a real URL (skips local origin):

npm run phase1 -- host --url "https://download.thinkbroadband.com/1GB.zip" --out "/tmp/thinkbroadband-1GB.zip" --key "devkey"

• Use the local deterministic origin fixture (no --url):

npm run phase1 -- host --out "/tmp/out.bin" --key "devkey"

The host prints:

• its detected LAN IP + health URLs
• the shared key used (--key), and
• an exact Computer B worker command you can copy/paste.

To have two workers across two computers (host is also a worker):

• On Computer A (another terminal):

npm run phase1 -- worker --coordinator "http://127.0.0.1:7332" --key "devkey" --id "w-a" --concurrency 1

• On Computer B: run the printed command (or replace <A_IP> yourself):

npm run phase1 -- worker --coordinator "http://<A_IP>:7332" --key "devkey" --id "w-b" --concurrency 1

Note: if you omit --key for phase1 host, it generates a random key and prints it.

End-to-end on one machine

Terminal 1 (origin):

npm run origin -- --host 127.0.0.1 --port 7331 --size 67108864

Terminal 2 (coordinator):

npm run coordinator -- start \
  --url http://127.0.0.1:7331/file \
  --out /tmp/out.bin \
  --host 127.0.0.1 --port 7332 \
  --key devkey \
  --chunkSize 1048576

The coordinator persists a JSON manifest at /tmp/out.bin.manifest.json and will resume from it on restart (assigned chunks are safely re-queued).

Terminal 3 (worker A):

npm run worker -- start --coordinator http://127.0.0.1:7332 --key devkey --id w1 --concurrency 1

Terminal 4 (worker B):

npm run worker -- start --coordinator http://127.0.0.1:7332 --key devkey --id w2 --concurrency 1

End-to-end on two computers (LAN)

Assume:

• Computer A runs origin + coordinator
• Computer B runs a worker

On Computer A, get its LAN IP (macOS):

ipconfig getifaddr en0

On Computer A (origin):

npm run origin -- --host 0.0.0.0 --port 7331 --size 67108864

On Computer A (coordinator), replace <A_IP>:

npm run coordinator -- start \
  --url http://<A_IP>:7331/file \
  --out /tmp/out.bin \
  --host 0.0.0.0 --port 7332 \
  --key devkey \
  --chunkSize 1048576

On Computer B (worker), replace <A_IP>:

npm run worker -- start --coordinator http://<A_IP>:7332 --key devkey --id w1 --concurrency 1

Optional: run a second worker on Computer A too:

npm run worker -- start --coordinator http://127.0.0.1:7332 --key devkey --id w2 --concurrency 1

Tests

Runs:

• Phase 0 sha256 correctness test
• Phase 1 coordinator+2 workers e2e sha256 correctness test

npm test