NordCrawl

LLM-ready batch crawling for Nordic company data.

NordCrawl crawls CSV/JSON URL lists, extracts structured Swedish company data, and can upsert the results directly into Supabase.

If this saves you time, star the repo — it helps the project grow and tells me to keep polishing it.

Why people star projects like this

• Instantly useful: real Swedish company extraction, not a toy demo
• Great first impression: rich CLI output, demo assets, examples, and docs
• Production-minded: retries, dedupe, rate limiting, and validation
• Easy adoption: pip install nordcrawl after release and clean CLI entry points

Features

• Batch crawling with configurable concurrency
• Per-domain rate limiting
• Retry logic with exponential backoff
• Swedish company data extraction
  • company name
  • org number
  • email
  • phone
  • address
• Consistent JSON schema via Pydantic
• CSV/JSON import and export
• Supabase upserts with dedupe by URL
• CLI-first workflow with a clean Python API

PyPI

The package name is nordcrawl.

After publishing, install it with:

pip install nordcrawl

Demo

The screenshot above is based on a real crawl run with Rich output.

• Demo input: examples/urls.json
• Verified output: examples/results.json
• More details: examples/README.md

Table of contents

• Why people star projects like this
• Features
• PyPI
• Demo
• Quick start
• CLI
• Python API
• Input formats
• Supabase table
• Example output
• Roadmap
• Contributing
• License

Quick start

1) Install dependencies

pip install -r requirements.txt

2) Configure environment

Copy the example file and fill in your Supabase keys:

cp .env.example .env

3) Run a crawl

python3 main.py crawl-urls \
  https://www.ikea.com/se/sv/ \
  https://www.hemkop.se/ \
  https://www.ica.se/ \
  --output results.json

CLI

Crawl from file

python3 main.py crawl data/test_urls.json --output results.json --format json

Save directly to Supabase

python3 main.py crawl data/test_urls.json --save-db

Show stats

python3 main.py stats --limit 50

Search stored companies

python3 main.py search "IKEA" --limit 20

Export from Supabase

python3 main.py export company_data.csv --format csv

Python API

import asyncio
from src import BatchCrawler, SupabaseManager

async def main() -> None:
    crawler = BatchCrawler(max_concurrent=5, rate_limit_delay=2.0)
    results = await crawler.crawl_urls([
        "https://www.ikea.com/se/sv/",
        "https://www.hemkop.se/",
        "https://www.ica.se/",
    ])

    print(results)

asyncio.run(main())

Input formats

CSV

url
https://www.example.se
https://www.example2.se

JSON

[
  "https://www.example.se",
  {"url": "https://www.example2.se"}
]

Supabase table

NordCrawl expects a table named company_data with these columns:

• url
• company_name
• org_number
• email
• phone
• address
• postal_code
• city
• website
• crawled_at
• status
• raw_html_hash
• extraction_confidence
• created_at
• updated_at

The migration is included here:

supabase/migrations/20240325000001_create_company_data_table.sql

Example output

{
  "url": "https://www.example.se",
  "company_name": "Exempel AB",
  "org_number": "556123-4567",
  "email": "info@example.se",
  "phone": "+46701234567",
  "address": "Storgatan 1",
  "postal_code": "123 45",
  "city": "Stockholm",
  "status": "completed",
  "extraction_confidence": 0.85
}

Docker / Supabase note

Local supabase start requires Docker access. If you do not want to run Supabase locally:

1. Apply the migration in your remote Supabase project.
2. Set SUPABASE_URL and SUPABASE_KEY in .env.
3. Run the crawler with --save-db.

Project structure

.
├── main.py
├── requirements.txt
├── src/
│   ├── api/
│   ├── crawlers/
│   ├── database/
│   ├── extractors/
│   └── models/
├── supabase/
│   └── migrations/
└── tests/

Roadmap

• Better selector coverage for Swedish business websites
• Richer address parsing
• Supabase auth-aware dashboard
• Web UI for crawl jobs
• More sample datasets and benchmarks

Support

• Questions and ideas: open a GitHub Discussion
• Bug reports: use the bug template in .github/ISSUE_TEMPLATE/bug_report.md
• Feature requests: use the feature template in .github/ISSUE_TEMPLATE/feature_request.md

Community

• Contributing guide
• Code of conduct
• Security policy
• Examples

Contributing

PRs are welcome. Please read the contributing guide before opening a pull request.

1. Fork the repo
2. Create a branch
3. Add tests
4. Open a pull request

License

Apache-2.0

---

If you like the project, star it on GitHub to help it grow.