sharmavaibhav31 · ArpitKumar8649 · May 10, 2026 · May 11, 2026 · May 11, 2026 · May 11, 2026
diff --git a/aggregator-service/ARCHITECTURE.md b/aggregator-service/ARCHITECTURE.md
@@ -0,0 +1,47 @@
+# Aggregator — Architecture & Contributor Notes
+
+This file focuses on contributor-oriented/internal architecture information: request/data flow, internal execution pipeline, important modules/files, debugging notes, and service interactions.
+
+## Overview
+
+The Aggregator service consumes raw job events from a Redis stream, deduplicates and normalizes them, and persists canonical job records to PostgreSQL for downstream services.
+
+## Request / Data Flow
+
+1. Consumer reads from Redis `jobs:raw` stream (consumer groups) and claims pending messages when necessary.
+2. Each message is parsed and normalized (company, role, stack, posted_at) and passed to `db.insert_job()`.
+3. `db.insert_job()` attempts `INSERT ... ON CONFLICT DO NOTHING` to avoid duplicates; successful inserts set a deduplication key in Redis with a TTL.
+4. The service exposes `GET /jobs`, `GET /jobs/{id}`, `PATCH /jobs/{id}/status`, and `GET /stats` for querying aggregated data.
+
+## Internal Execution Pipeline
+
+- `consumer.py`: connection and consumer group management, `run_consumer()` loop, pending message claiming, and batch processing using `XREADGROUP`/`XAUTOCLAIM`.
+- `_process_message()`: normalizes payloads, computes dedup key (MD5 of normalized company+role), and invokes `db.insert_job()`.
+- `db.py`: asyncpg pool lifecycle, schema setup with GIN/BTREE indexes, and query helpers supporting ILIKE and array GIN containment for stack queries.
+
+## Important Modules / Files
+
+- `main.py` — FastAPI entry, endpoint handlers, lifespan hooks for pool and consumer task management.
+- `consumer.py` — Redis stream consumer and message processing logic.
+- `db.py` — database schema, pool, and CRUD helpers.
+- `models.py` — Pydantic response/request models.
+
+## Service Interactions
+
+- Consumes `jobs:raw` Redis stream emitted by scraper-service.
+- Persists canonical job records to PostgreSQL, read by contact-discovery and email-generator services.
+
+## Debugging Notes
+
+- Missing fields in messages are logged and skipped; add extra logging in `_process_message()` for edge cases.
+- Database or Redis connectivity issues surface during lifespan startup; verify pool/connection settings and credentials.
+- Query timeouts default to 30s; increase when running long analytical queries.
+
+## Contributor Tips
+
+- To test deduplication, craft messages with similar normalized company+role and observe Redis dedup key TTL behavior.
+- Add unit tests for `db.insert_job()` when changing schema or uniqueness constraints.
+
+## Next (Operational) Docs
+
+Operational/API usage (example requests, deployment env vars) are deferred to a follow-up docs issue.
diff --git a/aggregator-service/README.md b/aggregator-service/README.md
@@ -1,238 +1,9 @@
-# Job Aggregator Service
+# Aggregator Service
 
-A FastAPI microservice that consumes normalized job events from the `jobs:raw` Redis Stream (produced by the Scrapy crawler service), deduplicates them, persists to PostgreSQL, and exposes a REST API for querying and managing jobs.
+This repository contains the Aggregator service. Contributor-oriented architecture and developer notes are moved to a dedicated file to reduce duplication with other services.
 
-```
-┌─────────────────┐    Redis Stream     ┌───────────────────┐    asyncpg    ┌──────────────┐
-│  Crawler Service│ ──► jobs:raw ──────► │ Aggregator Service│ ────────────► │  PostgreSQL  │
-│  (Scrapy)       │                      │ (FastAPI + asyncio)│              │  jobs table  │
-└─────────────────┘                      └───────────────────┘              └──────────────┘
-```
+See the contributor-oriented material in: [aggregator-service/ARCHITECTURE.md](aggregator-service/ARCHITECTURE.md)
 
-## Tech stack
+Operational/API usage and setup details are deferred to a follow-up docs issue to avoid duplication across services.
 
-| Layer | Library |
-|-------|---------|
-| API framework | FastAPI + Uvicorn |
-| DB driver | asyncpg (raw SQL, no ORM) |
-| Redis client | redis-py (asyncio) |
-| Schemas | Pydantic v2 |
-| Runtime | Python 3.11 |
-
----
-
-## Project layout
-
-```
-aggregator-service/
-├── main.py          # FastAPI app, lifespan, endpoints
-├── db.py            # asyncpg pool, schema init, query helpers
-├── consumer.py      # Redis Stream consumer loop
-├── models.py        # Pydantic response schemas
-├── requirements.txt
-├── Dockerfile
-├── docker-compose.yml
-└── README.md
-```
-
----
-
-## Configuration (environment variables)
-
-| Variable | Default | Description |
-|---|---|---|
-| `REDIS_HOST` | `localhost` | Redis hostname |
-| `REDIS_PORT` | `6379` | Redis port |
-| `DATABASE_URL` | — | asyncpg DSN, e.g. `postgresql://user:pass@host:5432/db` |
-| `PORT` | `8000` | Port the API listens on |
-| `POSTGRES_USER` | `jobuser` | (docker-compose only) |
-| `POSTGRES_PASSWORD` | `jobpass` | (docker-compose only) |
-| `POSTGRES_DB` | `jobsdb` | (docker-compose only) |
-
----
-
-## Quick start
-
-### With Docker Compose (recommended)
-
-> Run from the `aggregator-service/` directory.
-
-```bash
-# 1. Copy and tweak env (optional — defaults work out of the box)
-cp .env.example .env    # edit JOBSEEKER_ROLE, JOBSEEKER_STACK, etc.
-
-# 2. Start everything (Redis + Postgres + Crawler + Aggregator)
-docker compose up --build
-
-# 3. Once the aggregator is healthy, fire the crawler once
-docker compose run --rm crawler scrapy crawl remotive
-```
-
-### Local development (no Docker)
-
-```bash
-cd aggregator-service
-python -m venv .venv && source .venv/bin/activate
-pip install -r requirements.txt
-
-export REDIS_HOST=localhost
-export REDIS_PORT=6379
-export DATABASE_URL="postgresql://jobuser:jobpass@localhost:5432/jobsdb"
-
-uvicorn main:app --reload --port 8000
-```
-
----
-
-## API Reference
-
-Interactive docs: <http://localhost:8000/docs>
-
-### `GET /health` — liveness probe
-
-```bash
-curl http://localhost:8000/health
-```
-```json
-{"status": "ok"}
-```
-
----
-
-### `GET /jobs` — list jobs
-
-| Query param | Type | Default | Description |
-|---|---|---|---|
-| `role` | string | — | Substring match on role title (case-insensitive) |
-| `stack` | string | — | Comma-separated tags; returns jobs whose stack contains **all** of them |
-| `status` | string | — | `new` \| `applied` \| `ignored` |
-| `sort` | string | `latest` | `latest` or `oldest` (by `posted_at`) |
-| `limit` | int | `50` | Max results (1–500) |
-
-```bash
-# All new jobs
-curl "http://localhost:8000/jobs?status=new"
-
-# Backend roles using Python, sorted oldest first
-curl "http://localhost:8000/jobs?role=backend&stack=Python&sort=oldest&limit=20"
-
-# Jobs that require both FastAPI and PostgreSQL
-curl "http://localhost:8000/jobs?stack=FastAPI,PostgreSQL"
-```
-
-<details>
-<summary>Example response</summary>
-
-```json
-[
-  {
-    "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
-    "company": "Acme Corp",
-    "role": "Senior Backend Engineer",
-    "source": "remotive",
-    "url": "https://remotive.com/jobs/123",
-    "stack": ["Python", "FastAPI", "PostgreSQL"],
-    "product": "Platform",
-    "location": "Remote",
-    "posted_at": "2026-03-18T10:00:00Z",
-    "status": "new",
-    "created_at": "2026-03-19T04:00:00Z"
-  }
-]
-```
-</details>
-
----
-
-### `GET /jobs/{id}` — get single job
-
-```bash
-curl http://localhost:8000/jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6
-```
-
-Returns `404` if the job is not found.
-
----
-
-### `PATCH /jobs/{id}/status` — update status
-
-Valid statuses: `new`, `applied`, `ignored`.
-
-```bash
-# Mark a job as applied
-curl -X PATCH http://localhost:8000/jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6/status \
-     -H "Content-Type: application/json" \
-     -d '{"status": "applied"}'
-
-# Ignore a job
-curl -X PATCH http://localhost:8000/jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6/status \
-     -H "Content-Type: application/json" \
-     -d '{"status": "ignored"}'
-```
-
-Returns the updated job record, or `404` if not found.
-
----
-
-### `GET /stats` — aggregate counts
-
-```bash
-curl http://localhost:8000/stats
-```
-```json
-{
-  "by_source": {
-    "remotive": 142,
-    "greenhouse": 37
-  },
-  "by_status": {
-    "new": 155,
-    "applied": 18,
-    "ignored": 6
-  }
-}
-```
-
----
-
-## How deduplication works
-
-Each incoming event is hashed using **MD5(normalised_company + "|" + normalised_role)**.
-
-- Before inserting, the service checks whether the Redis key `dedup:agg:{hash}` exists.
-- If it exists → event is ACK-ed and skipped.
-- If not → job is inserted into Postgres and the key is set with a **7-day TTL**.
-
-This prevents the same role from being re-inserted if the crawler runs multiple times per day.
-
----
-
-## Consumer group & crash recovery
-
-The service subscribes as consumer group `aggregator-group` on the `jobs:raw` stream.  
-On startup it runs `XAUTOCLAIM` to reclaim any messages that were delivered but never ACK-ed (e.g. after a crash), ensuring **at-least-once delivery**.
-
----
-
-## Database schema
-
-```sql
-CREATE TABLE jobs (
-  id         UUID        PRIMARY KEY DEFAULT gen_random_uuid(),
-  company    TEXT        NOT NULL,
-  role       TEXT        NOT NULL,
-  source     TEXT,
-  url        TEXT,
-  stack      TEXT[],
-  product    TEXT,
-  location   TEXT,
-  posted_at  TIMESTAMPTZ,
-  status     TEXT        NOT NULL DEFAULT 'new',
-  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
-);
-
--- Indexes
-CREATE INDEX idx_jobs_stack     ON jobs USING GIN (stack);
-CREATE INDEX idx_jobs_posted_at ON jobs (posted_at DESC NULLS LAST);
-CREATE INDEX idx_jobs_status    ON jobs (status);
-```
+If you need quick operational references, inspect `main.py`, `consumer.py`, and `db.py` directly.
diff --git a/contact-discovery-service/ARCHITECTURE.md b/contact-discovery-service/ARCHITECTURE.md
@@ -0,0 +1,50 @@
+# Contact Discovery — Architecture & Contributor Notes
+
+This file focuses on contributor-oriented/internal architecture information: request/data flow, internal execution pipeline, important modules/files, debugging notes, and service interactions.
+
+## Overview
+
+The Contact Discovery service attempts to discover likely contact emails for a company using public data sources, pattern inference, and SMTP verification, then persists verified contacts to PostgreSQL for downstream use.
+
+## Request / Data Flow
+
+1. `POST /discover` starts a multi-stage pipeline for a given company (optionally with `job_id` and roles).
+2. Pipeline stages: domain inference, email pattern detection, name discovery (GitHub/LinkedIn), email construction, SMTP verification.
+3. Verified contacts are upserted into PostgreSQL using `storage.upsert_contact()`.
+4. API supports querying contacts via `GET /contacts` and deletion via `DELETE /contacts/{id}`.
+
+## Internal Execution Pipeline
+
+- `discovery.py`: orchestrates domain inference, pattern mining, name discovery, and verification.
+- `domain_inference()`: Clearbit first, fallback to suffix probes and heuristics.
+- `email_pattern_detection()`: mines GitHub/org data and infers common patterns with regex.
+- `verifier.py`: MX lookup and SMTP RCPT checks with rate limiting.
+- `storage.py`: asyncpg pool and upsert logic with unique constraint on (company, email).
+
+## Important Modules / Files
+
+- `main.py` — FastAPI app and background task wiring.
+- `discovery.py` — multi-stage discovery pipeline.
+- `verifier.py` — SMTP verification utilities.
+- `storage.py` — DB pool and CRUD/upsert helpers.
+
+## Service Interactions
+
+- Reads job records from aggregator-service to link contacts to jobs.
+- Persists contacts to PostgreSQL for consumption by email-generator-service.
+- Calls external services: Clearbit, GitHub API, LinkedIn scraping, and MX hosts for verification.
+
+## Debugging Notes
+
+- Enable a `GITHUB_TOKEN` to avoid rate limits during name mining; failures are logged but pipeline continues where possible.
+- SMTP verification can be noisy; use logging in `verifier.py` to record MX/RCPT outcomes and rate-limiting events.
+- Database upsert conflicts are handled by `ON CONFLICT`; add targeted logging when debugging merges.
+
+## Contributor Tips
+
+- Write unit tests for pattern inference when modifying regex or GitHub parsing logic.
+- Mock SMTP and DNS responses for deterministic verification tests.
+
+## Next (Operational) Docs
+
+Operational/runbook content (third-party API keys, env vars, deployment) is deferred to a follow-up docs issue.