feat: add channel scraper, geocoding enhancement, and SonarCloud CI

.github/workflows/sonar.yaml

@@ -0,0 +1,23 @@
+name: SonarCloud
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+permissions:
+  contents: read
+
+jobs:
+  sonar:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: SonarCloud analysis
+        uses: SonarSource/sonarqube-scan-action@689fb39b34b9aa95ebc5f8f119343ddd51542402 # v4
+        env:
+          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

.gitignore

@@ -49,6 +49,7 @@ infra/*.tfvars
 
 # Seed data (generated, keep local)
 seed-data/*.json
+seed-data/*.txt
 
 # AWS deployment config
 scripts/aws/config.env

README.md

@@ -17,6 +17,7 @@ AccountabilityAtlas/
 │   ├── seed-videos.sh                 # Seed videos from JSON into the running stack
 │   ├── aws/                           # AWS start, stop, and deploy scripts
 │   ├── extract-metadata/              # Python CLI for AI-powered video metadata extraction
+│   ├── list-channel/                  # Python CLI to list video URLs from a YouTube channel
 │   ├── lib/                           # Shared script utilities
 │   └── integration/                   # Cross-service integration tests
 ├── seed-data/                         # Generated seed data (JSON, not committed)
@@ -100,6 +101,7 @@ See [docs/07-InfrastructureArchitecture.md](docs/07-InfrastructureArchitecture.m
 
 The project includes tools for AI-powered extraction of video metadata (amendments, participants, location, date) using Claude:
 
+- **`scripts/list-channel/`** — Python CLI that lists video URLs from a YouTube channel using yt-dlp, with date and duration filtering (excludes Shorts). Output is compatible with `extract.py --file`. See [scripts/list-channel/README.md](scripts/list-channel/README.md).
 - **`scripts/extract-metadata/`** — Python CLI that uses yt-dlp and the Anthropic SDK to extract metadata from YouTube videos, including optional transcript analysis. See [scripts/extract-metadata/README.md](scripts/extract-metadata/README.md).
 - **`scripts/seed-videos.sh`** — Seeds videos from a JSON file into the running stack via the API. Reads output from the extract CLI.
 - **`/api/v1/videos/extract`** — Video-service REST endpoint for real-time extraction (title + description only, no transcript).

docs/llm-extraction-prompt.md

@@ -120,6 +120,9 @@ Extract location information where the encounter occurred.
 **Fields to extract:**
 - **name**: Location name such as a landmark (e.g., "Springfield City Hall") or street
   address. See special instructions below.
+- **streetAddress**: The street address of the named location (e.g., "800 E Monroe St").
+  If you know the physical street address from your training data, provide it. If you
+  are not confident, set to null. Do NOT fabricate addresses.
 - **city**: City name
 - **state**: State abbreviation (e.g., "CA", "TX")
 - **latitude** and **longitude**: Set these to null UNLESS they are explicitly stated in
@@ -154,6 +157,7 @@ The JSON structure:
   "videoDate": "YYYY-MM-DD or null",
   "location": {
     "name": "location name or null",
+    "streetAddress": "street address or null",
     "city": "city name or null",
     "state": "XX or null",
     "latitude": 0.0 or null,
@@ -247,6 +251,7 @@ The model's response contains XML thinking tags followed by the final JSON:
   "videoDate": null,
   "location": {
     "name": "City Hall",
+    "streetAddress": "800 E Monroe St",
     "city": "Springfield",
     "state": "IL",
     "latitude": null,
@@ -274,6 +279,7 @@ The service parses the last balanced `{...}` block from the response.
 | `videoDate` | `string \| null` | Yes | Date the incident occurred (ISO 8601 `YYYY-MM-DD`). `null` if not determinable. |
 | `location` | `object \| null` | Yes | Where the incident took place. `null` if not determinable. |
 | `location.name` | `string` | Yes* | Specific place name, prioritized: street address > specific landmark > general landmark. |
+| `location.streetAddress` | `string \| null` | Yes* | Street address of the named location (e.g., "800 E Monroe St"). `null` if unknown or not confident. |
 | `location.city` | `string \| null` | Yes* | City name. `null` if not determinable. |
 | `location.state` | `string \| null` | Yes* | US state abbreviation (e.g., "CA", "TX"). `null` if not determinable. |
 | `location.latitude` | `number \| null` | Yes* | Latitude. Always `null` unless explicitly stated in description text. |
@@ -334,3 +340,4 @@ Since the prompt instructs Claude to set latitude/longitude to `null` unless exp
 
 - **Web-app**: Calls the location-service geocode endpoint (`GET /locations/geocode?address=...`) using the extracted name/city/state
 - **Python CLI**: Uses the location-service geocode endpoint or a local geocoding library
+- **`streetAddress` improves geocoding precision**: When available, `streetAddress` should be preferred over `name` for geocoding queries. Street addresses like "800 E Monroe St, Springfield, IL" produce more accurate geocode results than landmark names like "City Hall, Springfield, IL".

scripts/extract-metadata/README.md

@@ -1,11 +1,16 @@
-# Video Metadata Extraction CLI
+# Video Metadata Extraction Pipeline
 
-A Python CLI tool that extracts structured metadata from YouTube videos for AccountabilityAtlas seed data. It uses `yt-dlp` to fetch video metadata and auto-generated transcripts, then calls Claude to extract amendments, participants, dates, and locations.
+A two-script Python pipeline that extracts structured metadata from YouTube videos for AccountabilityAtlas seed data.
+
+1. **`fetch_youtube.py`** — fetches video metadata and transcripts via yt-dlp, outputs intermediate JSON
+2. **`claude_extract.py`** — reads intermediate JSON, calls Claude to extract amendments/participants/dates/locations, outputs seed-data format JSON
+
+Splitting the pipeline lets each phase run independently. If the Claude prompt changes or extraction fails, you don't need to re-fetch from YouTube.
 
 ## Prerequisites
 
 - Python 3.10+
-- `ANTHROPIC_API_KEY` environment variable set with a valid Anthropic API key
+- `ANTHROPIC_API_KEY` environment variable set (only needed for `claude_extract.py`)
 
 ## Installation
 
@@ -22,80 +27,134 @@ source .venv/bin/activate  # On Windows Git Bash: source .venv/Scripts/activate
 pip install -r requirements.txt
 ```
 
-## Usage
-
-### Single URL
+## Pipeline Usage
 
-Prints JSON to stdout:
+### Full pipeline: URL list → YouTube data → seed data
 
 ```bash
-python extract.py "https://www.youtube.com/watch?v=VIDEO_ID"
-```
+# 1. (Optional) Generate URL list from a YouTube channel
+python ../list-channel/list_channel.py CHANNEL_ID > urls.txt
 
-### Bulk Processing
+# 2. Fetch YouTube metadata + transcripts
+python fetch_youtube.py --file urls.txt --output youtube-data.json
 
-Process a file of URLs (one per line) and write results to a JSON file:
+# 3. Extract structured metadata via Claude
+python claude_extract.py --input youtube-data.json --output seed-data/videos.json
+```
+
+### Single URL (quick test)
 
 ```bash
-python extract.py --file urls.txt --output seed-data/videos.json
+# Fetch metadata to stdout
+python fetch_youtube.py "https://www.youtube.com/watch?v=VIDEO_ID"
+
+# Or pipe directly to claude_extract.py
+python fetch_youtube.py "https://www.youtube.com/watch?v=VIDEO_ID" --output single.json
+python claude_extract.py --input single.json
 ```
 
-### Append to Existing File
+## fetch_youtube.py
 
-Add new entries to an existing JSON array file without overwriting:
+Fetches video metadata and auto-generated transcripts from YouTube using yt-dlp.
+
+### Usage
 
 ```bash
-python extract.py --file more-urls.txt --output seed-data/videos.json --append
-```
+# Single URL (prints JSON to stdout)
+python fetch_youtube.py "https://www.youtube.com/watch?v=VIDEO_ID"
 
-### Skip Transcript
+# Bulk from file
+python fetch_youtube.py --file urls.txt --output youtube-data.json
 
-Faster extraction using only title and description (lower confidence scores):
+# Skip transcripts (faster)
+python fetch_youtube.py --file urls.txt --output youtube-data.json --no-transcript
 
-```bash
-python extract.py --no-transcript "https://www.youtube.com/watch?v=VIDEO_ID"
+# Resume interrupted batch (skips already-fetched URLs)
+python fetch_youtube.py --file urls.txt --output youtube-data.json --append
 ```
 
-### Batch Processing
+### CLI Reference
 
-Use the [Message Batches API](https://docs.anthropic.com/en/docs/build-with-claude/batch-processing) for **50% cost savings** on bulk processing. Requests are submitted as a single batch and processed asynchronously:
+```
+usage: fetch_youtube.py [-h] [--file FILE] [--output OUTPUT] [--no-transcript]
+                        [--append]
+                        [url]
 
-```bash
-python extract.py --file urls.txt --output seed-data/videos.json --batch
+positional arguments:
+  url                   Single YouTube URL to process.
+
+options:
+  -h, --help            show this help message and exit
+  --file FILE, -f FILE  Path to a text file with one YouTube URL per line.
+  --output OUTPUT, -o OUTPUT
+                        Output file path for JSON results.
+  --no-transcript       Skip transcript fetch (faster, but less data for extraction).
+  --append, -a          Append to existing output file, skipping URLs already present.
 ```
 
-Batch processing may take minutes to hours depending on queue depth. The CLI polls for completion and prints progress updates. The output format is identical to sequential mode.
+### Intermediate JSON Format
 
-The `--batch` flag can be combined with other options:
+Output is a JSON array where each element has:
 
-```bash
-python extract.py --file urls.txt --output videos.json --batch --append --no-transcript
+```json
+{
+  "url": "https://www.youtube.com/watch?v=VIDEO_ID",
+  "title": "Video Title",
+  "description": "Full YouTube description",
+  "channel": "Channel Name",
+  "thumbnail": "https://i.ytimg.com/.../maxresdefault.jpg",
+  "duration": 1234,
+  "published": "20240315",
+  "transcript": "Full transcript text or null"
+}
 ```
 
-### Custom Model
+## claude_extract.py
 
-Override the default Claude model:
+Reads intermediate JSON from `fetch_youtube.py` and calls Claude to extract structured metadata in the seed-data format.
+
+### Usage
 
 ```bash
-python extract.py --model claude-sonnet-4-20250514 "https://www.youtube.com/watch?v=VIDEO_ID"
+# Sequential processing
+python claude_extract.py --input youtube-data.json --output seed-data/videos.json
+
+# Batch API (50% cost savings, async processing)
+python claude_extract.py --input youtube-data.json --output videos.json --batch
+
+# Resume interrupted extraction (skips already-processed URLs)
+python claude_extract.py --input youtube-data.json --output videos.json --append
+
+# Custom model
+python claude_extract.py --input youtube-data.json --output videos.json --model claude-sonnet-4-20250514
+
+# Combine flags
+python claude_extract.py --input youtube-data.json --output videos.json --batch --append
 ```
 
-## URL File Format
+Batch processing uses the [Message Batches API](https://docs.anthropic.com/en/docs/build-with-claude/batch-processing) and may take minutes to hours depending on queue depth. The CLI polls for completion and prints progress updates.
 
-One URL per line. Blank lines and lines starting with `#` are ignored:
+### CLI Reference
 
 ```
-# First Amendment audit videos
-https://www.youtube.com/watch?v=abc123
-https://www.youtube.com/watch?v=def456
+usage: claude_extract.py [-h] --input INPUT [--output OUTPUT] [--model MODEL]
+                         [--batch] [--append]
 
-# Police encounter videos
-https://www.youtube.com/watch?v=ghi789
+options:
+  -h, --help            show this help message and exit
+  --input INPUT, -i INPUT
+                        Input JSON file from fetch_youtube.py.
+  --output OUTPUT, -o OUTPUT
+                        Output file path for JSON results.
+  --model MODEL, -m MODEL
+                        Claude model to use (default: claude-haiku-4-5-20251001).
+  --batch, -b           Use the Message Batches API for 50% cost savings.
+  --append, -a          Append to existing output file, skipping URLs already present.
 ```
 
-## Output Format
+### Seed-Data Output Format
 
-Each entry in the output JSON array follows this schema:
+Each entry in the output JSON array:
 
 ```json
 {
@@ -110,6 +169,7 @@ Each entry in the output JSON array follows this schema:
   "videoDate": "2024-03-15",
   "location": {
     "name": "City Hall",
+    "streetAddress": "200 N Spring St",
     "city": "Los Angeles",
     "state": "CA",
     "latitude": null,
@@ -142,39 +202,37 @@ Each entry in the output JSON array follows this schema:
 | `amendments` | Constitutional amendments relevant to the video (e.g., `FIRST`, `FOURTH`) |
 | `participants` | Types of participants (e.g., `POLICE`, `CITIZEN`, `GOVERNMENT`) |
 | `videoDate` | Date of the incident (ISO 8601), or `null` if not determinable |
-| `location` | Location object with name, city, state, latitude, longitude; or `null` |
+| `location` | Location object with name, streetAddress, city, state, latitude, longitude; or `null` |
 | `confidence` | Confidence scores (0.0-1.0) for each extracted field |
 
 See [docs/llm-extraction-prompt.md](../../docs/llm-extraction-prompt.md) for the full extraction prompt specification, valid enum values, and extraction rules.
 
+## URL File Format
+
+One URL per line. Blank lines and lines starting with `#` are ignored:
+
+```
+# First Amendment audit videos
+https://www.youtube.com/watch?v=abc123
+https://www.youtube.com/watch?v=def456
+
+# Police encounter videos
+https://www.youtube.com/watch?v=ghi789
+```
+
 ## How It Works
 
+### fetch_youtube.py
+
 1. **Fetch metadata**: Uses the `yt-dlp` Python library to extract video title, description, publication date, channel, thumbnail, and duration without downloading the video.
 2. **Fetch transcript**: Optionally retrieves auto-generated English subtitles and parses them into plain text.
-3. **Call Claude**: Sends the extraction prompt with XML-tagged video data, following the shared extraction prompt spec from [`docs/llm-extraction-prompt.md`](../../docs/llm-extraction-prompt.md). In sequential mode, this is a user-only prompt (no system prompt) identical to the Java video-service. In batch mode (`--batch`), the shared instructions are sent as a system message with `cache_control` for prompt caching, and only the per-video data is in the user message. Claude responds with XML thinking tags (multi-step analysis) followed by the final JSON object.
-4. **Parse response**: Extracts the last balanced JSON object from the response (skipping the XML thinking tags), matching the Java service's parsing logic.
-5. **Combine results**: Merges YouTube metadata with Claude's extracted fields into the seed-data format.
-
-If a transcript is unavailable, the tool falls back to extracting from title and description only, which typically produces lower confidence scores.
+3. **Output**: Writes intermediate JSON with all YouTube data for downstream processing.
 
-## CLI Reference
+### claude_extract.py
 
-```
-usage: extract.py [-h] [--file FILE] [--output OUTPUT] [--model MODEL]
-                  [--no-transcript] [--append] [--batch]
-                  [url]
-
-positional arguments:
-  url                   Single YouTube URL to process.
+1. **Read input**: Loads intermediate JSON from `fetch_youtube.py`.
+2. **Call Claude**: Sends the extraction prompt with XML-tagged video data, following the shared extraction prompt spec from [`docs/llm-extraction-prompt.md`](../../docs/llm-extraction-prompt.md). In sequential mode, this is a user-only prompt (no system prompt) identical to the Java video-service. In batch mode (`--batch`), the shared instructions are sent as a system message with `cache_control` for prompt caching, and only the per-video data is in the user message. Claude responds with XML thinking tags (multi-step analysis) followed by the final JSON object.
+3. **Parse response**: Extracts the last balanced JSON object from the response (skipping the XML thinking tags), matching the Java service's parsing logic.
+4. **Combine results**: Merges YouTube metadata with Claude's extracted fields into the seed-data format.
 
-options:
-  -h, --help            show this help message and exit
-  --file FILE, -f FILE  Path to a text file with one YouTube URL per line.
-  --output OUTPUT, -o OUTPUT
-                        Output file path for JSON results.
-  --model MODEL, -m MODEL
-                        Claude model to use (default: claude-haiku-4-5-20251001).
-  --no-transcript       Skip transcript fetch (faster, less accurate).
-  --append, -a          Append to existing output file instead of overwriting.
-  --batch, -b           Use Message Batches API for 50% cost savings (requires --file).
-```
+If a transcript is unavailable, the tool falls back to extracting from title and description only, which typically produces lower confidence scores.