feat: add channel scraper, geocoding enhancement, and SonarCloud CI - #84
Merged
kelleyglenn merged 9 commits intoFeb 23, 2026
Merged
Conversation
- Add scripts/list-channel/ — YouTube channel video URL lister using yt-dlp with date/duration filtering (excludes Shorts). Output is compatible with extract.py --file. - Add streetAddress field to LLM extraction prompt, JSON schema, and output in extract.py. Claude provides street addresses for well-known government buildings, improving downstream geocoding accuracy. - Fix geocode response parsing bug in seed-videos.sh — was reading .latitude/.longitude but the API returns .coordinates.latitude/ .coordinates.longitude, causing geocoding to silently fail. - Prefer streetAddress over location name for geocoding queries in seed-videos.sh when available. - Update docs/llm-extraction-prompt.md and extract-metadata README with streetAddress documentation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract _build_ydl_opts() and _parse_entry() helpers to bring cognitive complexity from 22 to within Sonar's 15 threshold. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lity
- Add yt-dlp sleep options (sleep_requests, sleep_interval, sleep_subtitles)
to avoid triggering YouTube rate limits on subtitle fetches
- Add retry with backoff for subtitle URL fetch on HTTP 429
- Fix batch API custom_id to use video ID instead of full URL (must match
^[a-zA-Z0-9_-]{1,64})
- Switch list_channel to extract_flat mode for faster channel listing
- Move date filtering from yt-dlp DateRange to Python (fixes YYYY-MM-DD
format error and avoids per-video page fetches)
- Add seed-data/*.txt to .gitignore
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Separates the monolithic extraction script into two independent pipeline stages: YouTube metadata fetching (yt-dlp) and Claude LLM extraction. This allows re-running extraction without re-fetching from YouTube. Both scripts support --append for resumable batch processing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace python3 URL encoding with jq (@uri filter) — python3 is a non-functional WindowsApps stub on Windows, causing curl to receive unencoded URLs and exit with code 3 under set -euo pipefail - Require street address for geocoding fallback — generic name/city/state queries produce unreliable results - Skip video creation when no location available (locationId is required) - Handle location 409 by extracting existingLocationId from response - Always include locationId in video creation request body Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract helper functions across all three Python scripts to reduce SonarCloud cognitive complexity scores. Add incremental file writing to fetch_youtube.py so progress survives interruptions, and add --delay flag to throttle requests and avoid YouTube rate limiting. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --cookies-from-browser option to fetch_youtube.py for authenticated YouTube sessions (~6x higher rate limits). Extract helper functions from main() in both fetch_youtube.py and claude_extract.py to bring cognitive complexity under SonarCloud's threshold of 15. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kelleyglenn
deleted the
feature/channel-scraper-geocoding-enhancement
branch
February 23, 2026 03:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/list-channel/Python CLI to list video URLs from a YouTube channel using yt-dlp, with date/duration filtering and Shorts exclusion. Output is compatible withextract.py --file.streetAddressfield to the AI extraction prompt and preferring it over landmark names inseed-videos.shseed-videos.shto use.coordinates.latitude/.coordinates.longitudeTest plan
list_channel.pyagainst a known channel and verify URL outputextract.pyon a video and verifystreetAddressfield is populated when applicableseed-videos.shand verify geocoding usesstreetAddresswhen available🤖 Generated with Claude Code