Skip to content

feat: add confidence-based quality filters to seed-videos.sh - #91

Merged
kelleyglenn merged 1 commit into
masterfrom
feat/90-seed-data-quality-filters
Feb 25, 2026
Merged

feat: add confidence-based quality filters to seed-videos.sh#91
kelleyglenn merged 1 commit into
masterfrom
feat/90-seed-data-quality-filters

Conversation

@kelleyglenn

Copy link
Copy Markdown
Owner

Summary

Closes #90

Adds confidence-based filtering to seed-videos.sh so entries with unreliable metadata are automatically skipped during loading. Uses the confidence scores already produced by the AI extraction pipeline (claude_extract.py).

Filtering rules

  1. Location confidence: Skip below 0.6, unless the location has a name, street address, and coordinates (then allow down to 0.55 — enrichment-corroborated locations)
  2. Non-accountability content: Skip if both amendments confidence < 0.5 AND participants confidence < 0.8
  3. Video date confidence: Nullify dates with confidence < 0.5 (send video without date, don't exclude it)

Impact on current dataset (AP-after-2025-01-01)

Category Count
Loaded 420 (93%)
Filtered by location confidence 9
Filtered as non-accountability 2
Failed (no coordinates/participants) 21
Total 452

Test plan

  • Dry-run jq filter against full dataset — 28 entries filtered (matches expected)
  • Apple Store entry (0.55 with street address) correctly passes
  • Compilation/product videos correctly filtered
  • No bc dependency (uses jq for all float comparisons — works on Windows Git Bash)

🤖 Generated with Claude Code

Add filtering based on AI extraction confidence scores:
- Location: skip below 0.6, allow 0.55+ if corroborated by street address
- Non-accountability: skip if both amendments < 0.5 and participants < 0.8
- Video date: nullify dates with confidence < 0.5

Filters 32 of 452 entries (93% pass rate) from current dataset.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@kelleyglenn
kelleyglenn merged commit 1e82988 into master Feb 25, 2026
1 check passed
@kelleyglenn
kelleyglenn deleted the feat/90-seed-data-quality-filters branch February 25, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add confidence-based quality filters to seed-videos.sh

1 participant