[shaypal5]

Summary

Open-web discovery on prostitution/escort/massage keywords inherently pulls in an unbounded tail of one-off spam domains (escort listings, massage ads — SEO-optimised for exactly those terms). 68% of all domains in the store are single-candidate, and a fresh batch appears every run, so the domain denylist can never keep up.

The domain-frequency gate flips the default from "scrape unless blocklisted" to "earn your way in by recurring": a candidate is held out of the batch unless its domain is a curated known outlet (always exempt) or its domain has recurred ≥ N times across the store. Real outlets recur and pass; one-off spam is held back — never deleted, it becomes eligible automatically if the domain ever recurs.

This is the right tool because the discriminator isn't "known vs unknown" (which would discard valuable off-list outlets) — it's "recurring vs single-shot."

Measured on the live 2026 pool (batch of 60)

Config	one-off junk	off-list recurring	known outlets
no gate	20 (33%)	16	24
N≥2	0	36	24

newsru(×8), mignews(×2), israel.com(×2) — the off-list sources of 5 of our 6 verified enforcement records — all survive the gate.

API

• Config.scrape_min_domain_frequency: int | None
• CLI: denbust run … --balanced-batch 60 --min-domain-frequency 2
• balanced_selection.py: KNOWN_SOURCE_FAMILIES, domain_frequencies, filter_by_domain_frequency
• Gate applies in _run_candidate_scrape_job (frequencies over the whole store), logs pass count.

Test plan

• 5 new unit tests (counts by family, holds single-shot, keeps recurring, exempts known, N≤1 no-op)
• 1343 unit tests pass; ruff + mypy clean

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds an opt-in “domain-frequency gate” to the scrape_candidates balanced-batch flow to hold back single-shot spam domains unless they recur across the candidate store, while exempting curated known outlets. This fits into the discovery/scrape pipeline as an additional pre-selection filter (after thin prefilter, before balanced batch planning) to reduce manual Stage B2 load and scrape budget waste.

Changes:

• Introduces domain recurrence counting (domain_frequencies) and gating (filter_by_domain_frequency) in balanced batch selection.
• Wires a new config/CLI option (scrape_min_domain_frequency / --min-domain-frequency) through CLI → config → pipeline, and logs gate pass rate.
• Adds unit tests for frequency counting and gate behavior; documents the new protocol in batch scraping docs.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/denbust/discovery/balanced_selection.py	Adds domain-frequency counting and filtering, plus a curated-known-outlet exemption set.
src/denbust/pipeline.py	Applies the gate in balanced-batch candidate selection; threads config through runner; validates CLI overrides.
src/denbust/config.py	Adds scrape_min_domain_frequency to the root config model.
src/denbust/cli.py	Exposes --min-domain-frequency and passes it through to run_job.
tests/unit/test_balanced_selection.py	Adds unit tests covering counting, gate thresholding, and known-outlet exemptions.
tests/unit/test_pipeline_core.py	Updates test fakes/expected kwargs to include the new parameter plumbed through the pipeline wrapper.
tests/unit/test_cli.py	Updates CLI wiring test fake signature to accept the new argument.
docs/batch_scraping_protocol.md	Documents the new domain-frequency gate and its intended operational use.

[codecov]

Codecov Report

❌ Patch coverage is 66.66667% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.03%. Comparing base (32531df) to head (eac5224).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/denbust/pipeline.py	18.18%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #194      +/-   ##
==========================================
- Coverage   93.09%   93.03%   -0.07%     
==========================================
  Files          80       80              
  Lines       11921    11947      +26     
==========================================
+ Hits        11098    11115      +17     
- Misses        823      832       +9     

Files with missing lines	Coverage Δ
src/denbust/discovery/balanced_selection.py	88.00% <100.00%> (+2.28%)	⬆️
src/denbust/pipeline.py	96.81% <18.18%> (-0.62%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

pr-agent-context report:

This run includes unresolved review comments, a failing check, and a patch coverage gap on PR #194.

For each unresolved review comment, recommend one of: resolve as irrelevant, accept and implement
the recommended solution, open a separate issue and resolve as out-of-scope for this PR, accept and
implement a different solution, or resolve as already treated by the code.

After I reply with my decision per item, implement the accepted actions, resolve the corresponding
PR comments, fix the failing checks below, address the patch coverage gaps below, and push all of
these changes in a single commit.

# Copilot Comments

## COPILOT-1
Location: src/denbust/pipeline.py:3380
URL: https://github.com/DataHackIL/tfht_enforce_idx/pull/194#discussion_r3391148399
Root author: copilot-pull-request-reviewer

Comment:
    `--min-domain-frequency` is documented as “Balanced-batch mode only”, but `_run_job_from_config` currently accepts it even when `--balanced-batch` is not provided (and the loaded config may also have no balanced batch size). In that case the setting is silently ignored later because `_run_candidate_scrape_job` only applies the gate when `balanced_batch_size is not None`.

    Consider failing fast here when min-domain-frequency is provided without balanced-batch being enabled (either via the flag or via `config.scrape_balanced_batch_size`) to avoid a confusing no-op CLI.

## COPILOT-2
Location: src/denbust/pipeline.py:1458
URL: https://github.com/DataHackIL/tfht_enforce_idx/pull/194#discussion_r3391148425
Root author: copilot-pull-request-reviewer

Comment:
    Computing `frequencies = domain_frequencies(persistence.list_candidates())` triggers a full read of the entire candidate store (and, for the Supabase-backed persistence, the REST call is `select=*` with no pagination). In larger stores this can become a significant latency/cost driver (and risks timeouts) for every balanced-batch run that enables the gate.

    Consider adding a persistence-layer aggregation (e.g., count-by-domain/source-key) or at least a lighter-weight fetch (select only `domain`/`source_hints`, potentially paginated) so the gate doesn’t require loading the full table into memory each run.

# Failing Checks

## FAIL-1
Type: Commit status
Context: pre-commit.ci - pr
Status: error
URL: https://results.pre-commit.ci/run/github/1159993403/1781121343.wnkA04SATqSTBVTVufIjWw

Summary:
    error during mergeable check

# Patch coverage

Patch test coverage is 67.86%; please raise it to 100%. These are the uncovered code lines:
- src/denbust/pipeline.py: 1456, 1457, 1458, 1459, 1464, 3377, 3378, 3379, 3380

Run metadata:

Tool ref: v4.0.19
Tool version: 4.0.19
Trigger: pull request opened
Workflow run: 27302347808 attempt 1
Comment timestamp: 2026-06-10T19:59:23.441189+00:00
PR head commit: eac522462ec1f1492e2e2eb2efadeec2dea97f19