Repository navigation
feat(scrape): Stage B2 manual LLM filtering in the batch protocol - #189
Merged
Merged
Conversation
Stage B (NaiveBayes) lets through a long tail of keyword-rich spam (escort /
massage listings, ad and SEO-bait domains) whose title/snippet contain the
enforcement lexicon but which never yield an enforcement event. Stage B2 is a
mandatory judgment pass by the operating agent on the planned batch, before any
scrape budget is spent: review titles/snippets/domains and suppress clear junk.
Mechanism (src/denbust/discovery/manual_filter.py):
- suppress_candidates_b2(persistence, ids, note) — terminal SUPPRESSED with
b2_manual_filter / b2_reason provenance; skips missing + already-suppressed
- suppress_candidates_b2_by_domain(persistence, domains, note) — suppress all
still-scrapeable candidates on spam domains (subdomain-aware)
CLI:
denbust candidates-b2-suppress --ids a,b --domains spam1.com --note "..."
Docs:
- docs/batch_scraping_protocol.md — full end-to-end protocol incl. Stage B2
rule, decision heuristics, and per-batch reporting requirements
- AGENTS.md — Batch Scraping Protocol section making Stage B2 mandatory
- local_prefilter_cascade_design.md — Stage B2 cross-reference
Suppressed candidates leave the scrapeable pool permanently and never re-enter
a future balanced batch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR formalizes Stage B2 (manual LLM/agent filtering) as a mandatory pre-scrape step in the discovery batch protocol, adding durable suppression utilities (by candidate id or by spam domain) plus a CLI command and supporting documentation/tests.
Changes:
- Added
denbust.discovery.manual_filterwith durable Stage-B2 suppression helpers (--idsand subdomain-aware--domains). - Added
denbust candidates-b2-suppressCLI command to apply B2 suppressions against the configured discovery persistence. - Documented the batch scraping protocol (including mandatory Stage B2) in
docs/andAGENTS.md, and added unit tests for the new suppression behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/denbust/discovery/manual_filter.py |
Implements Stage B2 suppression transitions + metadata/provenance recording, including subdomain-aware domain suppression. |
src/denbust/cli.py |
Adds a candidates-b2-suppress Typer command to apply Stage B2 suppression via config-selected persistence. |
tests/unit/test_manual_filter.py |
Adds unit tests covering id suppression, domain suppression (including subdomains), and no-op behaviors. |
docs/batch_scraping_protocol.md |
Introduces the end-to-end batch protocol and makes Stage B2 a mandatory step with workflow guidance. |
docs/local_prefilter_cascade_design.md |
Cross-references Stage B2 in the broader prefilter/cascade design notes. |
AGENTS.md |
Adds a mandatory “Batch Scraping Protocol” section, including Stage B2 requirements and reporting expectations. |
Comment on lines
+72
to
+76
| for candidate_id in dict.fromkeys(candidate_ids): # de-dupe, preserve order | ||
| candidate = persistence.get_candidate(candidate_id) | ||
| if candidate is None or candidate.candidate_status is CandidateStatus.SUPPRESSED: | ||
| continue | ||
| suppressed.append(_b2_suppressed(candidate, note=note)) |
Comment on lines
+594
to
+596
| if not ids and not domains: | ||
| typer.echo("Provide --ids and/or --domains.") | ||
| raise typer.Exit(code=1) |
Comment on lines
+65
to
+66
| # Untouched candidate stays scrapeable. | ||
| assert store.get_candidate("a").candidate_status is CandidateStatus.NEW # type: ignore[union-attr] |
Comment on lines
+94
to
+98
| assert store.get_candidate("spam1").candidate_status is CandidateStatus.SUPPRESSED # type: ignore[union-attr] | ||
| assert store.get_candidate("spam2").candidate_status is CandidateStatus.SUPPRESSED # type: ignore[union-attr] | ||
| # Non-scrapeable status is left as-is; legitimate domain untouched. | ||
| assert store.get_candidate("already").candidate_status is CandidateStatus.SCRAPE_SUCCEEDED # type: ignore[union-attr] | ||
| assert store.get_candidate("keep").candidate_status is CandidateStatus.NEW # type: ignore[union-attr] |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #189 +/- ##
==========================================
+ Coverage 93.08% 93.09% +0.01%
==========================================
Files 79 80 +1
Lines 11884 11921 +37
==========================================
+ Hits 11062 11098 +36
- Misses 822 823 +1
🚀 New features to boost your workflow:
|
|
pr-agent-context report: This run includes unresolved review comments, a failing check, and patch coverage gaps on PR #189.
For each unresolved review comment, recommend one of: resolve as irrelevant, accept and implement
the recommended solution, open a separate issue and resolve as out-of-scope for this PR, accept and
implement a different solution, or resolve as already treated by the code.
After I reply with my decision per item, implement the accepted actions, resolve the corresponding
PR comments, fix the failing checks below, address the patch coverage gaps below, and push all of
these changes in a single commit.
# Copilot Comments
## COPILOT-1
Location: src/denbust/discovery/manual_filter.py:76
URL: https://github.com/DataHackIL/tfht_enforce_idx/pull/189#discussion_r3389047135
Root author: copilot-pull-request-reviewer
Comment:
`suppress_candidates_b2()` currently suppresses any non-missing candidate that isn't already `SUPPRESSED`, including candidates that are already terminal/non-scrapeable (e.g. `SCRAPE_SUCCEEDED`, `CLOSED`). Stage B2 is a pre-scrape step, so changing already-processed candidates to `SUPPRESSED` would corrupt history/state if an id is accidentally included. Align this with the domain-based suppressor by only allowing suppression for `SCRAPEABLE_CANDIDATE_STATUSES`.
## COPILOT-2
Location: src/denbust/cli.py:596
URL: https://github.com/DataHackIL/tfht_enforce_idx/pull/189#discussion_r3389047199
Root author: copilot-pull-request-reviewer
Comment:
For invalid option combinations this CLI generally raises `typer.BadParameter(...)` (which integrates with Typer's usage/error output). Here, echoing a message and exiting skips that formatting and is inconsistent with the rest of the file.
## COPILOT-3
Location: tests/unit/test_manual_filter.py:66
URL: https://github.com/DataHackIL/tfht_enforce_idx/pull/189#discussion_r3389047236
Root author: copilot-pull-request-reviewer
Comment:
Avoid `# type: ignore[union-attr]` here by asserting the candidate exists before reading `candidate_status`. Other discovery tests follow the `stored = ...; assert stored is not None` pattern, which keeps the test type-safe and clearer.
## COPILOT-4
Location: tests/unit/test_manual_filter.py:98
URL: https://github.com/DataHackIL/tfht_enforce_idx/pull/189#discussion_r3389047275
Root author: copilot-pull-request-reviewer
Comment:
Same as above: the `# type: ignore[union-attr]` assertions can be made type-safe by asserting the fetched candidate is not `None` before accessing `candidate_status`. This matches the style used elsewhere in the unit tests.
# Failing Checks
## FAIL-1
Type: Commit status
Context: pre-commit.ci - pr
Status: error
URL: https://results.pre-commit.ci/run/github/1159993403/1781100821.tS86nfCDS0mQGtYXdMY3Sg
Summary:
error during mergeable check
# Patch coverage
Patch test coverage is 61.29%; please raise it to 100%. These are the uncovered code lines:
- src/denbust/cli.py: 587, 588, 592, 594, 595, 596, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 614, 615
- src/denbust/discovery/manual_filter.py: 85Run metadata: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Formalises Stage B2 — manual LLM filtering as a mandatory step in the batch scraping protocol, with durable suppression tooling.
Stage B (NaiveBayes) is statistical and lets through keyword-rich spam — escort/massage listings, ad pages, SEO bait — whose title/snippet contain the enforcement lexicon (
זנות,עיסוי,ליווי…) but which never yield an enforcement event. Stage B2 is the agent's judgment pass on the planned batch, before scrape budget is spent.Mechanism —
denbust/discovery/manual_filter.pysuppress_candidates_b2(persistence, ids, note)— terminalSUPPRESSEDwithb2_manual_filter/b2_reasonprovenance; skips missing + already-suppressed.suppress_candidates_b2_by_domain(persistence, domains, note)— suppress all still-scrapeable candidates on spam domains (subdomain-aware).Suppressed candidates leave the scrapeable pool permanently and never re-enter a future balanced batch.
CLI
denbust candidates-b2-suppress --ids a,b --domains spam1.com --note "B2: escort spam"Docs
docs/batch_scraping_protocol.md— full end-to-end protocol: pipeline table, Stage B2 rule + decision heuristics, suppression workflow, per-batch reporting.AGENTS.md— new Batch Scraping Protocol section making Stage B2 mandatory.local_prefilter_cascade_design.md— Stage B2 cross-reference.Test plan
test_manual_filter.py): id suppression + provenance, skip missing/already-suppressed, subdomain domain matching, empty input.🤖 Generated with Claude Code