Module D — answer-side practice breakdown + Editorial dashboard#1
Merged
Conversation
Scope: practice extraction → aggregator → companion HDBSCAN → authority overlay → LLM narrative → beeswarm dashboard (per ADR 0001-0005). Pending grill items captured: - G8 sub-decisions (few-shot examples, refusal fallback, review pile, prompt version pinning) — deferred to next session - G11 probe-into-production milestones (D-1 through D-5) - G12 D-specific failure / reproducibility ops Interfaces to Module B (authority lookup) and Module C (query → canonical group) noted; can develop with stubs in parallel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sus clustering Probe pipeline (src/module_d/probe_d/) validated on the q54069253 "useState not reflecting immediately" canonical chain (2019 canonical pulled probe-local + 132 in-window duplicates from the main DB): - D-1 extraction: per-answer LLM -> practices list (M3), OpenAI strict structured output behind a provider-neutral seam; conditions tightened to stated caveats; evidence_type has real variance; empty list = no implementation practice. - D-2 clustering: LLM aggregator + companion. HDBSCAN-from-scratch degenerates on homogeneous short practices, so companion = matched-k agglomerative (embedding cross-check ARI ~0.38). k=3 co-association consensus defines the previously black-box majority vote: within-cluster agreement 0.99, head clusters rock-solid. - PLAN.md: captured grill decisions — Q2 (OpenAI strict + seam), Q3 (conditions as annotation), Q5 (stub = dup chain), Q6 (open-problem query framing + M3 schema), and the D-2 method findings. Deps: openai, pydantic, scikit-learn. Docs sync (CONTEXT/spec/ADR) still pending. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ADME Capture the grill + probe decisions into binding docs (history preserved: old ADRs get refinement pointers, new ADRs appended): - ADR 0006: query framing broadened to "open implementation problem -> surface community practices" (subsumes ADR 0005's implementation-choice persona). - ADR 0007: multi-practice extraction (M3); companion = matched-k agglomerative (HDBSCAN-from-scratch degenerates on homogeneous short practices); majority vote = k=3 co-association consensus; two agreement axes (LLM self-stability ~0.99 vs cross-method embedding ~0.38) reported separately. - ADR 0002/0004/0005: refinement pointers; conditions = displayed annotation, not boundary; per-query cost re-based to OpenAI GPT-5.4. - spec/CONTEXT/README: OpenAI provider + seam, multi-practice, companion method, ADR index 0001-0007. .env.example: OPENAI_API_KEY. PLAN: next-step refresh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tory, 3D cube showpiece Interactive prototypes (probe_d/viz/) that fix the dashboard default-view idiom: - force-floating circle-pack with organic gooey/metaball cluster blobs (boundary grows along its nodes, not a circle); drag a node / a whole bubble; click a practice -> its answers expand along a 2D continuous real-time axis (old->new) = primary temporal tool - 3D Space-Time Cube kept as demo showpiece only (orthographic, gated on 2019-22 backfill) - time = expandable dimension (demoted from colour); size proportional to log(vote); accepted = ring; long-tail collapsed; authority = vote proxy (pre-differentiator, age-confounded). Production dashboard leans D3/SVG; layout+animation deferred. Also lands the static D-5 probes (beeswarm / circle-pack / STC concept via matplotlib + circlify) and the frozen k=1 cluster assignment (d2_assignment.json). Docs synced: ADR 0004 amendment + PLAN Probe-D-5 / next-steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…backed) Promote D from probe scaffolding into a clean package. The dashboard reads only a normalized Breakdown contract; authority (Module B) and canonical grouping (Module C) enter through Protocol seams, so the SO-data proxies swap out for the real providers with zero change to contract/layout/render. - contract.py: normalized Breakdown (authority a first-class per-point field + provenance; answer cards for the detail panel) - ports.py: AuthorityProvider / CanonicalGroupProvider Protocols (the B/C seams) - providers/so_proxy.py: vote-proxy authority + duplicate-chain grouping (pre- differentiator, age-confounded; deleted wholesale when the real providers land) - layout.py / breakdown.py: circle-pack geometry + the seam integration point - dashboard/: build.py (proxy → viz_data.json) + render.py (force-floating organic bubbles · hover tooltip · click→original-answer · community card · ⏱ bubbles/timeline community-swimlane view with animated transitions) - data/: frozen D fixtures (extractions / clusters / canonical snapshot) Runs on the q54069253 slice (46 practices / 8 communities) with no API/DB calls. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…layout Replace the per-community trajectory (triggered by an unreliable double-click on a draggable blob) with a whole-group temporal overview, grounded in exploratory-dashboard UX principles (overview-first / details-on-demand; primary stays visible; minimize chrome): - top-center 🫧 bubbles | ⏱ timeline toggle (reliable single click) replaces double-click - timeline = community swimlanes: one lane per cluster along a shared bottom date axis, nodes compressed to fit, animated tween both directions (sim eases out, d3.timer eases in) - detail surfaces follow the primary-detail/drawer pattern: in timeline the answer panel insets the lanes' right edge (inline/push, lanes reflow beside it — never hidden), and the redundant community card is suppressed (lanes self-label) - removed the bottom-left legend + bottom-right caption (were occluding the lanes) - node drag unconstrained; stretchier gooey (stdDeviation 16) keeps blobs bridged Idiom/data/encoding decisions unchanged; this is layout + interaction only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ADR 0008 records the decoupling decision: AuthorityProvider (B) and
CanonicalGroupProvider (C) Protocol seams, a normalized Breakdown the dashboard
reads exclusively (authority a first-class field + provenance), SO-data proxies
quarantined in providers/so_proxy.py — swap = one provider + a build re-run.
- CONTEXT.md: dashboard surface terms ("original answer" = SO source vs our
"practice"; a Practice cluster is shown as a "community", flagged vs the SO-wide
community); the belonging unit is the Practice, so an answer with practices in
several clusters has no single home cluster; fixed the stale "one Practice"
relationship to "one or more" (ADR 0007); flagged the proxy authority/grouping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…orm scratch Color/type/texture identity for the Module D dashboard, converged via the brainstorming visual companion: warm paper + Fraunces/Newsreader/IBM Plex Mono + Paul-Tol muted (CVD-safe) data palette + ink/gold semantic accents. Re-skin only — data, encodings, interaction idiom, and B/C seams unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…detail - Visual identity "The Editorial · Ink" (spec): warm paper, Fraunces / Newsreader / IBM Plex Mono, Paul-Tol muted (CVD-safe) community palette, ink/gold semantic accents. Reskin only — idiom & B/C seams unchanged. - accepted no longer ringed in the bubble field (secondary, age-confounded signal); surfaced on hover + community/answer detail as a gold star / check. - Plumb conditions/evidence_type through contract -> breakdown -> build. - Answer panel: uniform practice cards (community dot + name, evidence chips, WHEN line from conditions); transient flash + auto-scroll to the clicked practice instead of a persistent highlight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The answer body is StackOverflow markdown, not HTML. Add a small escape-first markdown subset renderer (mdToHtml): fenced code -> <pre class="code">, inline code, bold, links, lists, blockquotes, paragraphs. Escape-first so any raw HTML in the body is shown literally, never executed. Code blocks read as code (mono, tinted, horizontally scrollable); prose flows as paragraphs — SO-like. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Fix stale font line: community names are Fraunces upright, not italic. - Add "Answer detail panel — practice-led" section (was only in code): contract plumbing of conditions/evidence_type, uniform practice cards (dot+name, no AI accent bar), evidence chips replacing the unclear "both", no persistent click highlight -> transient flash + auto-scroll, and SO-markdown body rendering. - Widen scope statement (now touches the data contract + answer panel), update mapping + deferred (syntax highlighting, richer markdown). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ran Step-1 extraction over the full q54069253 pool (21 canonical + 142 dup = 163 answers), replacing the 35-answer slice. 192 practices, 35 empty (21.5%, calibrates with slice's 20%), 48 multi-practice, $0.203. Finding: evidence_type flips from both-dominant on the slice (59%) to prose-dominant at full scale (119/192 = 62%) — the dup long-tail is thin low-score prose suggestions; code-rich answers are the canonical head the slice over-sampled. Three buckets stay non-degenerate, so evidence_type holds. conditions 15/192 (7.8%), still rare-but-real. Extraction artifact only; clusters.json / viz_data.json untouched (still the 46-pt fixture) — full-scale k=3 consensus clustering is the next step, pending a cost go-ahead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a build-shape writer to d2_consensus.py (emits {n, clusters:[{name,
members}], _note}; singletons omitted → long-tail), run k=3 co-association
consensus over the 192 practices, then promote + rebuild.
Results ($0.097): 24 multi-member consensus clusters + 15 singletons.
within-cluster agreement 0.94, unanimous 3/3 pairs 81%. Head is clean and
interpretable: useEffect-after-change (45) › local-computed-value (26) ›
functional-updater (16) › read-on-next-render (11) › useRef/async (8)…
Finding: cross-method ARI vs the embedding companion drops 0.38 (slice) →
0.255 (full) — reproducibility (LLM self-consistency 0.94) holds, but
embedding-vs-LLM agreement weakens as the borderline long-tail grows.
Falls in spec §4's "<0.5 → report & discuss" branch. The two agreement
axes must be reported separately.
viz_data.json now 192 pts / 23 clusters (two same-named consensus groups
merge by name in breakdown → 22 head + long-tail). force.html rerendered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y artifact Add d2_head_ari.py (reuses saved consensus assignment + re-embeds, no aggregator re-run) to test whether the full-scale cross-method ARI drop (0.38 slice -> 0.255 full) is long-tail-driven. It is not the tail — it is matched-k granularity. Restricting the same k=39 agglomerative partition to head subsets keeps ARI ~0.25 (n>=2 0.259, n>=3 0.255, n>=5 0.246): the 15 singletons inflate matched-k to 39, which over-splits even the big head clusters. Re-clustering only the 177 head points at head granularity (k=24) recovers ARI=0.422 — comparable to / slightly above the slice's 0.38. So the head breakdown IS recoverable under the embedding geometry (~0.42 moderate); 0.255 is a singleton-inflated pessimistic number, not method degradation at scale. Defensibility ARI should exclude singletons / match head granularity. Reproducibility (0.94) holds independently. (Still the 3-small stand-in, not local SBERT.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ecision ceiling)
Read all 24 head clusters' full member practices against the anchor
question (q54069253). Top ~8 are genuinely distinct, valid solution
approaches — the extract->cluster core works on quality answers. But ~1/3
of the head clusters are not clean solutions, from two root causes:
A. off-topic answers leak in via dup-chain over-pooling (controlled
input, event propagation, date validation, data-shape) — coherent
but a different React problem. Decisive evidence: these clusters are
almost all canonical=0 / score 0-1. Quantified evidence for the dup-
chain proxy's relevance-precision ceiling → motivates real Module C.
B. extraction noise on thin score-0 answers: content-free meta ("use
the approach shown"), per-case debug, vague grab-bags.
Records candidate levers that don't need C (relevance gate, score/
authority floor, meta/per-case filtering, authority-weighted view).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…iagnostic Verified A-type off-topic sourcing: of 13 off-topic practices, 10 come from 5 pure-offtopic dup questions (yield no good practice -> Module C can drop the whole question, its root-cause job) and 3 come from 2 mixed questions (which also yield GOOD practices, so C must keep them -> C structurally cannot touch these 3). Principle: off-topic can occur at practice granularity, finer than C's question granularity. So a practice-level relevance/quality filter is Module D's own job, independent of and complementary to C — not stealing C's role, but covering what C cannot see. The irreducibly-D core is all of B plus the 3 mixed-question A practices. D can also stand in for the 10, but their root cause stays C's; D can instead surface "whole question filtered out" as a mis-pooled signal to feed C. Also records the score-floor diagnostic: score>=1 kills 4/5 off-topic clusters while keeping all 8 good ones, but score>=2 also kills 2 valid low-score approaches — so a hard score floor conflates off-topic with niche-but-valid and is rejected in favor of relevance (per minority- position concern). Evidence question ids recorded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement the D-owned practice-level relevance + substance gate (d1b_gate.py, RelevanceGate schema, build_gate_messages). Per practice, two axes: relevant (drop only if off-topic to a different React subsystem) and substantive (drop only content-free placeholders / per-case one-offs); conservative, bias to keep. Iteration: a nano first pass framed relevance too narrowly and killed 34 valid approaches (incl the 985-score functional-updater). Broadening the canonical framing to "any state-handling approach a developer with this confusion would use" and moving to the mini tier cut GOOD-cluster casualties to 2, both defensible (placeholder phrasing / a TS typing fix). Result: 192 -> 146 practices (dropped A 12 / B 13 / TANG 9 / TAIL 10 / GOOD 2). Re-clustering the survivors: per-run counts [17,20,20] (was [34,33,45]), within-cluster agreement 0.94 -> 0.98, unanimous pairs 81% -> 94%, head clusters 24 -> 13 and every one is a genuine state-handling approach — the 5 off-topic phantom clusters and placeholder/per-case junk are gone, and removing the noise made clustering more stable. Dashboard rebuilt: 146 points / 13 clusters (also relieves the earlier "too many clusters / cramped timeline" feedback). d2_consensus.py takes an optional input path. data/extractions.json is now the post-gate 146; the raw 192 stays in probe_d/data/d1_extractions.json. Cost: gate $0.16 + recluster $0.065. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse Module D Step-1 (extract) + Step-1b (relevance/substance gate) into a single per-answer LLM call (d1_extract_gated.py). ~44% fewer answer-side calls and the gate now sees the full answer body. Kept = relevant AND substantive, applied as a deterministic Python post-filter so drops stay auditable. Validated vs the separate extract->gate baseline (quality read, not metric theater): no extraction-recall loss (230 vs 192 extracted), confirmed false-drops of core practices (functional updater, immutable update) recovered after prompt tuning, guards (placeholders / off-topic subsystems) still hold. evidence_type is no longer an LLM judgment: it is a structural property of the source answer (does it contain a code block?), computed deterministically at answer level (handles markdown code fences and DB [CODE] placeholders). Removed from the merged schema/prompt; existing outputs back-filled via d1_evidence_postprocess.py. d2_consensus.py default input flipped to d1_gated.json (merged is now canonical D1+D1b). Downstream re-cluster -> clusters.json / viz regeneration is deferred. New: d1_extract_gated.py, d1_ab_compare.py, d1_gated_check.py, d1_evidence_postprocess.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
D-4 narrative probe: a per-query pure function (signal table -> prose) that narrates the practice-breakdown shape as the spec's RQ2/RQ3/RQ4 for one query. - schema QueryNarrative (reasoning-first strict): shape convergent/mixed/divergent, authority_alignment, temporal w/ insufficient_coverage guard, headline + body - prompt build_narrative_messages + 2 hand-crafted few-shots - runner build_narrative.py (dry-run default; --run to call the LLM, ~$0.01) - authority attributed at ANSWER level, deduped WITHIN cluster (distinct_backing_answers + peak_vote); year + cross-year coverage from viz_data, no DB - agreement plumbing: d2_consensus writer now persists per-cluster agreement; backfilled clusters.json with the 13 values from the same run (no LLM re-run) First run mis-fired temporal=contested (outlier-stretched per-cluster year span overrode the histogram); fixed with a neutral CROSS-YEAR COVERAGE line, median-only per-cluster year, and a tightened temporal rule gating on query-level coverage only. shape made 3-way (runner-up-ratio based) after the first real query landed in the convergent/divergent grey zone. Re-run is faithful: mixed / dispersed / insufficient_coverage on the q54069253 anchor (146 practices / 13 clusters). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hboard Adopt the merged one-call extract+gate (d1_gated.json, 163 kept) as the live dashboard landscape, replacing the separate-pipeline 146 set. Finding is unchanged — useEffect-after-state-change still leads (48/163, 29%) over use-local-value (27), functional-updater (19), read-next-render (15); shape stays mixed/fragmented, within-cluster agreement 0.99 — so the cheaper merged pipeline loses nothing. Promote = copy (schemas identical): d1_gated.json -> data/extractions.json, d2_assignment.json -> data/clusters.json (d2_consensus now persists agreement). Then module_d.dashboard.build -> viz_data.json (163 pts / 12 shells) -> render. Note: two consensus clusters share the name "Direct set vs merge/immutability"; build_breakdown keys by name so they render as one n=8 head (11 named heads + long-tail). Accepted as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… D-4 The canonical artifacts were still from the old separate-gate run (146 practices) while the code had moved to the merged extract+gate pipeline (d1_gated). Promote d1_gated -> extractions.json, recluster (d2_consensus writer now persists per-cluster agreement automatically, superseding the manual backfill), rebuild viz_data, and re-verify D-4 narrative end-to-end. Code == data now self-consistent. D-4 narrative re-runs clean and faithful on the realigned data (same shape: mixed / dispersed / insufficient_coverage; guards still fire). Exact counts and cluster numbers will be re-recorded at the cross-module integration phase — the goal here is only that the pipeline runs end-to-end with sound results. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the D-4 per-query narrative on the preview dashboard via a producer→consumer seam, so the dashboard build stays deterministic/free and the narrative is a separate cheap LLM step: build_narrative.py --run → persists data/query_narrative.json build.py → folds it into Breakdown.narrative contract.to_dict() → emits viz_data.meta.narrative render.py → renders it (no-ops when meta.narrative null) Rendering idiom = "The Editorial": on load the narrative plays as an overture (dimmed bubble field + staggered reveal) and recedes to a top standfirst ribbon. Hierarchy = kicker (context) → query (the topic, hero) → headline (the LLM take, italic dek) → lede → three RQ verdict chips, with the honesty guards (insufficient_coverage / inconclusive) rendered visibly tentative (dashed, muted). bubbles/timeline switch moved to a bottom-center pill to free the top; timeline axis lifted to clear it. query / group_size are Module C placeholders (real canonical title from canonical_q54069253.json; group_size = distinct questions contributing answers). query_narrative.json is genuine D-4 output (regenerate via --run). Docstrings refreshed to the merged 163/12 landscape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Declutter probe_d/ before merging Module D with the other modules.
Everything removed is either superseded or regeneratable; git history
retains it, and the re-run path reproduces all data artifacts.
Removed:
- old/separated-path + diagnostic probes: d1b_gate, d2_cluster (old k=1),
d1_ab_compare, d1_gated_check, d2_head_ari, d1_evidence_postprocess
(evidence_type is now deterministic in d1_extract_gated)
- viz idiom explorations: d5_{beeswarm,circlepack,stc_concept} + the whole
probe_d/viz/ tree — all superseded by dashboard/{build,render}.py
- probe_d/data/* : duplicates of the promoted module_d/data/ fixtures
(d1_gated==extractions, d2_assignment==clusters, canonical dup) plus
stale intermediates (d1_extractions, d1_filtered, d1_gated_full) and
d5 PNGs (~1.3MB total)
Kept (production spine): schemas/prompts/llm, d1_extract.py (shares
build_slice/load_answers with d1_extract_gated), d1_extract_gated,
d2_consensus, build_narrative, fetch_canonical; and the 4 promoted
fixtures in module_d/data/ so the dashboard builds without re-spending
on extraction. Smoke-tested: production imports clean, build+render OK.
Re-run order to regenerate: d1_extract_gated --run -> d2_consensus --run
-> promote -> dashboard.build -> dashboard.render -> build_narrative --run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds Module D end-to-end: extracting answer-level implementation practices, clustering them into community approaches (with k=3 consensus), generating a per-query narrative, and rendering an interactive “Editorial · Ink” dashboard behind Module B/C provider seams.
Changes:
- Introduces provider-agnostic ports + normalized
Breakdowncontract, with SO-proxy providers for authority + canonical grouping. - Adds Probe D pipeline scripts (extract+gate, consensus clustering, narrative builder) and commits initial artifacts for the anchor canonical chain.
- Implements the dashboard data builder + HTML renderer (force “organic bubbles” + timeline + narrative overture).
Reviewed changes
Copilot reviewed 33 out of 39 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/module_d/init.py | Module D package intro and seam summary. |
| src/module_d/PLAN.md | Detailed design/probe notes and pipeline plan for Module D. |
| src/module_d/ports.py | Protocol seams for Module B authority and Module C canonical grouping. |
| src/module_d/providers/init.py | Providers package initializer. |
| src/module_d/providers/so_proxy.py | Proxy implementations for authority scoring + SO duplicate-chain grouping. |
| src/module_d/contract.py | Normalized dashboard contract (Breakdown, points, clusters, answer cards). |
| src/module_d/breakdown.py | Integration: joins extractions+clusters with providers, collapses long-tail, lays out. |
| src/module_d/layout.py | Circle-pack geometry for clusters/points sized by authority. |
| src/module_d/dashboard/init.py | Dashboard package initializer. |
| src/module_d/dashboard/build.py | Builds viz_data.json from providers + offline artifacts; optionally injects narrative. |
| src/module_d/dashboard/render.py | Renders the interactive HTML dashboard from viz_data.json. |
| src/module_d/probe_d/schemas.py | Pydantic schemas for extraction/gating/aggregation/narrative structured outputs. |
| src/module_d/probe_d/prompts.py | Prompts + few-shots for extraction, gate, aggregator, and narrative. |
| src/module_d/probe_d/llm.py | Provider-neutral LLM seam with OpenAI strict-structured-output adapter + usage tracking. |
| src/module_d/probe_d/fetch_canonical.py | Builds the out-of-window canonical fixture via StackExchange API. |
| src/module_d/probe_d/d1_extract.py | Probe D-1 extractor runner (baseline, pre-merged gating). |
| src/module_d/probe_d/d1_extract_gated.py | Probe D-1 merged extract+gate runner with deterministic evidence-type detection. |
| src/module_d/probe_d/d2_consensus.py | Probe D-2 consensus clustering via co-association connected-components. |
| src/module_d/probe_d/build_narrative.py | Probe D-4 narrative builder from precomputed signal tables (dry-run by default). |
| src/module_d/data/canonical_q54069253.json | Committed canonical fixture for the anchor chain. |
| src/module_d/data/clusters.json | Committed consensus cluster assignments (+ agreement) for the anchor run. |
| src/module_d/data/query_narrative.json | Committed per-query narrative output (placeholder query/group_size noted). |
| README.md | Updates ADR references to include new ADRs (0006/0007). |
| pyproject.toml | Adds Python dependencies needed by Module D and probes (OpenAI, Pydantic, sklearn, etc.). |
| docs/spec.md | Spec updates reflecting ADR 0006/0007 refinements and OpenAI strict structured outputs. |
| docs/superpowers/specs/2026-06-01-dashboard-editorial-ink-design.md | Design spec for “The Editorial · Ink” dashboard styling and panel behavior. |
| docs/adr/0002-practice-clustering-hierarchical-map-reduce.md | Updates ADR 0002 to reflect ADR 0007 refinements and revised cost assumptions. |
| docs/adr/0004-default-view-beeswarm-with-overlay.md | Amendment documenting the interactive viz idiom exploration and updated default view. |
| docs/adr/0005-demo-query-selection-stratified-hybrid.md | Notes persona broadening via ADR 0006. |
| docs/adr/0006-query-framing-open-implementation-problem.md | New ADR defining broadened “open implementation problem” query framing. |
| docs/adr/0007-extraction-multipractice-clustering-method.md | New ADR locking multi-practice extraction + matched-k companion + co-association voting. |
| docs/adr/0008-module-d-bc-data-contract-provider-seams.md | New ADR defining B/C seams + normalized contract for dashboard independence. |
| CONTEXT.md | Terminology updates (query framing, practice plurality, dashboard proxy caveats). |
| .env.example | Adds OPENAI_API_KEY for Module D LLM roles. |
| .gitignore | Ignores .superpowers/ local brainstorming artifacts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+33
to
+38
| # flat practice index -> cluster name, in extraction order (the order clustering was run on) | ||
| cluster_of_flat: dict[int, str] = {} | ||
| for c in assignment: | ||
| for i in c["members"]: | ||
| cluster_of_flat[i] = c["name"] | ||
| head = {name for name, n in Counter(cluster_of_flat.values()).items() if n >= min_cluster} |
Comment on lines
+61
to
+72
| dup_qids = snap["dup_question_ids"] | ||
| placeholders = ",".join("?" * len(dup_qids)) | ||
| rows = con.execute( | ||
| f"SELECT answer_id, score, is_accepted, creation_date, owner_user_id, " | ||
| f"body_text, owner_display_name " | ||
| f"FROM answers WHERE question_id IN ({placeholders})", | ||
| dup_qids, | ||
| ) | ||
| answers += [ | ||
| Answer(aid, score, bool(acc), "dup", cdate, reputation.get(owner), body, author) | ||
| for aid, score, acc, cdate, owner, body, author in rows | ||
| ] |
Comment on lines
+5
to
+6
| RAG+gate). Both are meant to be deleted wholesale once the real providers land — nothing else in | ||
| Module D imports this file, so the swap is local. |
Comment on lines
+46
to
+53
| def _load_key() -> str: | ||
| key = os.environ.get("OPENAI_API_KEY") | ||
| if key: | ||
| return key | ||
| for line in (ROOT / ".env").read_text().splitlines(): | ||
| if line.startswith("OPENAI_API_KEY"): | ||
| return line.split("=", 1)[1].strip() | ||
| raise SystemExit("OPENAI_API_KEY not found (env or .env)") |
Comment on lines
+46
to
+47
| url = f"{SE_BASE}{path}?{urllib.parse.urlencode(params)}" | ||
| data = json.load(urllib.request.urlopen(url)) |
Comment on lines
+20
to
+22
| for shell in breakdown.clusters: | ||
| kids = [{"id": id(p), "datum": max(p.authority, 1e-6)} for p in pts_by_cluster[shell.id]] | ||
| data.append({"id": shell.id, "datum": sum(k["datum"] for k in kids), "children": kids}) |
Comment on lines
+203
to
+205
| <script type="module"> | ||
| import * as d3 from 'https://esm.sh/d3@7'; | ||
| if (window.__fb) { console.warn('already initialized; skipping duplicate run'); } |
Comment on lines
+25
to
+29
| def load_key() -> str: | ||
| for line in (ROOT / ".env").read_text().splitlines(): | ||
| if line.startswith("SE_API_KEY"): | ||
| return line.split("=", 1)[1].strip() | ||
| raise SystemExit("SE_API_KEY not found in .env") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this delivers
Module D: for a canonical question group, extract the implementation practices from its answers, cluster them into the community's distinct approaches, and present the breakdown on an interactive dashboard with a per-query narrative.
Pipeline (offline)
d1_extract_gated.py) — one LLM call per answer extracts practices and inline-gates them for relevance/substance;evidence_typecomputed deterministically.d2_consensus.py) — k=3 aggregator + co-association majority vote → label-alignment-free consensus clusters, each with an agreement score. Current landscape: 163 practices → 12 head clusters.build_narrative.py) — per-query 2–3 sentence shape description (RQ2 convergent/mixed/divergent · RQ3 authority · RQ4 temporal), with two honesty guards welded into the schema (insufficient_coverage,inconclusive).Dashboard (
dashboard/build.py→render.py)B/C-decoupled seams (ADR 0008)
ports.py+contract.py: authority (Module B) and canonical grouping (Module C) plug in behind a provider seam — swapping a provider never touches the dashboard. Currently backed by SO-proxy stubs (vote-proxy authority, dup-chain grouping).query/group_sizeinbuild_narrative.py; the dashboard no-ops ifmeta.narrativeis null.Notes
stage3/+phase_1/work onmain(separate dirs) → expected clean merge.d1_extract_gated --run → d2_consensus --run → promote → dashboard.build → dashboard.render → build_narrative --run..env/ main DB are gitignored.🤖 Generated with Claude Code