Skip to content

feat: add minority opinion detection for ranking disagreements#129

Open
bledden wants to merge 3 commits intokarpathy:masterfrom
bledden:feature-minority-opinions
Open

feat: add minority opinion detection for ranking disagreements#129
bledden wants to merge 3 commits intokarpathy:masterfrom
bledden:feature-minority-opinions

Conversation

@bledden
Copy link
Copy Markdown

@bledden bledden commented Jan 6, 2026

Summary

Adds minority opinion detection to flag when a significant portion of council members disagree with the consensus ranking.

  • Detection algorithm: Flags when ≥30% of rankers place a model more than 1 position away from its tournament-based consensus position
  • Direction indication: Shows whether dissenters think the model is overvalued (ranked too high) or undervalued (ranked too low) by consensus
  • Frontend display: Yellow warning card in Stage 2 showing dissent details

Changes

Backend (backend/council.py):

  • New detect_minority_opinions() function
  • Uses tournament ranking as the consensus baseline
  • Returns: model, consensus_position, dissent_positions, dissent_rate, dissenters, direction
  • Configurable threshold (default 30%) and position tolerance (default 1)
  • Integrated into run_full_council() metadata

Frontend:

  • Stage2.jsx: Added minorityOpinions prop and display component
  • Stage2.css: Added warning-styled card with overvalued/undervalued badges
  • ChatInterface.jsx: Pass minority_opinions to Stage2

Validation

8 test cases in tests/test_minority_opinions.py:

Test Description
test_no_minority_when_consensus No false positives when all rankers agree
test_minority_detected_with_dissent Correctly detects 33% dissent (1/3 rankers)
test_minority_direction_undervalued Correctly identifies "undervalued" direction
test_below_threshold_not_flagged 25% dissent (below 30%) not flagged
test_within_tolerance_not_flagged ±1 position difference within tolerance
test_empty_inputs Handles edge cases gracefully
test_5_model_realistic_scenario 40% dissent correctly detected in 5-model council
test_custom_threshold Custom thresholds work correctly 
$ python3 tests/test_minority_opinions.py
✓ No minority when consensus - PASSED
✓ Minority detected with dissent - PASSED
✓ Minority direction undervalued - PASSED
✓ Below threshold not flagged - PASSED
✓ Within tolerance not flagged - PASSED
✓ Empty inputs - PASSED
✓ 5-model realistic scenario - PASSED
✓ Custom threshold - PASSED

==================================================
All minority opinion tests passed!
==================================================

Dependencies

This PR builds on the tournament ranking feature (PR #128) which provides the consensus baseline.

Test plan

  • Run python3 tests/test_minority_opinions.py - all 8 tests pass
  • Start the app and submit a query to verify minority opinions display correctly
  • Verify no minority opinions shown when council has strong consensus

🤖 Generated with Claude Code

bledden and others added 3 commits January 5, 2026 18:26
@bledden @claude
Add tournament-style pairwise ranking aggregation
b1bbb9a 
Adds calculate_tournament_rankings() as an alternative to simple mean ranking.

Algorithm:
- Convert ordinal rankings to pairwise matchups
- For each pair of models, majority vote determines winner
- Ties awarded 0.5 points to each
- Final score = wins / total_matchups

Benefits over mean ranking:
- More robust to outlier rankings
- Theoretically principled (Condorcet-style)
- Handles cyclic preferences gracefully

Both ranking methods now included in metadata:
- aggregate_rankings: mean position (existing)
- tournament_rankings: pairwise win percentage (new)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@bledden @claude
docs: add tournament ranking validation documentation to README
9d49e25 
Documents the tournament-style pairwise comparison algorithm with:
- Explanation of why it's more robust than mean averaging
- Concrete example showing self-promotion bias scenario
- Tables comparing mean vs tournament results
- Outlier robustness validation (mean degrades 1.0→1.5, tournament stays 100%)
- Summary of validation test coverage

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@bledden @claude
feat: add minority opinion detection for ranking disagreements
5a7e404 
Detects when ≥30% of rankers significantly disagree with the consensus
ranking for a model (placing it more than 1 position away from consensus).

Backend changes:
- Add detect_minority_opinions() function to council.py
- Uses tournament ranking as consensus baseline
- Reports dissent rate, positions, dissenters, and direction (overvalued/undervalued)
- Configurable threshold (default 30%) and position tolerance (default 1)
- Include minority_opinions in run_full_council metadata

Frontend changes:
- Add minorityOpinions prop to Stage2 component
- Display minority opinions in a warning-styled card
- Show direction badges (overvalued in red, undervalued in green)
- List consensus position, dissent positions, and dissenter models

Validation tests:
- 8 test cases covering consensus, dissent detection, direction,
  threshold filtering, tolerance, edge cases, and realistic scenarios

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@bledden bledden mentioned this pull request Jan 6, 2026
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bledden