LLM Model Value Chooser

chooser is a local CLI for reporting coding model value from Arena Code or Artificial Analysis.

It uses Arena by default for:

• model identity
• coding score and rank
• price
• context window
• license label

It can also use Artificial Analysis for:

• model identity
• benchmark scores
• price
• output speed
• time to first token

Then it answers two practical questions:

• which models look like the best value under your constraints
• which models sit on the Pareto frontier and which near-frontier choices deserve a look

What It Does

chooser can:

• refresh Arena snapshots
• refresh Artificial Analysis snapshots
• rank models by value score
• filter by budget and context length
• search by model name or vendor
• show a Pareto decision table
• render a small ASCII Pareto plot

Install

You need:

• Python 3.11 or newer
• uv

From repo root:

uv sync

Run commands with:

uv run chooser --help

best and pareto auto-download snapshots when none exist. They also try an auto-refresh when stored snapshots are older than 24 hours. You can force a refresh with chooser refresh.

Quick Start

Arena shortlist:

uv run chooser best
# OR
uv run chooser best --source arena

Add filters:

# Only models with context window >= 300k tokens that cost less than $10 per 1M tokens
# All price filters use dollars per million tokens.

uv run chooser best --max-output-price 10 --min-context 300000

# Weighted budget filter
uv run chooser best --max-weighted-cost 1.00

# Search by model name or vendor (OR match, case-insensitive)
uv run chooser best --search qwen deepseek

Pareto decision table:

uv run chooser pareto

Frontier plot:

uv run chooser pareto --plot

Artificial Analysis shortlist:

Artificial Analysis needs an API key. Put it in .env:

cp .env.example .env
# edit the .env file to add your Artifical Analysis API key

Or pass it through the shell environment:

ARTIFICIAL_ANALYSIS_API_KEY=aa_... uv run chooser refresh --source aa

Display the shortlist:

uv run chooser best
# OR
uv run chooser best --source aa
# OR with filtering
uv run chooser best --source aa --metric coding

Note that Artificial Analysis has some additional filter options.

Artificial Analysis Pareto view:

uv run chooser pareto --source aa --metric intelligence --plot

Force refresh:

uv run chooser refresh

CLI Option Map

flowchart TD
    cli["chooser"] --> refresh
    cli --> best["best"]
    cli --> pareto["pareto"]

    subgraph pareto_group[" "]
        pareto --> plot["--plot"]
    end

    subgraph best_group[" "]
        best --> mode_filters["shared filters"]
        pareto --> mode_filters

        mode_filters --> source["--source"]
        source --> arena["arena"]
        source --> aa["aa or artificial-analysis"]

        aa --> aa_metric["--metric"]
        aa_metric --> coding["coding"]
        aa_metric --> intelligence["intelligence"]
        aa_metric --> livecodebench["livecodebench"]
        aa_metric --> math["math"]
        aa_metric --> scicode["scicode"]

        mode_filters --> budget["budget filters"]
        budget --> max_input["--max-input-price"]
        budget --> max_output["--max-output-price"]
        budget --> max_weighted["--max-weighted-cost"]

        mode_filters --> search["search filter"]
        search --> name_search["--search"]

        mode_filters --> capability["capability filter"]
        capability --> min_context["--min-context"]

        mode_filters --> shaping["output shaping"]
        shaping --> top_n["--top-n"]
    end

    subgraph refresh_group[" "]
        direction RL
        refresh["refresh"]
        refresh --> refresh_arena["--source arena"]
        refresh --> refresh_aa["--source aa or artificial-analysis"]
    end

    style best_group fill:transparent,stroke:transparent
    style refresh_group fill:transparent,stroke:transparent
    style pareto_group fill:transparent,stroke:transparent

Loading

Commands Documentation

chooser refresh

Fetches current source data and writes snapshots under:

• data/raw/
• data/normalized/

Current snapshot files:

• data/raw/arena-latest.html
• data/normalized/arena-latest.json
• data/raw/artificial-analysis-latest.json
• data/normalized/artificial-analysis-latest.json

refresh is optional. best and pareto auto-download snapshots if none exist and try to refresh stale snapshots older than 24 hours. Use refresh to force an update.

Examples:

uv run chooser refresh --source arena
uv run chooser refresh --source aa

chooser best

Ensures snapshots exist, tries an auto-refresh when stored data is older than 24 hours, then loads snapshot rows, applies filters, scores surviving models, and prints a formatted shortlist table plus a summary line.

Current columns:

• model
• rank±
• arena_score
• price_per_1m
• ctx
• license
• value_score

Rendered order:

model | rank± | arena_score | price_per_1m | ctx | license | value_score

rank± is Arena rank spread for Arena. Artificial Analysis derives rank by sorting the selected metric.

price_per_1m uses input / output dollars per million tokens, for example 75 / 150.

ctx is compact context window, for example 128k or 1M.

license is compact license class: OSS for open-source rows and Prop for proprietary rows.

value_score is compacted for table output, for example 28.8M or 1.21B.

Rows missing price are excluded because they cannot be scored. Rows missing context stay visible unless --min-context is set.

chooser pareto

Prints a summary line, then a strict frontier table and near-frontier recommendations from the filtered shortlist.

Near-frontier rows are dominated rows within 5.0 score of the cheaper frontier. They are sorted by score, then weighted cost, then model name.

chooser pareto --plot

Prints an ASCII scatter plot with frontier labels.

Filters

best and pareto both support:

• --source {arena,aa,artificial-analysis}
• --metric {coding,intelligence,livecodebench,math,scicode}
• --max-input-price
• --max-output-price
• --max-weighted-cost
• --min-context
• --search TERM [TERM ...]

--max-input-price, --max-output-price, and --max-weighted-cost all use dollars per million tokens.

--search accepts one or more case-insensitive terms. A model passes the filter if its name or vendor contains any of the supplied terms. Example:

uv run chooser best --search qwen deepseek
uv run chooser pareto --search deepseek

Output Shaping

best and table-style pareto also support:

• --top-n

--top-n trims visible rows after command-specific display ordering. It does not change filtering rules.

How Scoring Works

Current weighted cost formula:

weighted_cost = input_price + (2 * output_price)

The CLI converts your price flags from dollars per million tokens into the stored unit before filtering.

Current value score:

value_score = score / weighted_cost

Arena uses Arena Code score. Artificial Analysis uses the selected metric:

• coding: Artificial Analysis Coding Index
• intelligence: Artificial Analysis Intelligence Index
• math: Artificial Analysis Math Index
• livecodebench
• scicode

Data Source

• Arena Code leaderboard
• Artificial Analysis API

Artificial Analysis free API requires attribution to https://artificialanalysis.ai/. It has a 1,000 requests/day limit, so chooser caches raw and normalized snapshots under data/.

Development

Run full test suite:

uv run pytest -q

Run one test file:

uv run pytest -q tests/unit/test_filters.py

Test multiple command-line invocations:

scripts/cli_matrix.sh