Skip to content

cost ledger: model ids with a quote or backslash write malformed JSON and are dropped from the cost total #165

Description

@RealDiligent

Problem

_ledger_append in src/trinity/llm/openai_compatible_pool.py builds each cost-ledger line by hand with an f-string:

f.write(
    f'{{"provider":"{provider}","m":"{model}","p":{int(prompt_tokens)},'
    f'"c":{int(completion_tokens)}}}\n'
)

model is route.model_id, taken verbatim from configs/models*.yaml. If a model id contains a double quote ("), a backslash (\), or a control character, the emitted line is not valid JSON.

The reader, ledger_cost_report in src/trinity/costing.py, does:

try:
    row = json.loads(line)
except json.JSONDecodeError:
    continue

So every malformed line is silently skipped — its prompt/completion tokens never contribute to cost_usd, cost_calls, or the per-model breakdown.

Impact

Silent under-reporting of API spend. A single pool model whose id contains a JSON metacharacter makes all of its calls invisible in the cost report, with no error and no warning — exactly the kind of drift the cost ledger exists to prevent. It also corrupts the JSONL file for any other consumer.

Reproduction

import os
os.environ["TRINITY_COST_LEDGER"] = "/tmp/ledger.jsonl"
from trinity.llm.openai_compatible_pool import _ledger_append
from trinity.costing import ledger_cost_report

_ledger_append("openrouter", 'weird"model', 1000, 2000)
print(ledger_cost_report("/tmp/ledger.jsonl")["cost_calls"])  # -> 0 (call dropped)

Suggested fix

Serialize the record with json.dumps instead of a hand-built f-string, so the writer and reader agree on escaping.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions