feat(citations): add citation merging to tests and tooling (closes #934)

[arcgod-design]

What

Adds a citation-merging utility to the citation-utils module and tests/tooling — closing issue #934's "Add citation merging to tests and tooling".

When two retrieval batches return overlapping citations (e.g. a streaming-resume retry that re-asks alongside a fresh batch), the assistant currently surfaces the same source twice. merge_citations() dedupes by canonical (source, chunk) key, aggregates per-citation occurrence counts and scores, and emits a provenance audit so a reviewer can trace which input lists contributed which citations.

Why

With streaming resume (routes/chat.py already retries on transient errors), it's possible for a second retrieval pass to return the exact same chunk that the first pass returned. Currently both appear in sources[] — a UX wart that also skews retrieval count metrics. Issue #933 added per-source ranking; this PR adds the symmetric merge operation: combine multiple citation lists into one canonical list while preserving fidelity (occurrence count, aggregated score, longest preview).

What changed

Backend service — backend/services/citation_utils.py

The existing build_sources() function (43 lines, issue #318-era) is unchanged. New helpers appended below it:

• default_score_merger(scores) — arithmetic mean; None entries skipped (no division-by-zero); non-numeric skipped. Mirrors Add source ranking to tests and tooling #933's relevance-collapse-to-neutral semantics.
• default_preview_merger(previews) — longest-wins (ties broken by earliest-seen order).
• _normalise_source_entry(entry) — promote legacy strings to {"source": str, "chunk": 0}; coerce chunk to int ("3"→3, "garbage"→0) so a malformed upstream entry doesn't break _merge_key.
• _merge_key(source) — returns (str(source), int(chunk)) tuple.
• MergeResult dataclass — merged (canonical list), duplicates (audit list of dropped entries), contributed_by (mapping dedupe-key → list of input-list indices). .to_dict() for JSON serialisation.
• merge_citations(*citation_lists, score_merger, preview_merger, scores) — variadic merge:
  • Inputs may be lists of citation dicts (canonical) or lists of strings (legacy) — mixed freely.
  • For each unique (source, chunk):
    • source & chunk taken from first-seen entry (deterministic).
    • preview aggregated via configurable preview_merger (longest-wins default).
    • score aggregated via configurable score_merger (arithmetic-mean default).
    • occurrence_count and contributors[] list updated.
    • Ad-hoc fields (e.g. metadata.created_at) preserved from first-seen entry only — no silently mixing two dicts' nested fields.
  • scores parameter is a parallel list-of-(list|None); length auto-padded with None to match citation_lists length.
• normalize_citations(citations) — convenience wrapper around _normalise_source_entry for callers that want a list of canonical dicts without invoking the merge.

Tooling — scripts/merge_citations.py

Offline CLI that:

• Reads N JSON files, each a JSON array of dict or string citations.
• Optionally reads parallel score vectors via --scores file1 [file2|none] file3.
• Emits either a JSON document (MergeResult.to_dict()) or a --plain report.
• Exit codes: 0 success, 1 bad JSON shape, 2 file-not-found. Mirrors the conventions established by scripts/rank_sources.py (Add source ranking to tests and tooling #933) and scripts/dedupe_docs.py (Add dedupe logic to build and deploy docs #928).

Example:

$ python scripts/merge_citations.py a.json b.json --plain
Merged 2 unique citations from 2 list(s) (1 duplicates dropped):
    1. [seen=2] a.md#0  (preview: "alpha chunk 0 text longer"...)
    2. [seen=1] b.md#0  (preview: "beta short"...)

Tests — backend/tests/test_citation_merging.py

45 pytest cases, all passing:

Class	Cases	Coverage
TestDefaultScoreMerger	7	empty, single, mean, None-skip, non-numeric skip, partial-None
TestDefaultPreviewMerger	6	empty, single, longest-wins, tie, None-skip
TestMergeCitationsBasic	6	no-inputs, single, disjoint, overlap-dedupe, different-chunks, ad-hoc fields preserved
TestMergeOrder	2	first-seen ordering, duplicates don't promote
TestLegacyStringCompatibility	3	string→dict promotion, mixed, duplicate strings
TestScoreAggregation	5	set from scores, aggregated on dup, short-scores padded, explicit None, no-score path
TestCustomMergers	2	max() score merger, first-seen preview merger
TestEdgeCases	6	empty list arg, non-list skip, malformed normalised, non-numeric chunk, three-way partial overlap, empty-preview path
TestNormalizeCitations	5	empty, shallow-copy isolation, string-promoted, non-list returns empty, mixed entries
TestMergeResultSerialisation	2	to_dict shape, empty path
TestBuildSourcesInterplay	1	end-to-end: build_sources output → merge_citations produces deduped + aggregated result

Existing test_citations.py (21 cases) still passes unmodified — backward-compatible.

Verification

$ cd backend && pytest tests/test_citation_merging.py tests/test_citations.py -v
============================= 66 passed in 1.91s ==============================
      (45 new + 21 existing)

$ ruff check backend/services/citation_utils.py \
           backend/tests/test_citation_merging.py scripts/merge_citations.py
All checks passed!

$ ruff format --check <same files>
3 files already formatted

Acceptance criteria

• Implement the change in the tests and tooling.
• Add or update tests, docs, or validation for the touched path.

Closes #934.

[vercel]

@arcgod-design is attempting to deploy a commit to the Darshan's projects Team on Vercel.

A member of the Team first needs to authorize it.

[imDarshanGK]

@arcgod-design