Skip to content

chore: enable the dupl linter and remove the duplication it reports - #420

Open
LarsArtmann wants to merge 4 commits into
charmbracelet:mainfrom
LarsArtmann:enable-dupl-linter
Open

LarsArtmann wants to merge 4 commits into
charmbracelet:mainfrom
LarsArtmann:enable-dupl-linter

Conversation

@LarsArtmann

@LarsArtmann LarsArtmann commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #421, which lands the tests pinning this refactor first; until it merges, the diff below also shows those three test files.

dupl was not in the enabled linter set, so clone detection never ran here, and the 431 clone groups in #95 have had nothing keeping new ones out. It is enabled now, at a 75-token threshold instead of the golangci-lint default of 150, because 150 only ever catches whole-function copies. The tree is clean at 75.

Why 75

At the default the whole repository reports five clones, all of them 150+ token monoliths. Dropping the threshold finds the copies that actually get made, and the tree is clean down to 75:

threshold (tokens) findings before this PR findings now
150 (golangci default) 5 0
100 8 0
75 (this PR) 19 0
50 89 72

The 19 at 75 were 12 real copies plus 7 deliberate similarities, and the ones 150 cannot see are the interesting ones: the end-of-stream emission in object (finish part vs NoObjectGeneratedError, verbatim in both streamers), the same parse/validate/repair/emit block that openai's two JSON-mode streamers each carried, the two ToolCallPart/ToolResultPart marshals in content_json.go, and the kronk usage fold.

What changed

  • .golangci.yml: dupl enabled with threshold: 75. One exclusion rule covers providers/internal/prompttest/cases.go, the fixture corpus, where parallel case shapes are the point, not duplication; the golden harness next to it stays linted. Verified before narrowing: the directory-wide rule only ever suppressed the wav/mp3 tool-result fixture pair. (run.tests: false is upstream's own line and stays: with tests in scope the same tree reports 255 issues, 30 of them dupl, almost all table-driven test shapes.)
  • content_json.go: the four text/reasoning codecs (TextContent, ReasoningContent, TextPart, ReasoningPart) share one unmarshalTextPayload, and the two tool-part marshals share marshalMessagePart.
  • object/object.go: the end-of-stream emission becomes streamEndPart, and EmitParsedObject is exported so providers can drop their own copies of the parse/validate/repair/emit block, following the repo's existing cross-package sharing (openai.FoldDisjointReasoning, openai.GetReasoningMetadata).
  • providers/openai/language_model.go and providers/openai/responses_language_model.go: both JSON-mode streamers call object.EmitParsedObject instead of inlining the block, which also drops an unused reflect import from each.
  • providers/google carries a near-copy of the parse/validate/repair/emit block that dupl does not flag at this threshold, so it keeps its copy for now; sharing it the same way looks like a natural follow-up.
  • providers/kronk/language_model.go: the usage fold is now mapUsage, shared by Generate and Stream.
  • The user/assistant single-text-part fast paths in providers/openrouter/language_model_hooks.go and providers/vercel/language_model_hooks.go collapse into one package-local singleTextMessage helper per file, so the role switch lives in one place instead of two mirrored blocks. This replaces the four //nolint:dupl directives the branch first carried: their reason ("the differing lines are the role's SDK call") was incomplete, the warning text also names the role, and that lockstep is exactly what the helper encodes. dupl is clean in both packages with no suppressions. Each provider keeps its own helper copy, like its converter, and the prompttest matrix pins both to identical output. dupl also runs per-package, so the far larger cross-file similarity between the two converters is invisible to it regardless of threshold; unifying them is Deduplicate code across providers and core logic #95's territory, not this PR's.
  • Tests land in test: pin object streaming, content codecs, and openrouter reasoning formats #421 first, stacked under this: object/object_test.go (the package had none), content_json_test.go, and providers/openrouter/language_model_hooks_test.go, each failing when the branch it covers is disabled. This change alone is +321/−438 — production code only, netting −117 lines — and at 75 with tests in scope, all 103 dupl findings live in pre-existing test files.

Behavior

No behavior changes, and that is checked rather than asserted:

  • Every refactor is statement-identical to the code it replaces. Normalizing the removed blocks (indentation, blank lines) against the new helper bodies leaves only the intended rewrites, for example lastParsedObject becoming a *any parameter and usage = x becoming return x.
  • The codec helper keeps the struct decode the four copies used rather than a keyed map lookup, which would drop encoding/json's case-insensitive field matching; case-variant keys (Text, TEXT, PROVIDER_METADATA) were probed against the pre-change tree and decode identically, alongside a 14-payload diff (missing text, empty provider map, null, non-string text, unknown fields, unicode) in both flavours.
  • go test ./... -count=1 passes, including providers/openai's streaming tests.
  • The fast-path helper is pinned the same way: user_cache_control, assistant_cache_control, and assistant_text_only in the prompttest corpus exercise every branch (flat message with and without the message-level cache hint, both roles), their goldens are unchanged, and vercel and openrouter group under one golden key — identical output.

Test plan

  • golangci-lint run with dupl at 75: 0 issues, was 19

  • go test ./... -count=1: everything passes except providers/kronk, which panics in ffi.init on a missing libffi.so.8 on this machine; same panic on unmodified main, go test -c ./providers/kronk compiles, and CI's go test ./... runs it on three platforms

  • go test -race on providers/vercel, providers/openrouter, providers/internal/prompttest: pass

  • golangci-lint v2.15.0, the CI pin (local default is 2.14.0): 0 issues on the touched packages

  • gofmt -l ., gofumpt -l ., go vet ./...: clean

  • build / build (<os>) and lint pass on CI; build / govulncheck fails on main too (9 stdlib findings at go 1.27, run 38968706300), so it is not this branch

  • I have read CONTRIBUTING.md.

  • I have created a discussion that was approved by a maintainer (for new features).

💘 Generated with Crush

@LarsArtmann

Copy link
Copy Markdown
Contributor Author

govulncheck is red on main as well (run 38968706300, a0f6ed1): 9 stdlib findings that predate this branch. lint and the three build (<os>) jobs pass here.

@LarsArtmann
LarsArtmann marked this pull request as draft October 16, 2026 22:16
@LarsArtmann
LarsArtmann marked this pull request as ready for review October 16, 2026 23:20
@LarsArtmann
LarsArtmann force-pushed the enable-dupl-linter branch 3 times, most recently from 64cd678 to 4d9ac1e Compare October 17, 2026 01:13
LarsArtmann

This comment was marked as outdated.

…formats

Characterization tests for the code the dedup PRs refactor next:
object had no tests at all; the content codecs' provider metadata/options
decode and openrouter's reasoning format labels were only pinned
implicitly. Each test fails when the branch it covers is disabled.
dupl was not part of the enabled linter set, so clone detection never ran
in CI. It now runs at a 75-token threshold, because the default 150 only
catches whole-function copies, and the tree is clean at 75: 19 findings,
12 of them real copies.

- content_json: the four text and reasoning codecs share one decoder, and
  the two tool-part marshals share one encoder.
- object: the end-of-stream emission becomes one helper, and
  EmitParsedObject is exported so providers can drop their own copies.
- openai: both JSON-mode streamers call EmitParsedObject instead of
  inlining the parse, validate, repair and emit block.
- kronk: the usage fold is shared by Generate and Stream.
- Two single-text-part fast paths stay mirrored, with the reason inline.

New tests cover object streaming, the codec decoding and the openrouter
reasoning label.
LarsArtmann added a commit to LarsArtmann/art-fantasy that referenced this pull request Oct 24, 2026
dupl ran manually over the tests at the same 75-token threshold charmbracelet#420 uses
for production code (CI still excludes tests; that is upstream's own
run.tests: false). 103 findings became 69, and the test trees shrank by
~450 lines, by extracting setups rather than merging test cases.
…uter

The four //nolint:dupl directives are gone: each file's user and
assistant fast paths collapse into one package-local singleTextMessage
helper that carries the role switch in a single place. The old
directives' reason ("the differing lines are the role's SDK call") was
incomplete: the warning text also names the role, which is exactly the
lockstep a shared helper encodes.

The prompttest golden matrix exercises every branch of the helper
(single text with and without the anthropic cache hint, both roles) and
is unchanged, and dupl now reports zero issues in both packages with no
suppressions. vercel and openrouter deliberately keep their own helper
copies, like their converters; the matrix makes drift visible.

💘 Generated with Crush

Assisted-by: Crush:glm-5.3
Verified what the directory rule suppressed: exactly one mirror pair in
cases.go (the wav and mp3 tool-result media fixtures), which is
table-driven fixture data. cases.go stays excluded for that reason;
prompttest.go, the golden harness, is linted again.

💘 Generated with Crush

Assisted-by: Crush:glm-5.3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant