Enable share groups by default and enhance protocol gates

.github/copilot-instructions.md

@@ -7,10 +7,12 @@ Pure-Rust async Kafka client. Tokio runtime, edition 2024, MSRV 1.88.
 ## Build & Test
 
 ```sh
-cargo fmt && cargo clippy --all-targets && cargo test --lib
+just pre-commit    # fmt, clippy, check — after every change
+just ci            # everything CI runs, except the Docker suites
 ```
 
-Run this after every change. All three must be clean before considering work done.
+The justfile is the single source of truth for what the checks are; CI calls the
+same recipes. Do not hand-roll the equivalent cargo commands.
 
 ## Architecture
 
@@ -45,7 +47,7 @@ This is not optional — apply it before declaring a task complete.
 ## PR Review Readiness
 
 Before submitting, verify:
-1. `cargo fmt && cargo clippy --all-targets` — zero warnings
-2. `cargo test --lib` — all pass
+1. `just ci` — clean, including the reachability and parity gates
+2. Touching the send path, accumulator or codec → also `just bench-check`
 3. Changed public API → update `site/content/docs/` in the same commit
 4. New metric → present in struct, Prometheus export, snapshot, reset, and `site/content/docs/metrics.md`

.github/workflows/ci.yml

@@ -169,6 +169,57 @@ jobs:
       - uses: extractions/setup-just@v3
       - run: just doc
 
+  # The gate behind every ```rust,compile block in README.md and
+  # site/content/docs/. It was in `just ci` — and therefore in the release
+  # recipe — for its whole life without ever running in a workflow, so a pull
+  # request could break every documented sample and stay green. That is the
+  # same shape as the SASL suite, which was runnable locally long before
+  # anyone noticed it had never run in CI.
+  #
+  # `doc_api.py` checks that names resolve; it cannot check that a call has
+  # the right shape. This compiles the snippets instead.
+  # Asserts that every recipe in `just ci` has a job here, that the
+  # `ci-success` aggregator needs every job, and that it carries
+  # `if: always()`. Closes the class that `docs-test` and `integration-sasl`
+  # were both instances of.
+  ci-job-parity:
+    name: CI job parity
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: extractions/setup-just@v3
+      - run: just ci-job-parity
+
+  docs-test:
+    name: Documentation snippets
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - uses: extractions/setup-just@v3
+      - run: just docs-test
+
+  # Classifies API changes against the last published release. Pre-1.0 this
+  # reports rather than blocks (D1 allows a minor bump to break), but the
+  # output is what the CHANGELOG's Breaking section owes the reader — and
+  # until now nothing checked that list was complete.
+  semver:
+    name: SemVer classification
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - uses: extractions/setup-just@v3
+      # A prebuilt binary rather than `cargo install`, which takes minutes.
+      # The check itself runs through the justfile so CI and a developer's
+      # machine cannot drift (D22).
+      - uses: taiki-e/install-action@v2
+        with:
+          tool: cargo-semver-checks
+      - run: just semver-check
+
   supply-chain:
     name: Supply chain (cargo-deny)
     runs-on: ubuntu-latest
@@ -281,3 +332,63 @@ jobs:
       # Here the job's toolchain already *is* 1.88, so the plain command is
       # both simpler and a genuine check that the pinned toolchain builds.
       - run: cargo check
+
+  # ── The single required status check ────────────────────────────────────────
+  #
+  # Branch protection names THIS job and nothing else. Without it, every job
+  # above has to be listed by hand in the repository settings, and a newly
+  # added job is not required by default — it runs, it can fail, and the merge
+  # button stays green until somebody remembers to go and add it.
+  #
+  # Two GitHub behaviours make the obvious implementation wrong, and both are
+  # guarded against here:
+  #
+  #   1. A required check whose job never runs stays *pending* forever, so the
+  #      pull request can never merge. `if: always()` makes this job run even
+  #      when a dependency failed or was skipped.
+  #   2. GitHub reads a *skipped* required check as **success**. So this job
+  #      must not simply skip on failure — it has to run and then fail
+  #      explicitly, which is what the `contains(needs.*.result, ...)` test
+  #      below does. Checking `success()` would not be enough either, because
+  #      a skipped dependency does not make `success()` false.
+  #
+  # `needs` is exhaustive on purpose. `just ci-job-parity` asserts that this
+  # list matches the jobs actually defined in this file, so adding a job
+  # without wiring it here fails the build rather than silently going
+  # unrequired.
+  ci-success:
+    name: CI
+    if: always()
+    runs-on: ubuntu-latest
+    needs:
+      - fmt
+      - clippy
+      - check
+      - test
+      - test-ring
+      - minimal-features
+      - cross-platform
+      - protocol-parity
+      - secret-debug
+      - version-check
+      - test-reachability
+      - config-reachability
+      - protocol-reachability
+      - ci-job-parity
+      - doc
+      - docs-test
+      - semver
+      - supply-chain
+      - integration
+      - integration-sasl
+      - integration-redpanda
+      - msrv
+    steps:
+      - name: Verify every job succeeded
+        run: |
+          echo "Results: ${{ join(needs.*.result, ', ') }}"
+          if ${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') || contains(needs.*.result, 'skipped') }}; then
+            echo "::error::At least one CI job did not succeed."
+            exit 1
+          fi
+          echo "All CI jobs succeeded."

.github/workflows/publish.yml

@@ -19,8 +19,36 @@ env:
   CARGO_TERM_COLOR: always
 
 jobs:
+  # A crates.io version is permanent: a yanked number is burned and can never
+  # be reused. So the tag is gated on the *same* evidence a pull request is,
+  # not on a lighter subset.
+  #
+  # This used to be two `cargo test` invocations and nothing else — no protocol
+  # parity, no reachability checks, no secret-Debug scan, no cargo-deny, no
+  # integration suite. The mitigating assumption was that a tag is cut from a
+  # green `main`, which was never actually enforced anywhere.
+  #
+  # `just ci` is the full local gate; the Docker suites stay out of it because
+  # they need a broker, and they have already run on this commit via ci.yml.
+  verify:
+    name: Verify release gate
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          components: rustfmt, clippy
+      - uses: Swatinem/rust-cache@v2
+      - uses: extractions/setup-just@v3
+      - run: just ci
+      - name: Supply chain
+        run: |
+          cargo install --locked cargo-deny || true
+          cargo deny check advisories bans licenses sources
+
   publish:
     name: Publish
+    needs: verify
     runs-on: ubuntu-latest
     steps:
       - name: Checkout

.gitignore

@@ -21,13 +21,19 @@ target
 #.idea/
 
 # Custom additions
-_prompts/
-API_VERSIONS.md
-BACKLOG*.md
-CONCEPT.md
-FINDINGS.md
 .DS_Store
 
+# Internal architecture notes. Not published: the `D` and `R` numbers they use
+# are resolvable only inside that folder, so nothing in the published tree may
+# cite one.
+concepts/
+
+# Third-party specifications — Apache Kafka message schemas and protocol guide,
+# KIP wiki pages, IETF RFCs, AWS and OpenTelemetry documents. Copyrighted
+# publications that are not ours to redistribute; `concepts/REFERENCES.md`
+# indexes every one with its retrieval URL, so the folder can be rebuilt.
+specs/
+
 # Zola documentation site (site/) build output.
 #
 # `giallo-*.css` are the syntax-highlighting themes Zola generates from
@@ -53,6 +59,3 @@ fuzz/coverage/
 
 # Python bytecode from the xtask check scripts.
 __pycache__/
-
-# Generated by `just docs-test` (xtask/docs_test.py).
-examples/__docs_test.rs

CHANGELOG.md

@@ -11,6 +11,119 @@ Entries before 0.17.0 were reconstructed from the release history and the
 `Upgrading` sections that previously lived in `README.md`. They are summaries,
 not a complete record.
 
+## [0.23.0] — 2026-09-13
+
+Share groups stopped being experimental in Apache Kafka 4.2, and krafka kept
+hiding them behind a feature flag called `unstable-protocol` for two more
+releases. The flag was accurate when written and nothing caught it when
+upstream moved, because no check here can test a sentence about Apache Kafka.
+
+This release fixes that, makes the class unrepeatable, and wires up three gates
+that existed without ever running.
+
+### Breaking
+
+- **The KIP-932 share consumer moved from `unstable-protocol` to its own
+  `share-groups` feature, on by default.** A build that enabled
+  `unstable-protocol` only to reach `ShareConsumer` no longer needs it; a build
+  that does not want the module turns off a default feature. The APIs are
+  unchanged.
+
+  The gate contradicted its own data: every share API is marked stable in
+  krafka's vendored Kafka 4.3 snapshot, the version table negotiates all of
+  them unconditionally on every build, and `lib.rs` described the feature as
+  covering versions Kafka marks `latestVersionUnstable` — which no share API
+  is. Only the API that could reach the protocol was hidden.
+
+  `share-groups` carries the same semver promise as the rest of the crate and
+  needs a Kafka 4.2+ broker.
+
+- **`unstable-protocol` now means exactly one thing**: versions Kafka's message
+  schema marks `latestVersionUnstable` — `ApiVersions` v5 (KIP-1242) and
+  `InitProducerId` v6 (KIP-939). `just protocol-parity` enforces the rule, so a
+  stable version can no longer be gated behind it.
+
+### Added
+
+- **A performance regression gate.** `benches/send_path.rs` measures the
+  producer send path end to end against the in-process fake broker, and
+  `just bench-check` fails when a mean regression exceeds 10% *and* the 95%
+  confidence interval excludes zero.
+
+  There was previously no throughput or latency measurement anywhere: a change
+  that halved producer throughput passed every gate. The fake broker cannot
+  support a *published* figure and none is produced — but a regression gate
+  compares krafka against krafka, and a constant harness overhead cancels
+  between runs. Not part of `just ci`: slow and noisy on a shared runner.
+
+- **`just ci-job-parity`** — asserts every recipe in `just ci` has a CI job,
+  that the new `ci-success` aggregator needs every job, and that it carries
+  `if: always()`. GitHub reads a *skipped* required check as success, so an
+  aggregator without it inverts the rule it enforces.
+
+- **A `ci-success` aggregating status check.** `ci.yml` had 19 jobs and no
+  `needs:`, so branch protection named each by hand and a newly added job was
+  not required by default. Branch protection should now require `CI` alone.
+
+- **A `docs-test` CI job.** The gate behind every compiled documentation
+  snippet was in `just ci` — and in the release recipe — while no pull request
+  ever ran it.
+
+- **`just semver-check` and a `semver` CI job**, running `cargo-semver-checks`
+  against the last published release. Pre-1.0 it reports rather than blocks,
+  but the output is what the `Breaking` section owes the reader.
+
+- **A `verify` job gating `publish.yml` on `just ci` and `cargo deny`.** The
+  release workflow previously ran two `cargo test` invocations and published.
+
+### Fixed
+
+- **`subscribe()` failed instead of retrying when the group coordinator was
+  moving.** `NOT_COORDINATOR`, `COORDINATOR_NOT_AVAILABLE` and
+  `COORDINATOR_LOAD_IN_PROGRESS` all mean "re-run FindCoordinator and try
+  again" — a freshly started or rebalancing cluster answers this way routinely,
+  and the Java client retries transparently.
+
+  krafka dropped the cached coordinator and then returned the error anyway. The
+  helper that did the invalidation returned a `bool` documented as "retriable
+  after re-discovery", and both call sites discarded it, so nothing ever made
+  the next attempt. Applications saw
+  `Failed to subscribe: Broker { code: NotCoordinator }`.
+
+  Both join paths now re-discover the coordinator and retry with jittered
+  backoff, bounded at five attempts. The KIP-848 path had the same defect in a
+  worse form — it returned the error without invalidating the cached
+  coordinator at all, so nothing downstream could recover either.
+
+  Found by the Redpanda integration suite, which is where a real coordinator
+  election actually happens. Both fixes carry a fake-broker regression test
+  verified against the defect.
+
+- **The README claimed KIP-1258 OAuth client assertion was not implemented.**
+  It has been implemented and public since the `oauth-oidc` feature landed, as
+  `ClientCredentials::assertion`. The same sentence now names
+  `StreamsGroupHeartbeat` (key 88) as the only KIP-1071 gap, and notes that
+  `StreamsGroupDescribe` (key 89) is implemented.
+
+- **The share-consumer guide said share groups were "stable as of Apache Kafka
+  4.0".** They reached general availability in 4.2. The protocol reference also
+  still flagged the four share APIs as requiring `unstable-protocol`.
+
+- **Documentation corrections.** The README's 536-line upgrade history is gone —
+  every version it covered is in this file. The performance guide now describes
+  the send-path regression gate and why the same fake-broker harness is wrong
+  for ranking codecs within a run and right for detecting a regression across
+  runs. `fuzz/README.md` documented three of six fuzz targets.
+
+- **The request-priority documentation overstated what it delivers.** Priority
+  channels order requests; they cannot reorder response bytes already on the
+  wire. One socket per broker carries one byte stream, so a large fetch
+  response delays every response behind it, heartbeat included. Usually moot,
+  since the coordinator is normally a different broker from the partition
+  leaders being fetched — but `ConnectionPool` keys on address, so on a
+  single-broker cluster they share a connection. `max_response_size` bounds the
+  stall and is now named as the lever.
+
 ## [0.22.0] — 2026-09-01
 
 A long-lived producer could permanently lose a topic it was actively writing