Skip to content

Conversation

@piochelepiotr
Copy link
Contributor

@piochelepiotr piochelepiotr commented Dec 1, 2025

What does this PR do?

Fixes support for Confluent's Protobuf wire format message indices in the kafka_consumer integration. The integration now correctly extracts and uses message indices to deserialize Protobuf messages, including support for multiple message types and nested types.

The Confluent Protobuf wire format is structured as:

[magic_byte][schema_id:4bytes][message_indices_length:varint][message_indices:varint...][protobuf_payload]

Motivation

Confluent's Protobuf wire format includes message indices after the schema ID to specify which message type from the .proto file should be used. Without this fix, deserialization fails when schemas contain multiple message types or when using non-zero indices.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

@codecov
Copy link

codecov bot commented Dec 1, 2025

Codecov Report

❌ Patch coverage is 91.56627% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.01%. Comparing base (a5342b2) to head (b4e89d0).
⚠️ Report is 20 commits behind head on master.

Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@piochelepiotr piochelepiotr force-pushed the piotr.wolski/fix-protobuf-with-schema-registry branch from 8768317 to 7b68c8d Compare December 1, 2025 19:33
@piochelepiotr piochelepiotr marked this pull request as ready for review December 1, 2025 19:40
@piochelepiotr piochelepiotr requested a review from a team as a code owner December 1, 2025 19:40
dkirov-dd
dkirov-dd previously approved these changes Dec 5, 2025
Kyle-Neale
Kyle-Neale previously approved these changes Dec 5, 2025
@temporal-github-worker-1 temporal-github-worker-1 bot dismissed stale reviews from dkirov-dd and Kyle-Neale December 5, 2025 15:49

Review from dkirov-dd is dismissed. Related teams and files:

  • agent-integrations
    • kafka_consumer/changelog.d/22020.fixed

Review from Kyle-Neale is dismissed. Related teams and files:

  • agent-integrations
    • kafka_consumer/changelog.d/22020.fixed
@dkirov-dd dkirov-dd added this pull request to the merge queue Dec 5, 2025
Merged via the queue into master with commit 44a9723 Dec 5, 2025
51 of 55 checks passed
@dkirov-dd dkirov-dd deleted the piotr.wolski/fix-protobuf-with-schema-registry branch December 5, 2025 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants