feat: add Confluent v2-compatible consumer REST API#73
Merged
haochenpan merged 5 commits intomainfrom Feb 28, 2026
Merged
Conversation
- Remove action_provider/, common/, lambda/, examples/, docs/ap/ - Rename web_service_v3/ to web_service/ as the canonical service - Merge common/ utilities into web_service/utils.py - Move test files from web_service_v3/ to tests/ with *_test.py naming - Remove old v2 web_service code and action_provider tests - Update Dockerfile, CI workflows, docs, and pyproject.toml accordingly Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move DynamoDB deletion after Kafka deletion check in delete_topic() so metadata isn't removed when Kafka deletion fails. Add direct Kafka cleanup fallback in race condition test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix the "not in audience" auth error that incorrectly passed a tuple instead of a string as the error reason. Also remove the unused validate_name method and its NAME_MIN/NAME_MAX constants from AuthManager, since NamespaceService.validate_name is the active one. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2 tasks
haochenpan
force-pushed
the
web-refactor
branch
from
00a2cbc to
2b860f9
Compare
Implement a full Kafka consumer REST API matching the Confluent Kafka
REST Proxy v2 spec, allowing users to consume events over HTTP using
only their Globus access token — no Kafka client or AWS credentials.
Endpoints (under /api/v3/{namespace}/consumers/...):
1. POST /{group} — Create consumer instance
2. DELETE /{group}/instances/{id} — Delete consumer instance
3. POST /{group}/instances/{id}/subscription — Subscribe to topics
4. GET /{group}/instances/{id}/subscription — Get subscription
5. DELETE /{group}/instances/{id}/subscription — Unsubscribe
6. GET /{group}/instances/{id}/records — Fetch records
7. POST /{group}/instances/{id}/offsets — Commit offsets
8. GET /{group}/instances/{id}/offsets — Get committed offsets
9. POST /{group}/instances/{id}/assignments — Assign partitions
10. GET /{group}/instances/{id}/assignments — Get assignments
11. POST /{group}/instances/{id}/positions — Seek to offset
12. POST /{group}/instances/{id}/positions/beginning — Seek to beginning
13. POST /{group}/instances/{id}/positions/end — Seek to end
New files:
- consumer_models.py: Pydantic models with Confluent dot-notation fields
- consumer_service.py: Consumer lifecycle with thread-safe registry,
per-consumer locks, idle timeout reaping, and MSK IAM auth
Key design decisions:
- Namespace-scoped isolation (group ID = {ns}.{group}, topic = {ns}.{topic})
- Confluent-compatible request/response shapes and HTTP status codes
- Header-only auth extraction (replaces extract_val body/header hybrid)
- Blocking poll() runs in threadpool via sync def to avoid event loop stall
- Daemon reaper thread cleans up consumers idle >5min (configurable)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
haochenpan
force-pushed
the
web-refactor
branch
from
2b860f9 to
e472434
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
web_service_v3,action_provider,common,lambda), renameweb_service_v3toweb_service, consolidate testsConsumer REST API endpoints
All under
/api/v3/{namespace}/consumers/...:/{group}/{group}/instances/{id}/{group}/instances/{id}/subscription/{group}/instances/{id}/subscription/{group}/instances/{id}/subscription/{group}/instances/{id}/records/{group}/instances/{id}/offsets/{group}/instances/{id}/offsets/{group}/instances/{id}/assignments/{group}/instances/{id}/assignments/{group}/instances/{id}/positions/{group}/instances/{id}/positions/beginning/{group}/instances/{id}/positions/endKey design decisions
{namespace}.{group_name}, topic ={namespace}.{topic}CONSUMER_INSTANCE_TIMEOUT_MS)KafkaConsumer(already a dependency)New files
web_service/consumer_models.py— Pydantic request models with Confluent dot-notation field aliasesweb_service/consumer_service.py— Consumer lifecycle management (registry, MSK IAM auth, reaper thread)Test plan
🤖 Generated with Claude Code