Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka integration #983

Open
wants to merge 26 commits into
base: main
Choose a base branch
from
Open

Conversation

TomBebb
Copy link

@TomBebb TomBebb commented Nov 8, 2024

/claim #936

Fixes #936

Description

Integrates with Kakfa, using its crate.
This is toggled on/off via the KAFKA_TOPIC environment variable.
This uses a background polling iterator to loop through messages, and ingesting messages to the stream with the name of the KAFKA_TOPIC.
Covers all possible config options via environment variables.


This PR has:

  • [ y] been tested to ensure log ingestion and log query works.
  • [ y] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • [ y] added documentation for new or modified features or behaviors.

@TomBebb
Copy link
Author

TomBebb commented Nov 8, 2024

Kafka-console-producer:
Screenshot_20241108_120807

Data:
Screenshot_20241108_120822

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feat: Add native Kafka integration for Parseable server
1 participant