Skip to content
This repository has been archived by the owner on Sep 2, 2024. It is now read-only.

Is there any way to use Broker/Trigger without creating additional Kafka topics? #233

Open
casssoft opened this issue Nov 25, 2020 · 14 comments
Open

Is there any way to use Broker/Trigger without creating additional Kafka topics? #233

casssoft opened this issue Nov 25, 2020 · 14 comments
Labels
kind/feature-request triage/accepted Issues which should be fixed (post-triage)

Comments

@casssoft
Copy link
Contributor

casssoft commented Nov 25, 2020 •
edited
Loading

Problem

In my cluster I already have a Kakfa topic managed through a non-knative system. I want a Broker/Trigger setup to read events off that kafka topic and filter them to different services, BUT ideally I don't want to have the overhead of an extra kafka topic just to copy the event into the Broker system.

This is a follow up from #215 (comment)

The reason I want to use the Broker/Trigger setup is that it supports event filtering. Also it seems like the best way to use knative eventing when you have a big bucket of events that you want to separate out to multiple knative services

Here's the best ways I could tackle this that I know of:

  • Just deal with the overhead: Create a KafkaSource to send events to a Broker that's implemented with an MTChannelBasedBroker + KafkaChannel which would create a new kafka topic (Or Apache Kafka broker which I think would work the same way). The downside for this is that there would be an extra kafka topic and I assume extra latency, worse throughput, and more data usage.
  • Avoid using a Broker and create multiple KafkaSources with filtering to go to different knative services. (Thanks @pierDipi for the suggestion.) The downside is that this is a little bit harder to manage since each KafkaSource needs all the info to be able to connect to the kafka cluster (bootstrap servers/sasl/tls).
  • Try to force the KafkaChannel that underpins the broker to use the kafka topic that already exists. It seems like this is not the intended use case and currently has issues because knative will try to delete the topic (Related to KafkaChannel Topic Deletion Handling #215)
  • Refactor the cluster so that Knative is in charge of the kafka topic instead of the external system and have the external system push either directly into the topic or into some type of source.
  • Create a new broker implementation to use existing kafka topics.

Persona:
Event consumer

Exit Criteria
A clear path for this use case.

(Sorry that this is vague, part of the reason for this issue is to move the discussion out of issue 215 and hopefully help guide others in a similar position.)

Additional context (optional)
See the discussion ending with this comment: #215 (comment)

@pierDipi Let me know if you had something else in mind when you said to create an issue.

Also I really appreciate the effort to guide me in the right direction!
@lionelvillard
Copy link
Contributor

/cc @aslom

@github-actions
Copy link

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 24, 2021
@lionelvillard
Copy link
Contributor

/remove-lifecycle stale

@knative-prow-robot knative-prow-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 24, 2021
@slinkydeveloper
Copy link
Contributor

I believe the eventing-kafka-broker can easily support this use case with a simple flag that just doesn't create a topic if it's not there... WDYT @pierDipi?

@pierDipi
Copy link
Member

pierDipi commented Mar 8, 2021

Yeah, we can support it.
We just need to decide the API surface and then it could even be a good first contribution.

@slinkydeveloper
Copy link
Contributor

@pierDipi can you open an issue? Whatever API is fine for me, even an annotation

@github-actions
Copy link

github-actions bot commented Jun 8, 2021

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 8, 2021
@pierDipi
Copy link
Member

cc @itsmurugappan, this should be related to what you were sharing on #eventing-delivery regarding using a user-provided topic.

@pierDipi pierDipi mentioned this issue Jun 10, 2021
Closed
@pierDipi
Copy link
Member

I opened an issue knative-extensions/eventing-kafka-broker#977

@itsmurugappan
Copy link
Contributor

itsmurugappan commented Jun 10, 2021 •
edited
Loading

yes @pierDipi this is what kafka-topic-channel addresses and this is how I defined the channel spec -> https://github.com/Optum/kafka-topic-channel/blob/main/samples/pingsourceevents/channel-cm.yaml. Broker works the same.

@github-actions github-actions bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 11, 2021
matzew pushed a commit to matzew/eventing-kafka that referenced this issue Jun 21, 2021
@cardil @lberk @devguyio
[v0.22.0] Backport PR 563 (knative-extensions#233)
8ce9b2e 
* [v0.22.0] Backport of PR 563 (use of knative-extensions#215)

* Backport of (knative-sandbox#476)

Dispatcher readiness e2e test

* Parameterize Partitions created via MustCreateTopic
* Add kafkaChannel creation/get obj helper functions
* Add subscription scale readiness test
* Add explicit t.Run statement
* Move test to helper function, use channelTestRunner
* Change from subscriptionv1beta1 to subscriptionv1 (abstraction prep)
* Support subscription v1beta1 and v1
* Fix copyright header
* Change channel creation depnding on typemeta
  Remove subscription v1beta1 testing, having multiple tests run at once
  will change the dispatcher replication in parallel
* Drop unneeded parameter
* Add a blank line
* Fix linter complaints
* Add build constraint tag to mark testing consolidated channel only
* run hack/update-codegen.sh
* run hack/update-codegen.sh
* Modify build constraints for test
* Remove v1alpha1 references as requested
* Fix old reference
* Change subscription creation function reference

* Backport of (knative-sandbox#563)

Continual tests for KafkaSource for upgrade testing

* Using knative/eventing#4815 to override wathola sender transport
* Actual way of replacing an image used for wathola sender
* Reformatting
* Wathola Kafka Sender utilizing plain connection
* Implementing SUT and custom sender
* KafkaSource upgrade test
* Update knative-eventing to 8270497041f2
* Using source config template
* Export kafkaTopicEndpoint to be able to read on TOML
* wathola config reader can read both string and map[string]interface{} for sender.address
* KafkaSource upgrade test works
* Allow to influence Kafka cluster connection details
* Channel tests working
* Removing unused variable name
* Review knative/eventing#5321 changes
* Updates after review of knative/eventing#4815
* Use latest integration/eventing-sandbox/eventing-kafka/pr-563 commit
* Update deps
* Update deps
* Refactor after review
* Shorten the TypeMeta field
* Tweak the merge slices func after review

* Adding generated test images

Co-authored-by: Lukas Berk <[email protected]>

* Restore manually changed vendor files

* Skipping subscription readiness when scaling dispatcher test (knative-sandbox#573)

Signed-off-by: Ahmed Abdalla <[email protected]>

Co-authored-by: Lukas Berk <[email protected]>
Co-authored-by: Ahmed Abdalla Abdelrehim <[email protected]>
@github-actions
Copy link

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 10, 2021
@pierDipi
Copy link
Member

/remove-lifecycle stale

@knative-prow-robot knative-prow-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 10, 2021
@github-actions
Copy link

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 10, 2021
@pierDipi
Copy link
Member

pierDipi commented Jan 4, 2022

/remove-lifecycle stale
/triage accepted

@knative-prow-robot knative-prow-robot added triage/accepted Issues which should be fixed (post-triage) and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 4, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/feature-request triage/accepted Issues which should be fixed (post-triage)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@casssoft @lionelvillard @slinkydeveloper @itsmurugappan @pierDipi @knative-prow-robot