Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(nifi): Bump to 2.2.0 #167

Merged
merged 10 commits into from
Mar 20, 2025
Merged

Conversation

dervoeti
Copy link
Member

@dervoeti dervoeti commented Mar 12, 2025

Part of stackabletech/nifi-operator#740

Summary of changes:

  • NiFi bumped to 2.2.0, this introduced breaking changes, since it's the bump from NiFi 1 to NiFi 2
  • Support for templates was removed in NiFi 2. To make our demos work, we now use flow versioning via the integrated GitHubFlowRegistryClient in NiFi 2.
  • The Python scripts were modified to register the GitHubFlowRegistryClient in NiFi and restore the flows of the specific demo
  • I manually converted the templates to NiFi compatible JSON definition of the contained flows. This included porting some processors from NiFi 1 to NiFi 2, e.g. replace PublishKafka_2_6 by PublishKafka, because PublishKafka_2_6 was removed. I did this by exporting the flow definition from NiFi 1, importing them into NiFi 2 and then manually replacing all processors with their NiFi 2 versions (and reconfiguring them if needed).

The branch from which the flows are restored is hardcoded, which is not ideal, but I have no good idea yet on how to make this dynamic. It must be changed if something in the .json dumps of the process groups is changed, which should rarely happen, so it's likely not a huge problem.

@dervoeti dervoeti force-pushed the bump/nifi-2.2.0-git-registry-client branch from 04ac1e8 to 700ce7b Compare March 12, 2025 20:36
@dervoeti dervoeti force-pushed the bump/nifi-2.2.0-git-registry-client branch from 700ce7b to afe10d3 Compare March 12, 2025 20:54
Copy link
Member

@razvan razvan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a full review. I was just curious.

@dervoeti dervoeti force-pushed the bump/nifi-2.2.0-git-registry-client branch from 77c4ca0 to 58333ac Compare March 18, 2025 07:36
@dervoeti dervoeti force-pushed the bump/nifi-2.2.0-git-registry-client branch 6 times, most recently from 5f0761f to 88031ce Compare March 18, 2025 16:07
@dervoeti
Copy link
Member Author

dervoeti commented Mar 18, 2025

This is ready for review now. A few things won't work in the demos, but this is related to MinIO TLS and the Waterlevel demo has a bug in the Superset measurements_per_day dataset. I'll fix both things in separate PRs. All the NiFi related stuff should work.

The current state of the code is laid out to pull the NiFi flows from the release-25.3 branch of this repo, which does not exist yet. To test it in the current state, replace all occurences of release-25.3 with bump/nifi-2.2.0-git-registry-client

@dervoeti dervoeti requested a review from razvan March 18, 2025 16:11
@dervoeti dervoeti force-pushed the bump/nifi-2.2.0-git-registry-client branch from 8739374 to fb79411 Compare March 18, 2025 16:12
@dervoeti dervoeti self-assigned this Mar 18, 2025
@razvan
Copy link
Member

razvan commented Mar 19, 2025

The current state of the code is laid out to pull the NiFi flows from the release-25.3 branch of this repo, which does not exist yet. To test it in the current state, replace all occurences of release-25.3 with bump/nifi-2.2.0-git-registry-client

How about using "main" instead if "release-25.3" ? This would at least align the flows with the rest of the manifest paths, that we always update in the release branches anyway?

@dervoeti
Copy link
Member Author

How about using "main" instead if "release-25.3" ? This would at least align the flows with the rest of the manifest paths, that we always update in the release branches anyway?

Yeah, I changed it to "main" now

Copy link
Member

@razvan razvan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works, thanks!

@dervoeti dervoeti added this pull request to the merge queue Mar 20, 2025
Merged via the queue into main with commit eb22237 Mar 20, 2025
2 checks passed
@dervoeti dervoeti deleted the bump/nifi-2.2.0-git-registry-client branch March 20, 2025 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

Successfully merging this pull request may close these issues.

2 participants