Nice to have: import current state

[BitProcessor]

Use case: running the tool against an existing Kafka cluster setup with some ACLs/topics already configure in plan mode (with a minimal state file) currently shows all the ACLs/topics that would be removed. While it is already possible to copy/paste this output and do a little bit of find/replace and other basic edits to come to a new state file that would keep these changes, it would be nice if:

• either the output is in copy/paste'able Yaml

or

• the command would have an import mode to generate an initial state.yaml from the current ACL configuration and current topic setup.

I can imagine that there are other potential users out there, with a running Kafka setup with historical, manual changes, that would like to switch to a GitOps approach. In my case, it was a rather fresh cluster, so replicating the current state manually into a state.yaml file was rather easy.

I think that the "output" proposal would be a rather fast fix?

[devshawn]

I agree, this is definitely a feature I'd like to have implemented and a completely valid use-case. Generating the topics block of the YAML is easy. The ACL section is a bit harder, due to how the services block is a lot simpler than direct ACLs.

I think a base implementation could be added that can handle consumer/producer/streams applications easily. ACLs that are completely custom, outside of our generated ones via the services block, could be easily imported to the custom ACL block. For connect, we'd likely need to implement #10 first.

I think a reasonable first approach would be to make an import function that generates a starter state.yaml. It might require some cleanup on the user's side after, depending on how complicated their ACL setup is, but the plan should be correct if run. I might take a stab at it this weekend and see what I can come up with.

[jrdi]

@devshawn any news here? It would be great to be able to easily generate the current state file with a single command 😄

[jaykatti]

@devshawn Hello Shawn, Would you pls. have any update on this feature of "Generating the current state of the cluster" ?

[jaykatti]

I am trying to use this on one of our clusters. Currently in PoC mode. This feature of "Generating the current state" would be a very crucial one. Kindly let us know when we can try that.

[stv-io]

I know that this is not a solution, but given the setup I'm working on I "generated" the state from the API of kafdrop using a script, consuming <kafdrop_url>/topics

#!/usr/bin/python3

import requests
import sys

config = { 
  "dev": "https://kafka-monitor.dev.example.com",
  "stage": "https://kafka-monitor.stage.example.com" 
}

excluded_topcis = ["__consumer_offsets"]

def help_and_exit():
    print(f"\nPlease re-run and provide one of the following clusters ..\n\n{[key for key in config.keys()]}\n")
    sys.exit(1)

def main():

  try:
    cluster = sys.argv[1]
  except IndexError:
    help_and_exit()
  
  if cluster not in config.keys():
    help_and_exit()
  
  url = config[cluster] + "/topic"
  
  response = requests.get(url)
  topics = response.json()

  file = "states/" + cluster + ".yaml"
  f = open(file,"w+")
  f.write("topics:\n")
  for t in topics:
    if t['name'] not in excluded_topcis:
      f.write(f"  {t['name']}:\n")
      f.write(f"    partitions: {len(t['partitions'])}\n")
      f.write(f"    replication: 3\n")
      if len(t['config']) > 0:
        f.write(f"    configs:\n")
        for k,v in t['config'].items():
          f.write(f"      {k}: {v}\n")
  f.close()

  print(f"\nState file generated into {file} ..\n")

if __name__ == '__main__':
  main()

not pretty, but for my case, get's the job done, on request.

[jaykatti]

I am trying to use this on one of our clusters. Currently in PoC mode. This feature of "Generating the current state" would be a very crucial one. Kindly let us know when we can try that.

I wrote a custom java program to generate the current state.yaml file for the ACL's.