3 blog posts #4
3 blog posts #4
Conversation
dasasathyan
requested review from
ashsridhar-platformatory,
ashwinvenkatesan229,
badri,
p6 and
p6-platformatory
_posts/2023-03-26-Kafka-IaC.md
Outdated
|
|
||
| 1. Topics - The data is written by many processes called produces and the same are read by consumers. The data are partitioned into different partitions called topics. Kafka runs on a cluster of one or more servers called brokers and the partitions are distributed across the cluster. | ||
|
|
||
| 1. Kafka Connect - Messages can be copied to and from external applications and data systems with Kafka Connect. There are 2 different types of connectors. They are Source connector and Sink Connectors. |
There was a problem hiding this comment.
Please also write about other clients for Kafka available as part of Confluent, Conduktor etc.
There was a problem hiding this comment.
Please write a decent tabular list of what resources need to be managed in the context of Kafka.
Examples (this is to help you and not an exhaustive list, so obv not to copy-pasta)
- Kafka Clusters
- Topics
- ACLs
- Schemas
etc
_posts/2023-03-26-Kafka-IaC.md
Outdated
|
|
||
| # Infrastructure as Code(IaC) | ||
|
|
||
| Organizations need to automate infrastructure provisioning. On the other hand, those tools can’t be too restrictive on development teams. Development teams should have as much autonomy as possible because that's usually how developers get their best work done. It is not easy to manage Infrastructure as going wrong in any of the steps worsens the situation. Also, replicating the same set of configurations for an additional cluster is quite tedious. Here comes the Infrastructure as Code(IaC) handy. It helps us to provision the same Infrastructure across environments with the help of a source code. The advantages of using an IaC are |
There was a problem hiding this comment.
Please formulate the sentences little better. You can probably use ChatGPT's help in writing definitions and advantages.
|
|
||
| 1. Kafka Connect - Messages can be copied to and from external applications and data systems with Kafka Connect. There are 2 different types of connectors. They are Source connector and Sink Connectors. | ||
|
|
||
| All the above-mentioned infrastructure like Topics, Connectors etc can be configured with IaC tools like julie-ops, terraform, pulumi. |
There was a problem hiding this comment.
Need to talk more about why we are comparing JulieOps, Terraform and Pulumi. Also some words about the basis for our comparison
| } | ||
| } | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Conclusion on the three different IaC and suggestions on when to use them. A scenario where one of them is more appropriate than others
|
|
||
| ### How to run | ||
|
|
||
| ``` |
There was a problem hiding this comment.
Talk about the things to keep in mind (or nuances) while defining the topology. I am assuming topology is the one which defines the resources to be provisioned.
There was a problem hiding this comment.
Basically, this is a useless section that talks about what the JulieOps examples already have. You should atleast talk through some concepts (such as the declarative topology) and why they are useful
| Save the above file to an `index.ts` and set the confluent cloud cluster credentials using `pulumi config set confluentcloud:cloudApiKey <cloud api key> --secret && pulumi config set confluentcloud:cloudApiSecret <cloud api secret> --secret`. It is important to pass the `--secret` flag to the config else the secrets will not be masked on the Pulumi infrastructure config files. On setting the credentials, pulumi will prompt for a stack to be selected. Select the stack if it already exists, else create a new stack. | ||
|
|
||
| Once the credentials are set run `pulumi up` command to provision the topics in confluent cloud. | ||
|
|
There was a problem hiding this comment.
We can highlight the difference among the different IaC tools with respect to configurations, defining resources, storing state etc.
| cloud_api_secret = var.confluent_cloud_api_secret # optionally use CONFLUENT_CLOUD_API_SECRET env var | ||
| } | ||
| ``` | ||
|
|
There was a problem hiding this comment.
I think terraform apply needs to be done for the changes to apply.
| ``` | ||
|
|
||
| If the Confluent Cloud cluster credentials are already set up, directly go ahead and run `pulumi up` command to provision the topics in the confluent cloud. If the Confluent Cloud cluster credentials aren’t set up, follow the steps from the topic provisioning and set them up. | ||
|
|
There was a problem hiding this comment.
Confluent has an official terraform registry for Confluent cloud but not Confluent platform. Like that, please talk a little about which IaC tool is right for which use case
There was a problem hiding this comment.
You need to talk about Kafka components (and not just Connect); Connect is just one example.
|
@dasasathyan - please only bring up Kafka IAC. The other two are not close to ready at this point. |
| All the above-mentioned infrastructure like Topics, Connectors etc can be configured with IaC tools like julie-ops, terraform, pulumi. | ||
|
|
||
| ## JulieOps | ||
|
|
There was a problem hiding this comment.
JulieOps is not a programming tool.
Please talk something useful about JulieOps. "Internally from confluent and external to confluent" -- is a complete wordsoup. Like who the heck is the guy?
How long has the project been around? How many stars on github?
who else uses it?
Is it mature? Would you recommend it? What is your subjective assessment? What are the advantages and disadvantages?
_posts/2023-03-26-Kafka-IaC.md
Outdated
|
|
||
| ## JulieOps | ||
|
|
||
| JulieOps is an open-open source project with many contributors internally from Confluent and external to Confluent. It's a tool that allows you to describe what the configurations look like for topics, RBAC, Schema Registry etc. It is a declarative programming tool. The developers explain what is required and that it's the responsibility of the tool to decide how to get it. The Interface os JulieOps is a simple YAML file. YAMLs are easy and understandable, especially within the Kubernetes world. |
There was a problem hiding this comment.
YAMLs are understandable in K8s world has no relevance to a description of JulieOps.
| ## Pulumi | ||
|
|
||
| Choosing the right IaC(Infrastructure as a Code) tool is important. Each tool has its own pros and cons. We have already seen a brief about IaC in the previous section. Here, we will be provisioning the Confluent Cloud Topics and Connectors with Pulumi. Though Pulumi supports a wide variety of programming languages like Python, Typescript, Go, C3, Java & YAML we will be using Typescript in this blog. | ||
|
|
There was a problem hiding this comment.
Again. Very generic. You need to talk about the Pulumi provider for Kafka. How does it work, advantages/disadv, maturity etc (same as JulieOps);
_posts/2023-03-26-Kafka-IaC.md
Outdated
|
|
||
| ## Terraform | ||
|
|
||
| Terraform is the leading IaC tool available in the market with support to various cloud, datacenter and services. It supports many cloud computing platforms like Azure, AWS, Oracle, Google and Container orchestration like Kubernetes etc. Read about the supported providers [here](https://registry.terraform.io/browse/providers). Infact pulumi provider is built on top of the official Confluent Terraform Provider. Infrastructure is provisioned with Terraform using HashiCorp Configuration Language (HCL). |
There was a problem hiding this comment.
As asked for JulieOps, please come up with a decent exploration of
- Why Terraform provider
- Which Terraform provider: There is Mongey/Kafka, and Confluent official; What are the differences?
- Pros,Cons
- Maturity
- Your subjective assessment
| ``` | ||
|
|
||
|
|
||
| | Features | Julie-Ops | Terraform | Pulumi | |
There was a problem hiding this comment.
add
- maturity (github stars, adopters)
- differentiators
Created 3 blogs