From 190a1fa85672009cf1e120f7aec1e46d4fb7b607 Mon Sep 17 00:00:00 2001 From: dasasathyan Date: Sun, 26 Mar 2023 18:11:07 +0530 Subject: [PATCH 1/4] 3 blog posts --- ...-03-26-Auth0-JWT-Authenticity-with-Kong.md | 120 +++++++++ ...23-03-26-Aws-Api-Gateway-with-Terraform.md | 28 ++ _posts/2023-03-26-Kafka-IaC.md | 248 ++++++++++++++++++ 3 files changed, 396 insertions(+) create mode 100644 _posts/2023-03-26-Auth0-JWT-Authenticity-with-Kong.md create mode 100644 _posts/2023-03-26-Aws-Api-Gateway-with-Terraform.md create mode 100644 _posts/2023-03-26-Kafka-IaC.md diff --git a/_posts/2023-03-26-Auth0-JWT-Authenticity-with-Kong.md b/_posts/2023-03-26-Auth0-JWT-Authenticity-with-Kong.md new file mode 100644 index 0000000000..81d99abcee --- /dev/null +++ b/_posts/2023-03-26-Auth0-JWT-Authenticity-with-Kong.md @@ -0,0 +1,120 @@ +--- +layout: post +title: "Verify Auth0 JWT Authenticity with Kong" +author: dasasathyan +categories: [ Platform Engineering, Data, Infrastructure, AWS, Cloud, API ] +featured: true +hidden: true +teaser: AWS API Gateway with Swagger in Terraform +toc: true +--- + +Kong is used to create and manage APIs, adjust for scaling, Authentication on services for protection, Traffic control to restrict inbound and outbound API traffic and a lot more. + +Auth0 is a readily available solution to authenticate and authorize services to the applications. The team and organization can avoid the cost, time, and risk that come with building their own solution to authenticate and authorize users. + +We use the JWT tokens to authenticate and authorize external services and applications. JWT is nothing but a simple JSON in encoded format. So how do we verify the authenticity of any JWT token as anyone can create a simple JSON and encode it. But there is also a hidden piece of information of the issuer of the JWT under the headers section which has the Algorithm & Token Type of the JWT. We have various ways to check for authenticity but Kong is one of the easiest ways to do it. + +Before proceeding the following are the few Prerequisites that are required + +1. Create an Auth0 account. Account name is referred to as "kong-auth0-blog" in the blog. +1. Setup a Kong instance on your machine. This guide assumes a brand new blank instance. +1. Install httpie - a http command line utility built for humans (unlike curl). + + +We need to create an API, service, consumer and a route to which the requests would be redirected to. + +1. Create API - http POST :8001/services name=example-api url=http://httpbin.org +1. Create Service - http POST :8001/services/example-api/routes hosts:='["example.com"]' +1. Create JWT Plugin - http POST :8001/services/example-api/plugins name=jwt +1. Download pem key from auth0 - http https://kong-auth0-blog.us.auth0.com/pem --download # change region accordingly +1. Transform Certificate into public key - openssl x509 -pubkey -noout -in kong-auth0-blog.pem > pubkey.pem +1. Create consumer for a user - http POST :8001/consumers username=demouser +1. Associate Consumer to public key - http post :8001/consumers/demouser/jwt algorithm=RS256 rsa_public_key@./pubkey.pem key=https://kong-auth0-blog.us.auth0.com/ -f + +The API, consumer and the plugin are created and the public key is associated with the consumer. We should now be able to make calls to the service with the generated JWT. If the JWT is valid, we get a valid response, else we see 401 or 403. +Make call using + + http GET :8000 Host:example.com Authorization:"Bearer {{TOKEN}}" -v + +The above steps work well when the kong is installed locally. What if the kong is running on a kubernetes(k8s) cluster. We need to create Custom Resource Definitions for KongConsumer, KongPlugin along with the pem key as a secret. + +It is important to have the Kong Ingress controller installed in the k8s cluster. It can be installed by following the official Kong docs https://docs.konghq.com/kubernetes-ingress-controller/latest/deployment/overview/ Also, we have a simple httpbin service running for testing. + +``` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: httpbin +spec: + progressDeadlineSeconds: 600 + replicas: 1 + revisionHistoryLimit: 10 + selector: + matchLabels: + app: httpbin + strategy: + rollingUpdate: + maxSurge: 25% + maxUnavailable: 25% + type: RollingUpdate + template: + metadata: + creationTimestamp: null + labels: + app: httpbin + spec: + containers: + - image: docker.io/kennethreitz/httpbin + imagePullPolicy: Always + name: httpbin + ports: + - containerPort: 80 + protocol: TCP + resources: {} + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + dnsPolicy: ClusterFirst + restartPolicy: Always + schedulerName: default-scheduler + securityContext: {} + terminationGracePeriodSeconds: 30 +``` + +Kong has a lot of in-built plugins to use. The list of available kong plugins can be found from https://docs.konghq.com/hub. We will first create the manifest for the Kong JWT Plugin with the following configuration. Save the following config to a yaml file and apply it using + + kubectl apply -f + + + +### jwt plugin + +``` +apiVersion: configuration.konghq.com/v1 +kind: KongPlugin +metadata: +name: app-jwt-k8s +plugin: jwt + +Next we need to add a route to our Kong Ingress and annotate the route to the plugin. + +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: +name: demo-jwt +annotations: +konghq.com/strip-path: "true" +konghq.com/plugins: app-jwt-k8s +spec: +ingressClassName: kong +rules: +- http: +paths: +- path: /bar +pathType: ImplementationSpecific +backend: +service: +name: httpbin +port: +number: 80 +``` diff --git a/_posts/2023-03-26-Aws-Api-Gateway-with-Terraform.md b/_posts/2023-03-26-Aws-Api-Gateway-with-Terraform.md new file mode 100644 index 0000000000..efe8479fad --- /dev/null +++ b/_posts/2023-03-26-Aws-Api-Gateway-with-Terraform.md @@ -0,0 +1,28 @@ +--- +layout: post +title: "AWS API Gateway with Terraform" +author: dasasathyan +categories: [ Platform Engineering, Data, Infrastructure, AWS, Cloud, API ] +featured: true +hidden: true +teaser: AWS API Gateway with Swagger in Terraform +toc: true +--- + +API Gateway helps developers to create, manage and monitor APIs at any scale between a client and one or more backend services. Amazon Web Services has its own fully managed API Gateway service with which developers can manage their APIs. It also manages thousands of concurrent API calls, traffic management, CORS support, authorization and access control, throttling, monitoring, and API version management. + +Terraform is an Infrastructure as a Code(IaC) tool. Read more about it [here](2023-03-24-Kafka-IaC.md) + +APIs can have a lot of endpoints and each endpoint would have a lot of configurations like request payload, response, success and error response. Creating and configuring such an API using the AWS console would consume a lot of time. API Gateway also lets us do that just by importing the Swagger file to API Gateway. If there are multiple APIs, importing them using the console is again time-consuming. + +An even simpler way is to use Terraform where you need to just pass the swagger file and run it. We can use Terraform's `aws_api_gateway_rest_api` resource to import an API with AWS API Gateway just by passing the swagger file as shown in the example below. + +``` + resource "aws_api_gateway_rest_api" "api_name" { + name = "websec-${terraform.workspace}" + body = templatefile("", { + ENV_VARIABLE_KEY_1 = ENV_VARIABLE_VALUE_1 + ENV_VARIABLE_KEY_2 = ENV_VARIABLE_VALUE_2 + }) + } +``` diff --git a/_posts/2023-03-26-Kafka-IaC.md b/_posts/2023-03-26-Kafka-IaC.md new file mode 100644 index 0000000000..5642159e88 --- /dev/null +++ b/_posts/2023-03-26-Kafka-IaC.md @@ -0,0 +1,248 @@ +--- +layout: post +title: "Kafka Infrastructure as Code(IaC)" +author: dasasathyan +categories: [ Platform Engineering, Data, Infrastructure, Kafka ] +featured: true +hidden: true +teaser: Provision Kafka Infrastructure with Juli-Ops, Pulumi, Terraform +toc: true +--- + +# Infrastructure as Code(IaC) + +Organizations need to automate infrastructure provisioning. On the other hand, those tools can’t be too restrictive on development teams. Development teams should have as much autonomy as possible because that's usually how developers get their best work done. It is not easy to manage Infrastructure as going wrong in any of the steps worsens the situation. Also, replicating the same set of configurations for an additional cluster is quite tedious. Here comes the Infrastructure as Code(IaC) handy. It helps us to provision the same Infrastructure across environments with the help of a source code. The advantages of using an IaC are +1. Very less time to provision infrastructure +1. Low risk of Human errors +1. Idempotency +1. Reduced configuration steps +1. Eliminate configuration drift + +A few of the IaC tools are Terraform, Pulumi etc. + +Apache Kafka is a real-time data streaming technology capable of handling trillions of events. It is a distributed system with servers and clients that communicate via a TCP network protocol. A couple of terminologies to keep in mind are: + +1. Brokers - Brokers are servers in Kafka that store event streams from various sources. A Kafka cluster is typically comprised of several brokers. Every broker in a cluster is also a bootstrap server, meaning if you can connect to one broker in a cluster, you can connect to every broker. + +1. Topics - The data is written by many processes called produces and the same are read by consumers. The data are partitioned into different partitions called topics. Kafka runs on a cluster of one or more servers called brokers and the partitions are distributed across the cluster. + +1. Kafka Connect - Messages can be copied to and from external applications and data systems with Kafka Connect. There are 2 different types of connectors. They are Source connector and Sink Connectors. + +All the above-mentioned infrastructure like Topics, Connectors etc can be configured with IaC tools like julie-ops, terraform, pulumi. + +## JulieOps + +JulieOps is an open-open source project with many contributors internally from Confluent and external to Confluent. It's a tool that allows you to describe what the configurations look like for topics, RBAC, Schema Registry etc. It is a declarative programming tool. The developers explain what is required and that it's the responsibility of the tool to decide how to get it. The Interface os JulieOps is a simple YAML file. YAMLs are easy and understandable, especially within the Kubernetes world. + +[julie-ops][julie-ops] tool helps us to provision Kafka-related tasks in Confluent Cloud Infrastructure as a code. +The related tasks are usually [Topics][Topics], [Access Control][Access Control], [Handling schemas][Handling schemas], +[ksql artifacts][ksql artifacts] etc. +All these tasks are configured as [topologies][topologies] in julie-ops. + +### Pre-Requisites + +- You need julie-ops installed locally or in docker +- Topologies +- Write the following configurations to a `.properties` file to connect to Kafka cluster: + ``` + bootstrap.servers="" + security.protocol=SASL_SSL + sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="" password=""; + ssl.endpoint.identification.algorithm=https + sasl.mechanism=PLAIN + # Required for correctness in Apache Kafka clients prior to 2.6 + client.dns.lookup=use_all_dns_ips + # Confluent Cloud Schema Registry + schema.registry.url="" + basic.auth.credentials.source=USER_INFO + schema.registry.basic.auth.user.info="":"" + ``` + +### How to run + +``` +julie-ops --broker --clientConfig --topology +``` + +Once the run is completed without any errors a successful run will look like + +``` +log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.admin.AdminClientConfig). +log4j:WARN Please initialize the log4j system properly. +log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. +List of Topics: + +List of ACLs: + +List of Principles: +List of Connectors: +List of KSQL Artifacts: +Kafka Topology updated +``` + +Want a quick start? checkout our sample JulieOps repo in [here]. + +[julie-ops]: https://julieops.readthedocs.io/en/latest/# +[Topics]: https://julieops.readthedocs.io/en/latest/futures/what-topic-management.html +[Handling schemas]: https://julieops.readthedocs.io/en/latest/futures/what-schema-management.html +[Access Control]: https://julieops.readthedocs.io/en/latest/futures/what-acl-management.html +[ksql artifacts]: https://julieops.readthedocs.io/en/latest/futures/what-ksql-management.html +[topologies]: https://julieops.readthedocs.io/en/latest/the-descriptor-files.html?highlight=topology +[here]: https://github.com/Platformatory/kafka-cd-julie + +## Pulumi + +Choosing the right IaC(Infrastructure as a Code) tool is important. Each tool has its own pros and cons. We have already seen a brief about IaC in the previous section. Here, we will be provisioning the Confluent Cloud Topics and Connectors with Pulumi. Though Pulumi supports a wide variety of programming languages like Python, Typescript, Go, C3, Java & YAML we will be using Typescript in this blog. + +### Provisioning Kafka Topics + +There are a couple of mandatory configs that are needed to create Kafka Topics. To begin with we need cluster arguments on which the infrastructure needs to be provisioned. + +``` +let clusterArgs: KafkaTopicKafkaCluster = { + id: cluster_id, +}; +``` + +Then, we will be needing the Kafka credentials the API Key & API Secret. + +``` +let clusterCredentials: KafkaTopicCredentials = { + key: kafka_api_key, + secret: kafka_api_secret, +}; +``` +Then comes the topic configs. There are various Kafka topic configurations. Read about them [here](https://kafka.apache.org/documentation/#topicconfigs) + +``` + let topic_args: KafkaTopicArgs = { + kafkaCluster: clusterArgs, + topicName: topicName.toLowerCase(), + restEndpoint: rest_endpoint, + credentials: clusterCredentials, + config: { + ["retention.ms"]: "-1", + ["retention.bytes"]: "-1", + ["num.partitions"]: "6", + }, + }; +``` + +Finally the creation of topics +``` +const topics = new confluent.KafkaTopic( + topicNames[i].toLowerCase(), + topic_args + ); +``` + +Save the above file to an `index.ts` and set the confluent cloud cluster credentials using `pulumi config set confluentcloud:cloudApiKey --secret && pulumi config set confluentcloud:cloudApiSecret --secret`. It is important to pass the `--secret` flag to the config else the secrets will not be masked on the Pulumi infrastructure config files. On setting the credentials, pulumi will prompt for a stack to be selected. Select the stack if it already exists, else create a new stack. + +Once the credentials are set run `pulumi up` command to provision the topics in confluent cloud. + +### Provisioning Kafka Connectors + +Messages from Kafka topics can be copied to and from external applications and data systems with Kafka Connect. Connectors are of two types, +Source connectors - The connector that takes data from a Producer and feeds them into a topic is called a Source connector. +Sink connectors. The connector that takes data from a Topic and delivers them to a Consumer is called a Sink Connector. + +Let’s provision a Kafka Sink Connector that writes data from a Kafka topic to an Azure Data Lake Storage(ADLS) + +The mandatory configs for provisioning a Kafka Connector are +The name of the resource +Environment +Cluster +And a couple of connector-specific configs like connector class, source topics etc. + +The configs can contain secrets like passwords, api keys, tokens. They are by default masked by Pulumi. Those configs have to be under `configSensitive` block in config and non sensitive configs have to be under `configNonsensitive` block. + +``` +let connector_args: confluent.ConnectorArgs = { + configNonsensitive: { + ["connector.class"]: "AzureDataLakeGen2Sink", # The class of the Connector. List of [supported connectors](https://docs.confluent.io/cloud/current/connectors/index.html#supported-connectors) + ["name"]: "Connector Name", + ["kafka.auth.mode"]: "KAFKA_API_KEY", + ["topics"]: topicNames, + ["input.data.format"]: "JSON", + ["output.data.format"]: "JSON", + ["time.interval"]: "HOURLY", + ["tasks.max"]: "2", + ["flush.size"]: "1000", + ["rotate.schedule.interval.ms"]: "3600000", + ["rotate.interval.ms"]: "3600000", + ["path.format"]: "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH", + ["topics.dir"]: "", + }, + configSensitive: { + ["kafka.api.key"]: kafka_api_key, + ["kafka.api.secret"]: kafka_api_secret, + ["azure.datalake.gen2.account.name"]: azure_data_lake_account_name, + ["azure.datalake.gen2.access.key"]: azure_data_lake_access_key, + }, + environment: cluster_environment, + kafkaCluster: cluster, + }; + + new confluent.Connector("pulumi-connector", connector_args); +``` + +If the Confluent Cloud cluster credentials are already set up, directly go ahead and run `pulumi up` command to provision the topics in the confluent cloud. If the Confluent Cloud cluster credentials aren’t set up, follow the steps from the topic provisioning and set them up. + +## Terraform + +Terraform is the leading IaC tool available in the market with support to various cloud, datacenter and services. It supports many cloud computing platforms like Azure, AWS, Oracle, Google and Container orchestration like Kubernetes etc. Read about the supported providers [here](https://registry.terraform.io/browse/providers). Infact pulumi provider is built on top of the official Confluent Terraform Provider. Infrastructure is provisioned with Terraform using HashiCorp Configuration Language (HCL). + +### Provisioning Topics + +First, Initialize the provider with + +``` +terraform { + required_providers { + confluent = { + source = "confluentinc/confluent" + version = "1.13.0" + } + } +} +``` +This installs the confluent cloud provider. + +Configure the confluent secrets with + +``` +provider "confluent" { + cloud_api_key = var.confluent_cloud_api_key # optionally use CONFLUENT_CLOUD_API_KEY env var + cloud_api_secret = var.confluent_cloud_api_secret # optionally use CONFLUENT_CLOUD_API_SECRET env var +} +``` + +Once the Confluent provider and the configurations are set up, run `terraform init` + +``` +resource "confluent_kafka_topic" "dev_topics" { + kafka_cluster { + id = var.cluster_id + } + for_each = toset(var.topics) + topic_name = each.value + rest_endpoint = data.confluent_kafka_cluster.dev_cluster.rest_endpoint + partitions_count = 6 + config = { + "retention.ms" = "604800000" + } + credentials { + key = var.api_key + secret = var.api_secret + } +} +``` + + +| Features | Julie-Ops | Terraform | Pulumi | +| -------- | --------- | -------- | -------- | +| Language Support | YAML | HashiCorp Configuration Language (HCL) |Python, TypeScript, JavaScript, Go, C#, F#, Java, YAML | +| Import code from other IaC tools | No | No | Yes | +| Secrets Encryption | Secrets are retrieved from `.properties` file | Secrets are stored in Vault and aren’t encrypted in the state file. | Secrets are encrypted. | +| Confluent Environment | Both Confluent Cloud & Confluent Platform | Confluent Cloud | Confluent Cloud | +| Open Sourced | Yes | Yes | Yes From 640ac5be444a1f002507316d671e3284e871f5b9 Mon Sep 17 00:00:00 2001 From: dasasathyan Date: Mon, 27 Mar 2023 11:52:37 +0530 Subject: [PATCH 2/4] resources supported --- _posts/2023-03-26-Kafka-IaC.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_posts/2023-03-26-Kafka-IaC.md b/_posts/2023-03-26-Kafka-IaC.md index 5642159e88..79e8f50b42 100644 --- a/_posts/2023-03-26-Kafka-IaC.md +++ b/_posts/2023-03-26-Kafka-IaC.md @@ -242,7 +242,8 @@ resource "confluent_kafka_topic" "dev_topics" { | Features | Julie-Ops | Terraform | Pulumi | | -------- | --------- | -------- | -------- | | Language Support | YAML | HashiCorp Configuration Language (HCL) |Python, TypeScript, JavaScript, Go, C#, F#, Java, YAML | +| Supported Resources to Provision | Topics, RBACs (for Kafka Consumers, Kafka Producers, Kafka Connect, Kafka Streams applications ( microservices ), KSQL applications, Schema Registry instances, Confluent Control Center, KSQL server instances), Schemas, ACLs | confluent_api_key, confluent_byok_key, confluent_cluster_link, confluent_connector, confluent_environment, confluent_identity_pool, confluent_identity_provider, confluent_invitation, confluent_kafka_acl, confluent_kafka_client_quota, confluent_kafka_cluster, confluent_kafka_cluster_config, confluent_kafka_mirror_topic, confluent_kafka_topic, confluent_ksql_cluster, confluent_network, confluent_peering, confluent_private_link_access, confluent_role_binding, confluent_schema, confluent_schema_registry_cluster, confluent_schema_registry_cluster_config, confluent_schema_registry_cluster_mode, confluent_service_account, confluent_subject_config, confluent_subject_mode, confluent_transit_gateway_attachment | ApiKey, ByokKey, ClusterLink, Connector, Environment, IdentityPool, IdentityProvider, Invitation, KafkaAcl, KafkaClientQuota, KafkaCluster, KafkaClusterConfig, KafkaMirrorTopic, KafkaTopic, KsqlCluster, Network, Peering, PrivateLinkAccess, Provider, RoleBinding, Schema, SchemaRegistryCluster, SchemaRegistryClusterConfig, SchemaRegistryClusterMode, ServiceAccount, SubjectConfig, SubjectMode, TransitGatewayAttachment | | Import code from other IaC tools | No | No | Yes | | Secrets Encryption | Secrets are retrieved from `.properties` file | Secrets are stored in Vault and aren’t encrypted in the state file. | Secrets are encrypted. | | Confluent Environment | Both Confluent Cloud & Confluent Platform | Confluent Cloud | Confluent Cloud | -| Open Sourced | Yes | Yes | Yes +| Open Sourced | Yes | Yes | Yes | From cddb0fe055485d6039dcfb35014b0f44a138fe93 Mon Sep 17 00:00:00 2001 From: dasasathyan Date: Thu, 13 Apr 2023 11:34:49 +0530 Subject: [PATCH 3/4] addressed comments --- _posts/2023-03-26-Kafka-IaC.md | 38 ++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/_posts/2023-03-26-Kafka-IaC.md b/_posts/2023-03-26-Kafka-IaC.md index 79e8f50b42..f3156a37f3 100644 --- a/_posts/2023-03-26-Kafka-IaC.md +++ b/_posts/2023-03-26-Kafka-IaC.md @@ -11,12 +11,7 @@ toc: true # Infrastructure as Code(IaC) -Organizations need to automate infrastructure provisioning. On the other hand, those tools can’t be too restrictive on development teams. Development teams should have as much autonomy as possible because that's usually how developers get their best work done. It is not easy to manage Infrastructure as going wrong in any of the steps worsens the situation. Also, replicating the same set of configurations for an additional cluster is quite tedious. Here comes the Infrastructure as Code(IaC) handy. It helps us to provision the same Infrastructure across environments with the help of a source code. The advantages of using an IaC are -1. Very less time to provision infrastructure -1. Low risk of Human errors -1. Idempotency -1. Reduced configuration steps -1. Eliminate configuration drift +Organizations require automated infrastructure provisioning, but the tools used should not impose too many restrictions on development teams. Developers typically do their best work with a high degree of autonomy. Managing infrastructure is not an easy task, and any missteps can worsen the situation. Replicating configurations for additional clusters is also a tedious process. This is where Infrastructure as Code (IaC) comes in handy. By using source code, IaC helps to provision the same infrastructure across environments. The advantages of IaC include faster infrastructure provisioning, reduced risk of human errors, idempotency, fewer configuration steps, and the elimination of configuration drift. A few of the IaC tools are Terraform, Pulumi etc. @@ -24,15 +19,21 @@ Apache Kafka is a real-time data streaming technology capable of handling trilli 1. Brokers - Brokers are servers in Kafka that store event streams from various sources. A Kafka cluster is typically comprised of several brokers. Every broker in a cluster is also a bootstrap server, meaning if you can connect to one broker in a cluster, you can connect to every broker. -1. Topics - The data is written by many processes called produces and the same are read by consumers. The data are partitioned into different partitions called topics. Kafka runs on a cluster of one or more servers called brokers and the partitions are distributed across the cluster. +2. Topics - The data is written by many processes called produces and the same are read by consumers. The data are partitioned into different partitions called topics. Kafka runs on a cluster of one or more servers called brokers and the partitions are distributed across the cluster. -1. Kafka Connect - Messages can be copied to and from external applications and data systems with Kafka Connect. There are 2 different types of connectors. They are Source connector and Sink Connectors. +3. Kafka Connect - Messages can be copied to and from external applications and data systems with Kafka Connect. There are 2 different types of connectors. They are Source connector and Sink Connectors. + +4. Schema Registry - Schema Registry is a centralized repository that facilitates the management and validation of schemas for messages in Kafka topics. With Schema registry, producers and consumers of Kafka topics can ensure that data is consistent and compatible as schemas evolve over time. + +5. Kafka Streams - Kafka Streams library provides real-time stream processing capabilities, built on top of the Kafka producer and consumer APIs. It is used to perform real-time data processing, apply various transformations, perform powerful aggregations on the messages. + +6. kSql - Similar to K Streams, KSQL is used to perform filtering, aggregations, joins, and windowing operations, and generate real-time analytics and data transformations against Kafka Topics with SQL like interface. All the above-mentioned infrastructure like Topics, Connectors etc can be configured with IaC tools like julie-ops, terraform, pulumi. ## JulieOps -JulieOps is an open-open source project with many contributors internally from Confluent and external to Confluent. It's a tool that allows you to describe what the configurations look like for topics, RBAC, Schema Registry etc. It is a declarative programming tool. The developers explain what is required and that it's the responsibility of the tool to decide how to get it. The Interface os JulieOps is a simple YAML file. YAMLs are easy and understandable, especially within the Kubernetes world. +JulieOps, formally known as Kafka Topology Builder, is an open source project licensed under MIT License. It has got over 350+ stars on github. It is a tool designed to simplify the process of configuring topics, role-based access control (RBAC), Schema Registry, and other components. JulieOps is based on declarative programming principles, which means that developers can specify what is needed, and the tool takes care of the implementation details. The interface of JulieOps is a YAML file, which is known for its user-friendliness and straightforwardness. With JulieOps, developers can easily describe their configuration requirements and delegate the rest of the work to the tool. [julie-ops][julie-ops] tool helps us to provision Kafka-related tasks in Confluent Cloud Infrastructure as a code. The related tasks are usually [Topics][Topics], [Access Control][Access Control], [Handling schemas][Handling schemas], @@ -92,7 +93,7 @@ Want a quick start? checkout our sample JulieOps repo in [here]. ## Pulumi -Choosing the right IaC(Infrastructure as a Code) tool is important. Each tool has its own pros and cons. We have already seen a brief about IaC in the previous section. Here, we will be provisioning the Confluent Cloud Topics and Connectors with Pulumi. Though Pulumi supports a wide variety of programming languages like Python, Typescript, Go, C3, Java & YAML we will be using Typescript in this blog. +Selecting the appropriate Infrastructure as Code (IaC) tool is crucial, as each tool has its own advantages and disadvantages. As discussed earlier, IaC helps in automating infrastructure provisioning and eliminates the possibility of human errors. In this section, we will be using Pulumi to provision Confluent Cloud Topics and Connectors. While Pulumi supports various programming languages, such as Python, Typescript, Go, C#, Java, and YAML, we will be using Typescript in this blog post. By utilizing Pulumi, we can automate the deployment process and achieve faster and more reliable infrastructure provisioning. The provider is developed utilizing the official Terraform Provider from ConfluentInc and is accessible across multiple languages and platforms. ### Provisioning Kafka Topics @@ -142,7 +143,7 @@ Once the credentials are set run `pulumi up` command to provision the topics in ### Provisioning Kafka Connectors -Messages from Kafka topics can be copied to and from external applications and data systems with Kafka Connect. Connectors are of two types, +Kafka Connect allows for the seamless integration of messages from Kafka topics with external applications and data systems. Connectors come in two types: Source connectors and Sink connectors. Source connectors - The connector that takes data from a Producer and feeds them into a topic is called a Source connector. Sink connectors. The connector that takes data from a Topic and delivers them to a Consumer is called a Sink Connector. @@ -190,7 +191,7 @@ If the Confluent Cloud cluster credentials are already set up, directly go ahead ## Terraform -Terraform is the leading IaC tool available in the market with support to various cloud, datacenter and services. It supports many cloud computing platforms like Azure, AWS, Oracle, Google and Container orchestration like Kubernetes etc. Read about the supported providers [here](https://registry.terraform.io/browse/providers). Infact pulumi provider is built on top of the official Confluent Terraform Provider. Infrastructure is provisioned with Terraform using HashiCorp Configuration Language (HCL). +Terraform is a widely-used IaC tool that supports a range of cloud, datacenter, and service providers. It is capable of provisioning infrastructure for popular cloud platforms such as Azure, AWS, Oracle, Google, and Kubernetes orchestration. A list of supported providers can be found on the official Terraform Registry. It's worth noting that the Pulumi provider is built on top of the Confluent Terraform Provider. Terraform uses the HashiCorp Configuration Language (HCL) to describe and provision infrastructure. ### Provisioning Topics @@ -217,7 +218,7 @@ provider "confluent" { } ``` -Once the Confluent provider and the configurations are set up, run `terraform init` +Once the Confluent provider and the configurations are set up, run `terraform init` and `terraform apply`. ``` resource "confluent_kafka_topic" "dev_topics" { @@ -238,12 +239,19 @@ resource "confluent_kafka_topic" "dev_topics" { } ``` - | Features | Julie-Ops | Terraform | Pulumi | | -------- | --------- | -------- | -------- | | Language Support | YAML | HashiCorp Configuration Language (HCL) |Python, TypeScript, JavaScript, Go, C#, F#, Java, YAML | | Supported Resources to Provision | Topics, RBACs (for Kafka Consumers, Kafka Producers, Kafka Connect, Kafka Streams applications ( microservices ), KSQL applications, Schema Registry instances, Confluent Control Center, KSQL server instances), Schemas, ACLs | confluent_api_key, confluent_byok_key, confluent_cluster_link, confluent_connector, confluent_environment, confluent_identity_pool, confluent_identity_provider, confluent_invitation, confluent_kafka_acl, confluent_kafka_client_quota, confluent_kafka_cluster, confluent_kafka_cluster_config, confluent_kafka_mirror_topic, confluent_kafka_topic, confluent_ksql_cluster, confluent_network, confluent_peering, confluent_private_link_access, confluent_role_binding, confluent_schema, confluent_schema_registry_cluster, confluent_schema_registry_cluster_config, confluent_schema_registry_cluster_mode, confluent_service_account, confluent_subject_config, confluent_subject_mode, confluent_transit_gateway_attachment | ApiKey, ByokKey, ClusterLink, Connector, Environment, IdentityPool, IdentityProvider, Invitation, KafkaAcl, KafkaClientQuota, KafkaCluster, KafkaClusterConfig, KafkaMirrorTopic, KafkaTopic, KsqlCluster, Network, Peering, PrivateLinkAccess, Provider, RoleBinding, Schema, SchemaRegistryCluster, SchemaRegistryClusterConfig, SchemaRegistryClusterMode, ServiceAccount, SubjectConfig, SubjectMode, TransitGatewayAttachment | | Import code from other IaC tools | No | No | Yes | | Secrets Encryption | Secrets are retrieved from `.properties` file | Secrets are stored in Vault and aren’t encrypted in the state file. | Secrets are encrypted. | -| Confluent Environment | Both Confluent Cloud & Confluent Platform | Confluent Cloud | Confluent Cloud | | Open Sourced | Yes | Yes | Yes | +| Github Stars | 350+ | 81 | 6 | +| State Store | Stored in `.cluster-state` file | Stored in `.tfstate` file or Backend of user's choice | Managed by Pulumi Service and Backend of user's choice | + +# Conclusion + +Although Terraform remains the dominant Infrastructure as Code (IaC) tool in the industry, Pulumi is rapidly gaining traction. Each tool has its strengths and weaknesses, with Terraform being more established and providing a wider range of resources, while Pulumi is renowned for its ease of use and its growing community, which is continuously improving its functionality. A person who possesses coding experience but lacks familiarity with infrastructure as code tools may find Pulumi to be a more straightforward or fascinating tool to start with due to its support for multiple programming languages, including Python, TypeScript, JavaScript, Go, C#, F#, Java, and YAML. + +A suitable tool ultimately depends on specific needs, such as prioritizing stability and access to a vast resource and knowledge base, in which case Terraform may be the superior option, or prioritizing efficiency and the ability to use a familiar language, in which case Pulumi might be the ideal solution. Regardless of the chosen tool, both can effectively help in managing infrastructure code. + From b08390c457635d1114b4348631a265c0095dd4be Mon Sep 17 00:00:00 2001 From: dasasathyan Date: Thu, 13 Apr 2023 11:35:18 +0530 Subject: [PATCH 4/4] removed kong authenticity & API GW --- ...-03-26-Auth0-JWT-Authenticity-with-Kong.md | 120 ------------------ ...23-03-26-Aws-Api-Gateway-with-Terraform.md | 28 ---- 2 files changed, 148 deletions(-) delete mode 100644 _posts/2023-03-26-Auth0-JWT-Authenticity-with-Kong.md delete mode 100644 _posts/2023-03-26-Aws-Api-Gateway-with-Terraform.md diff --git a/_posts/2023-03-26-Auth0-JWT-Authenticity-with-Kong.md b/_posts/2023-03-26-Auth0-JWT-Authenticity-with-Kong.md deleted file mode 100644 index 81d99abcee..0000000000 --- a/_posts/2023-03-26-Auth0-JWT-Authenticity-with-Kong.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -layout: post -title: "Verify Auth0 JWT Authenticity with Kong" -author: dasasathyan -categories: [ Platform Engineering, Data, Infrastructure, AWS, Cloud, API ] -featured: true -hidden: true -teaser: AWS API Gateway with Swagger in Terraform -toc: true ---- - -Kong is used to create and manage APIs, adjust for scaling, Authentication on services for protection, Traffic control to restrict inbound and outbound API traffic and a lot more. - -Auth0 is a readily available solution to authenticate and authorize services to the applications. The team and organization can avoid the cost, time, and risk that come with building their own solution to authenticate and authorize users. - -We use the JWT tokens to authenticate and authorize external services and applications. JWT is nothing but a simple JSON in encoded format. So how do we verify the authenticity of any JWT token as anyone can create a simple JSON and encode it. But there is also a hidden piece of information of the issuer of the JWT under the headers section which has the Algorithm & Token Type of the JWT. We have various ways to check for authenticity but Kong is one of the easiest ways to do it. - -Before proceeding the following are the few Prerequisites that are required - -1. Create an Auth0 account. Account name is referred to as "kong-auth0-blog" in the blog. -1. Setup a Kong instance on your machine. This guide assumes a brand new blank instance. -1. Install httpie - a http command line utility built for humans (unlike curl). - - -We need to create an API, service, consumer and a route to which the requests would be redirected to. - -1. Create API - http POST :8001/services name=example-api url=http://httpbin.org -1. Create Service - http POST :8001/services/example-api/routes hosts:='["example.com"]' -1. Create JWT Plugin - http POST :8001/services/example-api/plugins name=jwt -1. Download pem key from auth0 - http https://kong-auth0-blog.us.auth0.com/pem --download # change region accordingly -1. Transform Certificate into public key - openssl x509 -pubkey -noout -in kong-auth0-blog.pem > pubkey.pem -1. Create consumer for a user - http POST :8001/consumers username=demouser -1. Associate Consumer to public key - http post :8001/consumers/demouser/jwt algorithm=RS256 rsa_public_key@./pubkey.pem key=https://kong-auth0-blog.us.auth0.com/ -f - -The API, consumer and the plugin are created and the public key is associated with the consumer. We should now be able to make calls to the service with the generated JWT. If the JWT is valid, we get a valid response, else we see 401 or 403. -Make call using - - http GET :8000 Host:example.com Authorization:"Bearer {{TOKEN}}" -v - -The above steps work well when the kong is installed locally. What if the kong is running on a kubernetes(k8s) cluster. We need to create Custom Resource Definitions for KongConsumer, KongPlugin along with the pem key as a secret. - -It is important to have the Kong Ingress controller installed in the k8s cluster. It can be installed by following the official Kong docs https://docs.konghq.com/kubernetes-ingress-controller/latest/deployment/overview/ Also, we have a simple httpbin service running for testing. - -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - name: httpbin -spec: - progressDeadlineSeconds: 600 - replicas: 1 - revisionHistoryLimit: 10 - selector: - matchLabels: - app: httpbin - strategy: - rollingUpdate: - maxSurge: 25% - maxUnavailable: 25% - type: RollingUpdate - template: - metadata: - creationTimestamp: null - labels: - app: httpbin - spec: - containers: - - image: docker.io/kennethreitz/httpbin - imagePullPolicy: Always - name: httpbin - ports: - - containerPort: 80 - protocol: TCP - resources: {} - terminationMessagePath: /dev/termination-log - terminationMessagePolicy: File - dnsPolicy: ClusterFirst - restartPolicy: Always - schedulerName: default-scheduler - securityContext: {} - terminationGracePeriodSeconds: 30 -``` - -Kong has a lot of in-built plugins to use. The list of available kong plugins can be found from https://docs.konghq.com/hub. We will first create the manifest for the Kong JWT Plugin with the following configuration. Save the following config to a yaml file and apply it using - - kubectl apply -f - - - -### jwt plugin - -``` -apiVersion: configuration.konghq.com/v1 -kind: KongPlugin -metadata: -name: app-jwt-k8s -plugin: jwt - -Next we need to add a route to our Kong Ingress and annotate the route to the plugin. - -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: -name: demo-jwt -annotations: -konghq.com/strip-path: "true" -konghq.com/plugins: app-jwt-k8s -spec: -ingressClassName: kong -rules: -- http: -paths: -- path: /bar -pathType: ImplementationSpecific -backend: -service: -name: httpbin -port: -number: 80 -``` diff --git a/_posts/2023-03-26-Aws-Api-Gateway-with-Terraform.md b/_posts/2023-03-26-Aws-Api-Gateway-with-Terraform.md deleted file mode 100644 index efe8479fad..0000000000 --- a/_posts/2023-03-26-Aws-Api-Gateway-with-Terraform.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -layout: post -title: "AWS API Gateway with Terraform" -author: dasasathyan -categories: [ Platform Engineering, Data, Infrastructure, AWS, Cloud, API ] -featured: true -hidden: true -teaser: AWS API Gateway with Swagger in Terraform -toc: true ---- - -API Gateway helps developers to create, manage and monitor APIs at any scale between a client and one or more backend services. Amazon Web Services has its own fully managed API Gateway service with which developers can manage their APIs. It also manages thousands of concurrent API calls, traffic management, CORS support, authorization and access control, throttling, monitoring, and API version management. - -Terraform is an Infrastructure as a Code(IaC) tool. Read more about it [here](2023-03-24-Kafka-IaC.md) - -APIs can have a lot of endpoints and each endpoint would have a lot of configurations like request payload, response, success and error response. Creating and configuring such an API using the AWS console would consume a lot of time. API Gateway also lets us do that just by importing the Swagger file to API Gateway. If there are multiple APIs, importing them using the console is again time-consuming. - -An even simpler way is to use Terraform where you need to just pass the swagger file and run it. We can use Terraform's `aws_api_gateway_rest_api` resource to import an API with AWS API Gateway just by passing the swagger file as shown in the example below. - -``` - resource "aws_api_gateway_rest_api" "api_name" { - name = "websec-${terraform.workspace}" - body = templatefile("", { - ENV_VARIABLE_KEY_1 = ENV_VARIABLE_VALUE_1 - ENV_VARIABLE_KEY_2 = ENV_VARIABLE_VALUE_2 - }) - } -```