Confluent Cloud Connectors

[rohits-dev]

Hello

Do you have plans for adding features to add source / sink managed connectors to this provider?

Thanks

[rohits-dev]

@Mongey
Hi I see you use https://github.com/cgroschupp/go-client-confluent-cloud
How do you know the underlying confluent api, I couldn't find any documentation of it?

[rjudin]

@Mongey
Hi I see you use https://github.com/cgroschupp/go-client-confluent-cloud
How do you know the underlying confluent api, I couldn't find any documentation of it?

good question :)
in my understanding instead of direct

• tf-provider <-> ConfluentCloud_API
  we are observing
• tf-provider <-> go-client-of-ccloud <-> ConfluentCloud_API

Why? Because (currently) available API methods of ConfluentCloud are behind CLI (ccloud).
CLI documentation? use verbose mode to ccloud xx yy -vvvv - example

[mlovic-earnin]

Confluent has now added a Connect API to their REST API documentation and marked it as released for General Availability. https://confluent.cloud/api/docs#tag/Connectors-(v1)

[askoriy]

There is similar issue in terraform-provider-kafka-connect repo
Mongey/terraform-provider-kafka-connect#33

[albrechtflo-hg]

I added the required functions to the base library, and created a Pull Request here, for this provider: #82

[askoriy]

@albrechtflo-hg I tried to create connectors with your implementation and found two bugs:

1. There is no way to hide any sensitive data during create connector, all are printed as plain-text
   I think the solution could be providing separate map variable sensitive_config besides config with merging both maps into final connector confiugration.

2. After creating connector next run of terraform plan show the difference:

  # confluentcloud_connector.connector will be updated in-place
  ~ resource "confluentcloud_connector" "connector" {
        cluster_id     = "lkc-51gd2"
      ~ config         = {
            "cloud.environment"            = "prod"
            "cloud.provider"               = "gcp"
            "connector.class"              = "PubSubSource"
            "errors.tolerance"             = "all"
          ~ "gcp.pubsub.credentials.json"  = <<~EOT
              - ****************
              + {
              +   "type": "service_account",
              +   "project_id": "test-project-1234",
              +   "private_key_id": "bd1a89a58853e9e819d3d99a26083aa6e39fe634",
              <...>
              + }
            EOT
            "gcp.pubsub.max.retry.time"    = "5"
            "gcp.pubsub.message.max.count" = "1000"
            "gcp.pubsub.project.id"        = "test-project-1234"
            "gcp.pubsub.subscription.id"   = "test-project-1234-subscription1"
            "gcp.pubsub.topic.id"          = "topic1"
            "internal.kafka.endpoint"      = "PLAINTEXT://kafka-0.kafka.pkc-4r297.svc.cluster.local:9071,kafka-1.kafka.pkc-4r297.svc.cluster.local:9071,kafka-2.kafka.pkc-4r297.svc.cluster.local:9071"
            "kafka.api.key"                = "****************"
            "kafka.api.secret"             = "****************"
            "kafka.dedicated"              = "false"
            "kafka.endpoint"               = "SASL_SSL://pkc-4r297.europe-west1.gcp.confluent.cloud:9092"
            "kafka.region"                 = "europe-west1"
            "kafka.topic"                  = "topic1"
            "name"                         = "connector-name"
          - "schema.registry.url"          = "https://psrc-lgy7n.europe-west3.gcp.confluent.cloud" -> null
            "tasks.max"                    = "1"
            "valid.kafka.api.key"          = "true"
        }
        environment_id = "env-ab123"
        id             = "connector-name"
        name           = "connector-name"
    }

Plan: 0 to add, 1 to change, 0 to destroy.

futher terraform apply doesn't solve the difference

[askoriy]

I tried to eliminate these problems with PR #97 and #98

[Samarthramesh5]

Hello,
Need to use IAM Roles for s3 authorization for confluent cloud, anything on this in terraform?
I dont want to use (AccessKey and Id).