MSK IAMs Authentication Support

[niccrane]

Hi there!

This is such a fantastic project and it's going to be super useful for our usecase. I was just wondering if the standard docker container has MSK IAM authentication support?

Looking at the AWS documentation, you can see an extra class is required with a few extra configuration options. Is this currently supported by kafka-gitops? If not would it be as simple as placing the MSK class in the classpath within the container and setting the required properties?

Required properties:

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Thanks in advance!

[niccrane]

Had a little play - Created the following Dockerfile

FROM devshawn/kafka-gitops

RUN cd /usr/local/openjdk-8/lib && \
    curl -LJO https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar -o /usr/local/openjdk-8/lib/.

MSK IAM Jar now exists in the docker image.
Used environment variables which look like this:

KAFKA_BOOTSTRAP_SERVERS=REDACTED:9098
KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=AWS_MSK_IAM
KAFKA_SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;
KAFKA_SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Verbose output

Generating execution plan...

11:11:26.016 [main] INFO com.devshawn.kafka.gitops.config.KafkaGitopsConfigLoader - Kafka Config: {security.protocol=SASL_SSL, sasl.mechanism=AWS_MSK_IAM, sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;, bootstrap.servers=REDACTED:9098, sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler, client.id=kafka-gitops}
11:11:26.023 [main] INFO com.devshawn.kafka.gitops.service.ConfluentCloudService - Using ccloud executable at: ccloud
11:11:26.025 [main] INFO com.devshawn.kafka.gitops.service.ParserService - Parsing desired state file...
java.lang.NullPointerException
	at com.devshawn.kafka.gitops.service.KafkaService.buildAdminClient(KafkaService.java:115)
	at com.devshawn.kafka.gitops.service.KafkaService.getAcls(KafkaService.java:41)
	at com.devshawn.kafka.gitops.manager.PlanManager.planAcls(PlanManager.java:137)
	at com.devshawn.kafka.gitops.StateManager.generatePlan(StateManager.java:90)
	at com.devshawn.kafka.gitops.StateManager.plan(StateManager.java:80)
	at com.devshawn.kafka.gitops.cli.PlanCommand.call(PlanCommand.java:38)
	at com.devshawn.kafka.gitops.cli.PlanCommand.call(PlanCommand.java:19)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1783)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2141)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975)
	at picocli.CommandLine.execute(CommandLine.java:1904)
	at com.devshawn.kafka.gitops.MainCommand.main(MainCommand.java:76)

I've even tried setting the CLASSPATH to /usr/local/openjdk-8/lib/aws-msk-iam-auth-1.1.1-all.jar with the same output.

It's possible I've missed something but it appears kafka-gitops is not compatible with MSK IAMs authentication.

I have also tested this with the same result as @nicoodle. I was however able (minimal testing so far) to enable SASL/SCRAM (in addition to IAM which we'll use for applications) and was able to get kafka-gitops working with the following environment variables after creating a Secret and associating it with the MSK Cluster:

KAFKA_BOOTSTRAP_SERVERS=XXXXXX.amazonaws.com:9096
KAFKA_SASL_JAAS_PASSWORD=XXXXXX
KAFKA_SASL_JAAS_USERNAME=XXXXXX
KAFKA_SASL_MECHANISM=SCRAM-SHA-512
KAFKA_SECURITY_PROTOCOL=SASL_SSL

Maybe not the solution you were looking for, but a work around until a better solution is available.

[mantoine96]

Hey!

We've been using kafka-gitops with MSK and IAM auth like this:

$ export KAFKA_SASL_CLIENT_CALLBACK_HANDLER_CLASS="software.amazon.msk.auth.iam.IAMClientCallbackHandler"
$ export KAFKA_SASL_JAAS_CONFIG="software.amazon.msk.auth.iam.IAMLoginModule required;"
$ export KAFKA_SASL_MECHANISM="AWS_MSK_IAM"
$ export KAFKA_SECURITY_PROTOCOL="SASL_SSL"
$ java -cp ${AWS_MSK_IAM_PLUGIN_PATH}:${KAFKA_GITOPS_PATH} com.devshawn.kafka.gitops.MainCommand ...

With AWS_MSK_IAM_PLUGIN_PATH being the path to the MSK IAM Auth plugin jar, and KAFKA_GITOPS_PATH being the path to the kafka-gitops executable.

That works really well, as long as you have valid AWS credentials :)