AWS IAM authentication #178

andsel · 2024-08-19T15:06:18Z

Release notes

Creation and use of new setting to load uber jars which contains cloud providers IAM authentication SASL client libraries.

What does this PR do?

Introduces the setting sasl_iam_jar_paths which is an array of library paths that points to uber jars containing all the classes needed by specific cloud provider IAM for SASL authentication.
During registration phase, uses the Ruby require to load the jars configured.

Why is it important/What is the impact to the user?

Permit to the users that deploy Logstash in a cloud provider to leverage the cloud provider's IAM to authenticate the client with SASL.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~[ ] I have made corresponding change to the default configuration files (and/or docker env variables)~~
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

check all transitive are re-packageable
- the repackaged bundled libraries are licensed with Apache 2 or MIT license, so can be repackaged without any issue.
update docs

How to test this PR locally

create an EC2 instance
create an Amazon MKS (managed Kafka service)
setup a IAM policy as described in this guide
setup on EC2 a Logstash instance to use this latest version of the plugin
download the AWS cloud provider uber jar from http://to.do
and use following configuration to proof that it's able to connect to AWS MKS using the IAM SASL authentication

input {
  kafka {
    topics => ["logstash"]
    bootstrap_servers => "Kafka endpoint as shoved during cluster creation"
    security_protocol => "SASL_SSL"
    sasl_mechanism => "AWS_MSK_IAM"
    sasl_iam_jar_paths => ["/path/to/aws_iam_uber.jar"]
    sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;"
    sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
  }
}
output {
  stdout {}
}

Related issues

Relates Exposes sasl.client.callback.handler.class Kafka client setting to select a specific SASL Handler class #177

andsel · 2024-08-23T15:29:53Z

Verified on EC2 with a policy connecting to an Amazon MKS

Running pipeline as described in the description of this PR.

And was able to create and connect client with AWS IAM

[2024-08-23T15:27:18,667][INFO ][logstash.runner          ] Log4j configuration path used is: /home/ubuntu/logstash/logstash-8.15.0/config/log4j2.properties
[2024-08-23T15:27:18,673][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.15.0", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
[2024-08-23T15:27:18,675][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-08-23T15:27:18,678][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-08-23T15:27:18,678][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-08-23T15:27:18,895][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2024-08-23T15:27:19,606][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-08-23T15:27:20,085][INFO ][org.reflections.Reflections] Reflections took 164 ms to scan 1 urls, producing 138 keys and 481 values
[2024-08-23T15:27:20,382][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2024-08-23T15:27:20,423][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/home/ubuntu/logstash/input_kafka_pipeline.conf"], :thread=>"#<Thread:0x4d4a40cf /home/ubuntu/logstash/logstash-8.15.0/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-08-23T15:27:20,974][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.55}
[2024-08-23T15:27:20,980][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2024-08-23T15:27:21,007][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2024-08-23T15:27:21,048][INFO ][org.apache.kafka.clients.consumer.ConsumerConfig][main][faab892c9600277d4182df64a79b8bb11b32e9078457a4dde3333bb1458e042d] ConsumerConfig values:
	allow.auto.create.topics = true
	auto.commit.interval.ms = 5000
	auto.include.jmx.reporter = true
	auto.offset.reset = latest
	bootstrap.servers = [boot-etkoetue.c1.kafka-serverless.us-east-1.amazonaws.com:9098]
	check.crcs = true
	client.dns.lookup = use_all_dns_ips
	client.id = logstash-0
	client.rack =
	connections.max.idle.ms = 540000
	default.api.timeout.ms = 60000
	enable.auto.commit = true
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = logstash
	group.instance.id = null
	heartbeat.interval.ms = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	internal.throw.on.fetch.stable.offset.unsupported = false
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor, class org.apache.kafka.clients.consumer.CooperativeStickyAssignor]
	receive.buffer.bytes = 32768
	reconnect.backoff.max.ms = 50
	reconnect.backoff.ms = 50
	request.timeout.ms = 40000
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler
	sasl.jaas.config = [hidden]
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.connect.timeout.ms = null
	sasl.login.read.timeout.ms = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.login.retry.backoff.max.ms = 10000
	sasl.login.retry.backoff.ms = 100
	sasl.mechanism = AWS_MSK_IAM
	sasl.oauthbearer.clock.skew.seconds = 30
	sasl.oauthbearer.expected.audience = null
	sasl.oauthbearer.expected.issuer = null
	sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
	sasl.oauthbearer.jwks.endpoint.url = null
	sasl.oauthbearer.scope.claim.name = scope
	sasl.oauthbearer.sub.claim.name = sub
	sasl.oauthbearer.token.endpoint.url = null
	security.protocol = SASL_SSL
	security.providers = null
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

[2024-08-23T15:27:21,206][INFO ][org.apache.kafka.common.security.authenticator.AbstractLogin][main][faab892c9600277d4182df64a79b8bb11b32e9078457a4dde3333bb1458e042d] Successfully logged in.
[2024-08-23T15:27:21,325][INFO ][org.apache.kafka.common.utils.AppInfoParser][main][faab892c9600277d4182df64a79b8bb11b32e9078457a4dde3333bb1458e042d] Kafka version: 3.4.1
[2024-08-23T15:27:21,326][INFO ][org.apache.kafka.common.utils.AppInfoParser][main][faab892c9600277d4182df64a79b8bb11b32e9078457a4dde3333bb1458e042d] Kafka commitId: 8a516edc2755df89
[2024-08-23T15:27:21,326][INFO ][org.apache.kafka.common.utils.AppInfoParser][main][faab892c9600277d4182df64a79b8bb11b32e9078457a4dde3333bb1458e042d] Kafka startTimeMs: 1724426841320
[2024-08-23T15:27:21,330][INFO ][org.apache.kafka.clients.consumer.KafkaConsumer][main][faab892c9600277d4182df64a79b8bb11b32e9078457a4dde3333bb1458e042d] [Consumer clientId=logstash-0, groupId=logstash] Subscribed to topic(s): logstash

kssminus · 2024-08-27T23:42:38Z

👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 👍 You are A LIFE SAVER!!

robbavey

Comment on library versions

robbavey · 2024-08-28T16:42:51Z

build.gradle

@@ -70,6 +70,32 @@ dependencies {
    implementation 'com.github.luben:zstd-jni:1.5.5-4'
    implementation 'org.lz4:lz4-java:1.8.0'
    implementation 'org.xerial.snappy:snappy-java:1.1.10.5'
+    implementation 'software.amazon.msk:aws-msk-iam-auth:1.1.4'


Can we use more modern versions of these libraries? These are somewhat out of date

Tested with commit 4c37ca9 and was able to connect.
So @robbavey please could you review it again 🙏

robbavey · 2024-10-03T21:45:15Z

build.gradle

@@ -70,6 +70,52 @@ dependencies {
    implementation 'com.github.luben:zstd-jni:1.5.5-4'
    implementation 'org.lz4:lz4-java:1.8.0'
    implementation 'org.xerial.snappy:snappy-java:1.1.10.5'
+    implementation 'software.amazon.msk:aws-msk-iam-auth:2.2.0'
+    // transitive deps used by software.amazon.msk:aws-msk-iam-auth


😳

This seems like... a lot. Are we concerned that we are going to have issues with dependency clashes with other plugins and Logstash core?

cc @jsvd

In 4c37ca9 you can see the difference switching from 1.1.4 to 2.2.0. While there is a consistent part of AWS libraries (part of groups software.amazon.awssdk and com.amazonaws) with this new version it adds:

commons-codec:commons-codec

commons-logging:commons-logging

io.netty:netty-*

org.reactivestreams:reactive-streams

org.apache.httpcomponents httpclient and httpcore

While Netty http and http2 codecs are present it was added also a dependency on Apache's httpclient so seems redundant.
The risky one is Netty, because used in other Logstash plugins.

From Logstash core we have:

httpcore-4.4.16.jar vs this 4.4.13 (no apparent conflict)

httpclient-4.5.14.jar vs this 4.5.13 (no apparent conflict)

commons-codec-1.17.0.jar vs this 1.15 API should be compatible

commons-logging-1.3.1.jar vs this 1.2 API should be compatible

Netty in Beats/TCP/HTTP inputs are at 4.1.109 vs this 4.1.108 so not so far and shouldn't break

The pluggable security classes root in 2 classes:

software.amazon.msk.auth.iam.IAMLoginModule

software.amazon.msk.auth.iam.IAMClientCallbackHandler

The IAMClientCallbackHandler explicitly instantiates MSKCredentialProvider at: https://github.com/aws/aws-msk-iam-auth/blob/v2.2.0/src/main/java/software/amazon/msk/auth/iam/IAMClientCallbackHandler.java#L54

The MSKCredentialProvider has direct dependency on 3 main AWSSDK packages:

software.amazon.awssdk.auth provided by software.amazon.awssdk:auth

software.amazon.awssdk.core provided by software.amazon.awssdk:sdk-core

software.amazon.awssdk.services.sts provide by software.amazon.awssdk:sts

Looking at the source of aws-msk-iam-auth the artifact software.amazon.awssdk:sdk-core requires the following dependency tree:

+--- software.amazon.awssdk:sdk-core:2.26.8 +--- software.amazon.awssdk:annotations:2.26.8 +--- software.amazon.awssdk:http-client-spi:2.26.8 | +--- software.amazon.awssdk:annotations:2.26.8 | +--- software.amazon.awssdk:utils:2.26.8 (*) | +--- software.amazon.awssdk:metrics-spi:2.26.8 | | +--- software.amazon.awssdk:annotations:2.26.8 | | \--- software.amazon.awssdk:utils:2.26.8 (*) | \--- org.reactivestreams:reactive-streams:1.0.4 +--- software.amazon.awssdk:metrics-spi:2.26.8 (*) +--- software.amazon.awssdk:endpoints-spi:2.26.8 | \--- software.amazon.awssdk:annotations:2.26.8 +--- software.amazon.awssdk:http-auth-spi:2.26.8 | +--- software.amazon.awssdk:annotations:2.26.8 | +--- software.amazon.awssdk:utils:2.26.8 (*) | +--- software.amazon.awssdk:http-client-spi:2.26.8 (*) | +--- org.reactivestreams:reactive-streams:1.0.4 | \--- software.amazon.awssdk:identity-spi:2.26.8 | +--- software.amazon.awssdk:annotations:2.26.8 | \--- software.amazon.awssdk:utils:2.26.8 (*) +--- software.amazon.awssdk:http-auth-aws:2.26.8 | +--- software.amazon.awssdk:annotations:2.26.8 | +--- software.amazon.awssdk:utils:2.26.8 (*) | +--- software.amazon.awssdk:identity-spi:2.26.8 (*) | +--- software.amazon.awssdk:http-client-spi:2.26.8 (*) | +--- software.amazon.awssdk:http-auth-spi:2.26.8 (*) | +--- software.amazon.awssdk:checksums-spi:2.26.8 | | \--- software.amazon.awssdk:annotations:2.26.8 | \--- software.amazon.awssdk:checksums:2.26.8 | +--- software.amazon.awssdk:annotations:2.26.8 | \--- software.amazon.awssdk:checksums-spi:2.26.8 (*) +--- software.amazon.awssdk:checksums-spi:2.26.8 (*) +--- software.amazon.awssdk:checksums:2.26.8 (*) +--- software.amazon.awssdk:identity-spi:2.26.8 (*) +--- software.amazon.awssdk:utils:2.26.8 (*) +--- software.amazon.awssdk:profiles:2.26.8 | +--- software.amazon.awssdk:utils:2.26.8 (*) | \--- software.amazon.awssdk:annotations:2.26.8 +--- software.amazon.awssdk:retries-spi:2.26.8 | +--- software.amazon.awssdk:annotations:2.26.8 | \--- software.amazon.awssdk:utils:2.26.8 (*) +--- software.amazon.awssdk:retries:2.26.8 | +--- software.amazon.awssdk:retries-spi:2.26.8 (*) | +--- software.amazon.awssdk:annotations:2.26.8 | \--- software.amazon.awssdk:utils:2.26.8 (*) \--- org.reactivestreams:reactive-streams:1.0.4

software.amazon.awssdk:sts has direct dependency on software.amazon.awssdk:apache-client and software.amazon.awssdk:netty-nio-client which kicks in all the netty related dependencies.

Despite the class MSKCredentialProvider doesn't directly refer to classes contained in software.amazon.awssdk:sso and software.amazon.awssdk:ssooidc, removing those doesn't provide any gain in terms of numerosity of dependencies.

So my conclusion is that we can't strip any jar.

bn-npakki · 2024-12-19T21:29:33Z

any update on this plugin? no updates since august? we are waiting this plugin to integrate logstash ->msk

andsel · 2025-01-17T16:32:36Z

@bn-npakki due to the large number of dependencies that thisPR transitively kicks in, we are discussing in elastic/logstash#16673 how to isolate from the rest of Logstash classpath. Once we have an agreement on how to manage it, we can move forward with this PR.

Co-authored-by: Akos Korsos <[email protected]>

… use AWS IAM authentication

…dule to build the uber-jar for AWS IAM

andsel self-assigned this Aug 19, 2024

jsvd added the status:work-in-progress label Aug 19, 2024

andsel changed the title ~~Aws iam authentication~~ AWS IAM authentication Aug 21, 2024

andsel marked this pull request as ready for review August 23, 2024 15:37

jsvd added status:needs-triage and removed status:work-in-progress labels Aug 23, 2024

andsel force-pushed the aws_iam_authentication branch from 18f8e55 to bef696e Compare August 26, 2024 08:26

robbavey requested changes Aug 28, 2024

View reviewed changes

jsvd added status:changes-requested and removed status:needs-triage labels Aug 28, 2024

andsel requested a review from robbavey September 4, 2024 17:08

robbavey reviewed Oct 3, 2024

View reviewed changes

andsel mentioned this pull request Nov 13, 2024

Dependencies pollution wrap-up elastic/logstash#16673

Open

3 tasks

andsel and others added 4 commits February 11, 2025 14:41

Added dependency on AWS IAM jars and updated documentation

7199a8b

Co-authored-by: Akos Korsos <[email protected]>

Bumped 11.5.2 version and updated Changelog to expose the settings to…

9340525

… use AWS IAM authentication

Introduce setting to load external IAM SASL providers and add a submo…

9e8ad9f

…dule to build the uber-jar for AWS IAM

Avoid to relocate AWS classes which generate runtime NoDefClassError

da05745

andsel force-pushed the aws_iam_authentication branch from 758a37d to da05745 Compare February 11, 2025 13:51

andsel added 2 commits February 12, 2025 09:37

Added sasl_iam_jar_paths setting also to Kafka output and minor changes

16cdd92

Removed exported part of uber jar creation

9760159

andsel added the enhancement New feature or request label Feb 17, 2025

Updated documentation

749d485

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS IAM authentication #178

AWS IAM authentication #178

andsel commented Aug 19, 2024 •

edited

Loading

andsel commented Aug 23, 2024 •

edited

Loading

kssminus commented Aug 27, 2024

robbavey left a comment

robbavey Aug 28, 2024

andsel Oct 2, 2024

robbavey Oct 3, 2024

andsel Oct 4, 2024

andsel Oct 4, 2024

andsel Nov 4, 2024

bn-npakki commented Dec 19, 2024

andsel commented Jan 17, 2025

AWS IAM authentication #178

Are you sure you want to change the base?

AWS IAM authentication #178

Conversation

andsel commented Aug 19, 2024 • edited Loading

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Related issues

andsel commented Aug 23, 2024 • edited Loading

kssminus commented Aug 27, 2024

robbavey left a comment

Choose a reason for hiding this comment

robbavey Aug 28, 2024

Choose a reason for hiding this comment

andsel Oct 2, 2024

Choose a reason for hiding this comment

robbavey Oct 3, 2024

Choose a reason for hiding this comment

andsel Oct 4, 2024

Choose a reason for hiding this comment

andsel Oct 4, 2024

Choose a reason for hiding this comment

andsel Nov 4, 2024

Choose a reason for hiding this comment

bn-npakki commented Dec 19, 2024

andsel commented Jan 17, 2025

andsel commented Aug 19, 2024 •

edited

Loading

andsel commented Aug 23, 2024 •

edited

Loading