Ockam for Kafka

[gtato]

Hi,

I want to use Ockam to enable end-to-end encryption for Kafka, but I had a few questions which are not fully clear to me:

• Does Ockam run fully locally to the Kafka clients? Does each client need to start a service which forwards traffic to the brokers? Or there is some sort of Ockam cluster which can be used by multiple clients?
• What does the Confluent add-on actually do? Does it use Kafka to exchange information about keys between producers and consumers?
• How does key rotation work?
• Why is the add-on specific to Confluent? Can't it work for any Kafka distribution?

Thanks

[mrinalwadhwa]

Hey @gtato great questions. Thank you for trying out the examples.

We're writing some detailed documentation on this topic but in the meantime, I'll try to answer briefly here:

Why is the add-on specific to Confluent? Can't it work for any Kafka distribution?

Yes it can, we've only done the work to set things up for Confluent. Support of other distributions in on our roadmap. If you have a specific distribution you would like support for first, let us know.

Does Ockam run fully locally to the Kafka clients?

There are two parts involved that both intercept specific Kafka protocols messages to facilitate encryption.

• Kafka Inlets that run near a Kafka Client

  • we have two specialized forms of this currently - one for producers and one for the consumers

• Kafka Outlets that run near a Kafka Cluster

  • these run in Ockam Orchestrator, a managed cloud service. It's on our roadmap to make it possible to have these run in private environments.

Consumer End

When you run:

ockam service start kafka-consumer --node consumer --project-route /project/default \
  --bootstrap-server-ip 127.0.0.1 --bootstrap-server-port 4000 \
  --brokers-port-range 4001-4100

• In starts a kafka inlet and an interceptor that intercepts specific kafka protocol messages to do two things:

  • manipulate meta data so to your kafka client the kafka bootstrap server and brokers appear as though they are running on localhost.
  • decrypt messages are they fetched.

• Talks to Kafka Outlets to setup secure channels discovers bootstrap info and list of brokers. Makes the Kafka bootstrap server and brokers appear as though they are running on localhost.

• It starts a secure channel listener so that Producers can create end-to-end encrypted and mutually authenticated secure channels with it by running the Ockam Secure Channel protocol's key establishment phase https://docs.ockam.io/reference/command/secure-channels#relays

• It starts a forwarding relay in your Ockam Orchestrator project so that a producer anywhere, in any private network can establish secure channels with it.

  • Ockam Relays make is easy to traverse NATs and run end-to-end protocols between Ockam Nodes in far away private networks. Read more here and here

• decrypt messages are they fetched by your kafka client

Producer End

When you run:

ockam service start kafka-consumer --node consumer --project-route /project/default \
  --bootstrap-server-ip 127.0.0.1 --bootstrap-server-port 4000 \
  --brokers-port-range 4001-4100

• In starts a kafka inlet and an interceptor that intercepts specific kafka protocol messages to do two things:

  • manipulate meta data so to your kafka client the kafka bootstrap server and brokers appear as though they are running on localhost.
  • encrypt messages are they produced.

• Talks to Kafka Outlets to setup secure channels, discovers bootstrap info and list of brokers. Makes the Kafka bootstrap server and brokers appear as though they are running on localhost.

• Establishes a mutually authenticated secure channel with the consumer through its. Read more here about how this can work across all sorts of communication topologies - across private networks, clouds and protocols https://docs.ockam.io/reference/command/secure-channels

• Encrypts messages as they are produced by your kafka clients.

Orchestrator Project

• Run Kafka Outlets - talk to kafka bootstrap server and brokers - re-manipulate the kafka protocol metadata to talk to the actual bootstrap server and brokers
  • This communication happens using Ockam routing so it can be setup to talk to Kafka clusters in private environments https://docs.ockam.io/reference/command/routing Support for this is on our roadmap
• Receive encrypted data from Inlets over Secure Channels and write it to Kafka topics.
• Enable end-to-end secure channels between producers and consumers in private networks - with forwarding relays and routing.
• Run enrollment protocols, credential authorities and distribute ABAC authorization polices for granular access control.

How does key rotation work?

• There is no centralized key distribution server.
• Unique encryption keys are established for every secure channel, using an authenticated key establishment protocol described at the following links:
  • https://docs.ockam.io/reference/command/secure-channels#secure-channels
  • https://docs.ockam.io/reference/protocols/secure-channels
• Encryption keys are cryptographically derived and never transported over the wire.
• Encryption keys are ratchet after a fixed window of message to gain forward secrecy - this means that even if specific window of keys is compromised, only a small number of past messages are compromised.
• Ockam Secure Channels are designed for specific properties - they provide sender and receiver authentication and are resistant to key compromise impersonation attacks. They also ensures integrity and secrecy of application data and provide strong forward secrecy.

I hope that helps, please don't hesitate to ask more questions.