Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 116 additions & 2 deletions documentation/0.13.0/html/proxy-quick-start/index.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ title: Proxy quick start
version: 0.13.0
---

This quick start will guide you through deploying the proxy.
This quick start will guide you through deploying the proxy, either locally on your machine, or in a container via Docker Compose.

Kroxylicious is a Java application based on [Netty](https://netty.io/), which means it will run anywhere you can run a JVM.

# Getting started
# Running Kroxylicious locally

## Prerequisites

Expand Down Expand Up @@ -152,3 +152,117 @@ kaf -b $KROXYLICIOUS_BOOTSTRAP consume my_topic
[//]: # (====================================================================)
[//]: # (END - Use Section Examples Tabbed Card)
[//]: # (====================================================================)

# Running Kroxylicious via Docker Compose

## Prerequisites

Make sure to have either [Docker Compose](https://docs.docker.com/compose/) or [Podman Compose](https://docs.podman.io/en/stable/markdown/podman-compose.1.html) installed.

## Step 1: Create the Compose file

Create a file named _docker-compose.yaml_ with the following contents:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: It's just called compose.yaml nowadays:

The default path for a Compose file is compose.yaml (preferred) or compose.yml that is placed in the working directory. Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility of earlier versions. If both files exist, Compose prefers the canonical compose.yaml.

Suggested change
Create a file named _docker-compose.yaml_ with the following contents:
Create a file named `compose.yaml` with the following contents:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, TIL. Thanks for bringing this up!


```yaml
services:
# from https://github.com/apache/kafka/blob/trunk/docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: Use the tagged commit for Kafka 4.0.0 to keep the contents of the docker-compose.yml file stable.

Suggested change
# from https://github.com/apache/kafka/blob/trunk/docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml
# from https://github.com/apache/kafka/blob/4.0.0/docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml

kafka:
image: apache/kafka:4.0.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll probably want to extend our renovate config to bump the image refs. Does not need to be part of this PR.

ports:
- '9092:9092'
Comment on lines +171 to +172
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Do we need to expose any port here?

The (local) clients will use Kroxylicious to connect to the cluster, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, it's not needed. I still like it, as it allows local clients to connect directly, when needed.

environment:
KAFKA_NODE_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_HOST://localhost:9092,PLAINTEXT://kafka:19092'
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
KAFKA_LISTENERS: 'CONTROLLER://:29093,PLAINTEXT_HOST://:9092,PLAINTEXT://:19092'
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
networks:
- my-network
kroxylicious:
image: quay.io/kroxylicious/kroxylicious:{{ page.version }}
ports:
- 9192:9192
- 9193:9193
volumes:
- ./proxy-config.yaml:/opt/kroxylicious/config/proxy-config.yaml
command: --config=/opt/kroxylicious/config/proxy-config.yaml
networks:
- my-network
Comment on lines +192 to +201
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ended up playing with compose for an issue and ended up with

  kroxylicious:
    image: 'quay.io/kroxylicious/proxy:0.15.0-SNAPSHOT'
    container_name: kroxylicious
    command: --config /opt/kroxylicious/config/proxy-config.yaml
    ports:
      - "9192-9196:9192-9196"
    configs:
      - source: proxy-config
        target: /opt/kroxylicious/config/proxy-config.yaml

configs:
  proxy-config:
    file: ./proxy-config.yaml

The specifying the config/volumes at the top level seemed to help make things work with podman machine on a mac.

I'm not sure what syntax for ports is easier to read either. The one entry per line from your snippet vs the range on each side notation from mine.

networks:
my-network:
name: kroxylicious-network
```

This will start a single node Apache Kafka cluster as well as Kroxylicious.

## Step 2: Configure the proxy

Create a file named _proxy-config.yaml_ with the following contents:

```yaml
management:
endpoints:
prometheus: {}
virtualClusters:
- name: demo
targetCluster:
bootstrapServers: kafka:9092
gateways:
- name: mygateway
portIdentifiesNode:
bootstrapAddress: kroxylicious:9192
nodeIdRanges:
- name: mixed
start: 1
end: 1
logNetwork: false
logFrames: false
```

This lets you access the upstream Kafka cluster from _within_ the Docker network under the address `kroxylicious:9192`.
Alternatively, if you want to access the cluster from your _host_ system instead, replace the `bootstrapAddress` with `localhost:9192`.

## Step 3: Start the proxy

In the directory with both the _docker-compose.yaml_ file and the _proxy-config.yaml_ file, run the following command:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: see previous comment.

Suggested change
In the directory with both the _docker-compose.yaml_ file and the _proxy-config.yaml_ file, run the following command:
In the directory with both the `compose.yaml` file and the `proxy-config.yaml` file, run the following command:


```shell
docker compose up
```

Or, when using Podman:

```shell
podman compose up
```

## Step 4: Access Kafka through Kroxylicious

### i. Create a topic via the proxy
Create a topic called "my_topic" using the `kafka-topics.sh` command line client:
```shell
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --create --topic my_topic --bootstrap-server kroxylicious:9192
```

### ii. Produce a string to your topic via the proxy
Produce the string "hello world" to your new topic using the `kafka-console-producer.sh` command line client:
```shell
docker compose exec kafka sh -c "echo 'hello world' | /opt/kafka/bin/kafka-console-producer.sh --topic my_topic --bootstrap-server kroxylicious:9192"
```

### iii. Consume your string from your topic via the proxy
Consume your string ("hello world") from your topic ("my_topic") using the `kafka-console-consumer.sh` command line client:
```shell
docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh --topic my_topic --from-beginning --bootstrap-server kroxylicious:9192
```