Skip to content

index.markdown

Latest commit

 

History

History
268 lines (203 loc) · 11.8 KB

index.markdown

File metadata and controls

268 lines (203 loc) · 11.8 KB
layout quickstart
title Proxy quick start
version 0.13.0

This quick start will guide you through deploying the proxy, either locally on your machine, or in a container via Docker Compose.

Kroxylicious is a Java application based on Netty, which means it will run anywhere you can run a JVM.

Running Kroxylicious locally

Prerequisites

Java

To get started deploying Kroxylicious, you will need to install a Java Runtime Environment (JRE) with minimum version 17. This does not come included with Kroxylicious.

Some operating systems come with a JRE already installed. You can check what Java version you have installed by running the following command:

java -version

{% capture jre_note %} If you get an error or the command doesn't return anything, you may not have a JRE installed, or your JRE may not have been correctly added to your system's PATH variable. {% endcapture %}

{% include bs-alert.html type="primary" icon="info-circle-fill" content=jre_note %}

Apache Kafka®

You will also need a running Apache Kafka® cluster for Kroxylicious to proxy. The official Apache Kafka® quick start has instructions for setting up a local bare metal cluster.

Once your cluster is set up, the cluster bootstrap address used by Kroxylicious can be changed in the configuration YAML file (see the Configure section below).

Step 1: Download and install the proxy

Kroxylicious can be downloaded from the releases page of the Kroxylicious GitHub repository, or from Maven Central.

Download [kroxylicious-app-{{ page.version }}-bin.zip](https://github.com/kroxylicious/kroxylicious/releases/download/v{{ page.version }}/kroxylicious-app-{{ page.version }}-bin.zip) from the [GitHub release page](https://github.com/kroxylicious/kroxylicious/releases/tag/v{{ page.version }}). Optionally, verify the contents of the package with the attached [kroxylicious-app-{{ page.version }}-bin.zip.asc](https://github.com/kroxylicious/kroxylicious/releases/download/v{{ page.version }}/kroxylicious-app-{{ page.version }}-bin.zip.asc) file.

{% capture os_archive_note %} If you're trying Kroxylicious out on Linux or macOS, you may find the .tar.gz format easier to work with. We're using the .zip files in this quick start for cross-platform compatibility, but we recommend you use whichever format you're most familiar with. {% endcapture %}

{% include bs-alert.html type="primary" icon="info-circle-fill" content=os_archive_note %}

Extract the downloaded Kroxylicious Zip file into the directory you would like to install Kroxylicious in. Ensure the kroxylicious-start.sh and run-java.sh files in the bin/ directory within the extracted folder have at least read and execute (r-x) permissions for the owner.

Step 2: Configure the proxy

Kroxylicious is configured with YAML. From the configuration file you can specify how Kroxylicious presents each Apache Kafka® broker to clients, where Kroxylicious will locate the Apache Kafka® cluster(s) to be proxied, and which filters Kroxylicious should use along with any configuration for those filters.

An example configuration file can be found in the config/ directory of the extracted Kroxylicious folder, which you can either modify or use as reference for creating your own configuration file.

For this quickstart we will use Kroxylicious in Port-Per-Broker configuration, and assume that both your Apache Kafka® cluster and clients are running on your local machine and using their default configuration. This means we can use the example proxy config file that comes with Kroxylicious.

If your machine uses a non-standard port configuration, or if you have used custom settings for your Apache Kafka® cluster (or if your cluster is running on a different machine) you will need to adjust your Kroxylicious configuration accordingly. More information about configuring Kroxylicious can be found in the documentation.

Step 3: Start the proxy

From within the extracted Kroxylicious folder, run the following command:

./bin/kroxylicious-start.sh --config config/example-proxy-config.yaml

To use your own configuration file instead of the example, just replace the file path after --config.

Step 4: Configure Kafka clients to use it

Your client(s) will need to point to the proxy (using the configured address) rather than directly at the Apache Kafka® cluster.

{% capture use_examples_htmlid %}{{ page.htmlid }}-useExamples{% endcapture %}

{% capture apacheClientContent %} In each command below, substitute $KROXYLICIOUS_BOOTSTRAP for the bootstrap address of your proxy instance.

i. Create a topic via the proxy

Create a topic called "my_topic" using the kafka-topics.sh command line client:

bin/kafka-topics.sh --create --topic my_topic --bootstrap-server $KROXYLICIOUS_BOOTSTRAP

ii. Produce a string to your topic via the proxy

Produce the string "hello world" to your new topic using the kafka-console-producer.sh command line client:

echo "hello world" | bin/kafka-console-producer.sh --topic my_topic --bootstrap-server $KROXYLICIOUS_BOOTSTRAP

iii. Consume your string from your topic via the proxy

Consume your string ("hello world") from your topic ("my_topic") using the kafka-console-consumer.sh command line client:

bin/kafka-console-consumer.sh --topic my_topic --from-beginning --bootstrap-server $KROXYLICIOUS_BOOTSTRAP

{% endcapture %}

{% capture kafClientContent %} In each command below, substitute $KROXYLICIOUS_BOOTSTRAP for the bootstrap address of your proxy instance.

i. Create a topic with Kaf via the proxy

Create a topic called "my_topic" using Kaf:

kaf -b $KROXYLICIOUS_BOOTSTRAP topic create my_topic

ii. Produce a string to your topic with Kaf via the proxy

Produce the string "hello world" to your new topic:

echo "hello world" | kaf -b $KROXYLICIOUS_BOOTSTRAP produce my_topic

iii. Consume your string from your topic with Kaf via the proxy

Consume your string ("hello world") from your topic ("my_topic"):

kaf -b $KROXYLICIOUS_BOOTSTRAP consume my_topic

{% endcapture %}

{% capture apache_htmlid %}{{use_examples_htmlid}}-apacheClient{% endcapture %} {% capture kaf_htmlid %}{{use_examples_htmlid}}-kafClient{% endcapture %}

{% capture use_example_tabs %} {% include nested-card-tabbed/tab.html htmlid=apache_htmlid is_active_tab=true tab_title="Using Kroxylicious with the Apache Kafka® command line clients" %} {% include nested-card-tabbed/tab.html htmlid=kaf_htmlid is_active_tab=false tab_title="Using Kroxylicious with the Kaf command line client" %} {% endcapture %}

{% capture use_example_content %} {% include nested-card-tabbed/tab-content.html htmlid=apache_htmlid tabindex=0 is_active_tab=true content=apacheClientContent %} {% include nested-card-tabbed/tab-content.html htmlid=kaf_htmlid tabindex=1 is_active_tab=false content=kafClientContent %} {% endcapture %}

{% include nested-card-tabbed/card.html htmlid=use_examples_htmlid tabs=use_example_tabs content=use_example_content %}

Running Kroxylicious via Docker Compose

Prerequisites

Make sure to have either Docker Compose or Podman Compose installed.

Step 1: Create the Compose file

Create a file named docker-compose.yaml with the following contents:

services:
  # from https://github.com/apache/kafka/blob/trunk/docker/examples/docker-compose-files/single-node/plaintext/docker-compose.yml
  kafka:
    image: apache/kafka:4.0.0
    ports:
      - '9092:9092'
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_HOST://localhost:9092,PLAINTEXT://kafka:19092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
      KAFKA_LISTENERS: 'CONTROLLER://:29093,PLAINTEXT_HOST://:9092,PLAINTEXT://:19092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
    networks:
      - my-network
  kroxylicious:
    image: quay.io/kroxylicious/kroxylicious:{{ page.version }}
    ports:
      - 9192:9192
      - 9193:9193
    volumes:
      - ./proxy-config.yaml:/opt/kroxylicious/config/proxy-config.yaml
    command: --config=/opt/kroxylicious/config/proxy-config.yaml
    networks:
      - my-network
networks:
  my-network:
    name: kroxylicious-network

This will start a single node Apache Kafka cluster as well as Kroxylicious.

Step 2: Configure the proxy

Create a file named proxy-config.yaml with the following contents:

management:
  endpoints:
    prometheus: {}
virtualClusters:
  - name: demo
    targetCluster:
      bootstrapServers: kafka:9092
    gateways:
    - name: mygateway
      portIdentifiesNode:
        bootstrapAddress: kroxylicious:9192
        nodeIdRanges:
        - name: mixed
          start: 1
          end: 1
    logNetwork: false
    logFrames: false

This lets you access the upstream Kafka cluster from within the Docker network under the address kroxylicious:9192. Alternatively, if you want to access the cluster from your host system instead, replace the bootstrapAddress with localhost:9192.

Step 3: Start the proxy

In the directory with both the docker-compose.yaml file and the proxy-config.yaml file, run the following command:

docker compose up

Or, when using Podman:

podman compose up

Step 4: Access Kafka through Kroxylicious

i. Create a topic via the proxy

Create a topic called "my_topic" using the kafka-topics.sh command line client:

docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --create --topic my_topic --bootstrap-server kroxylicious:9192

ii. Produce a string to your topic via the proxy

Produce the string "hello world" to your new topic using the kafka-console-producer.sh command line client:

docker compose exec kafka  sh -c "echo 'hello world' | /opt/kafka/bin/kafka-console-producer.sh --topic my_topic --bootstrap-server kroxylicious:9192"

iii. Consume your string from your topic via the proxy

Consume your string ("hello world") from your topic ("my_topic") using the kafka-console-consumer.sh command line client:

docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh --topic my_topic --from-beginning --bootstrap-server kroxylicious:9192