Skip to main content

Home

Kafka

To connect Unravel server to a Kafka cluster, set the Kafka configurations as follows:

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. From the installation directory, on the core node, in case of multi-cluster deployment, create a new cluster entry.

    <Unravel installation directory>/unravel/manager config cluster add <cluster-name>

    Example: /opt/unravel/manager config Kafka add local-node

  3. Create a Kafka cluster within that new cluster.

    manager config kafka add <cluster-name> <kafka-cluster-name>

    For example: /opt/unravel/manager config kafka add mycluster mykafkacluster

  4. Add brokers to the Kafka cluster.

    manager config kafka broker add <cluster-name> <kafka-cluster-name> <broker-id>

    For example: /opt/unravel/manager config kafka broker add mycluster mykafkacluster 123 first-broker 444 555 /opt/unravel/manager config kafka broker add mycluster mykafkacluster 234 second-broker 666 777

  5. Set the Kafka properties using any of the following options:

    • Option 1: Set properties by importing the properties file

      1. Create a file containing the properties of Kafka that you want to configure and their corresponding values. For example:

        com.unraveldata.ext.kafka.cluster.list=CLTR1,CLTR2
com.unraveldata.ext.kafka.CLTR1.bootstrap_servers=broker1-host:9092,broker2-host:9092,broker3-host:9092
com.unraveldata.ext.kafka.CLTR1.jmx_servers=kafka1,kafka2,kafka3
com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.host=broker1-host
com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.port=9393
com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.host=broker2-host
com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.port=9393
com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.host=broker3-host
com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.port=9393
com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.run_period_sec=60
com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.run_period_sec=60
com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.run_period_sec=60
com.unraveldata.ext.kafka.CLTR2.bootstrap_servers=broker1-host:9092,broker2-host:9092,broker3-host:9092
com.unraveldata.ext.kafka.CLTR2.jmx_servers=kafka1,kafka2,kafka3
com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.host=broker1-host
com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.port=9393
com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.host=broker2-host
com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.port=9393
com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.host=broker3-host
com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.port=9393
com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.run_period_sec=60
com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.run_period_sec=60
com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.run_period_sec=60
com.unraveldata.ext.kafka.insight.interval_min=5
com.unraveldata.ext.kafka.insight.lag_threshold=75
com.unraveldata.ext.kafka.insight.sw_size=10

      2. From the installation directory, where Unravel binaries are installed, run the following command and provide the path to the properties file.

        <Unravel installation directory>/unravel/manager config properties import <path to the properties file>

        For example: /opt/unravel-install/unravel/manager config properties import /opt/properties.txt

    • Option 2: Set the properties with manager config properties set

      <Unravel installation directory>/unravel/manager config properties set property key

      For example:

      <Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.cluster.list CLTR1,CLTR2
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.bootstrap_servers broker1-host:9092,broker2-host:9092,broker3-host:9092
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx_servers kafka1,kafka2,kafka3
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.host broker1-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.host broker2-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.host broker3-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.bootstrap_servers broker1-host:9092,broker2-host:9092,broker3-host:9092
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx_servers kafka1,kafka2,kafka3
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.host broker1-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.host broker2-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.host broker3-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.insight.interval_min 5
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.insight.lag_threshold 75
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.insight.sw_size 10

    Refer to Kafka properties for the complete details of Kafka properties.

  6. Enable Kafka monitoring service on the Unravel UI.

    <Unravel installation directory>/unravel/manager config feature enable kafka-monitoring

  7. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  8. Start Unravel

    <Unravel installation directory>/unravel/manager start
In this section: