Skip to main content

Home

Kafka

You must configure Unravel for Kafka monitoring. The Kafka monitoring is differently configured for Kafka managed by CDH, CDP, and HDP platforms and for external Kafka.

Setting Kafka monitoring for Kafka managed by CDH, CDP, and HDP platforms

Note

In the case of multi-cluster Unravel deployment, run the following steps on the core node.

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Enable Kafka monitoring service on the Unravel UI.

    <Unravel installation directory>/unravel/manager config feature enable kafka-monitoring

  3. Add the brokers.

    Note

    To add the brokers, you will require the following:

    • Cluster key and Kafka service key

      You can run the following command to get the Cluster name and Kafka service key. 

      <Unravel installation directory>/unravel/manager config edge show
      -- Running: config edge show
------------ | ---------------------------------------- | ------------
    EDGE KEY | - local-node                             | Enabled
             |     Cluster manager:                     | Enabled
             |     Clusters:                            |
 CLUSTER KEY |       - ClusterFromTemplate              | Enabled
             |           KAFKA:                         |
 SERVICE KEY |             - kafka                      | Enabled

    • Broker ID

      For CDH and CDP platforms, go to Cloudera Manager > Kafka > Instances to get the broker IDs.

    • Broker Hostname

    • Broker port number

      By default, the Broker port number is:

      • 9092 for PLAINTEXT, SASL_PLAINTEXT protocols

      • 9093 for SASL_SSL protocol

      You can cross-check the port numbers for CDH and CDP platforms as follows:

      Go to Cloudera Manager > Kafka > Configuration and check for TCP port.

    • JMX port number

      The default JMX port for Cloudera Manager is 9398. You can cross-check the JMX port numbers for CDH and CDP platforms as follows:

      Go to Cloudera Manager > Kafka > Configuration and search for jmx_port.

    <Unravel installation directory>/unravel/manager config kafka broker add <cluster-key> <kafka-key> <broker-id> <broker-hostname> <broker-port> <JMX-port>

    For example: /opt/unravel/manager config kafka broker add ClusterFromTemplate kafka 1546333347 abc.example.com 9092 9393

  4. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  5. Start Unravel.

    <Unravel installation directory>/unravel/manager start

  6. Verify if the Kafka monitoring is enabled. 

    <Unravel installation directory> manager config feature show

    The following output is shown:

    -- Running: config feature show
kafka-monitoring     : Enabled
-- OK

  7. For CDH and CDP: Login to Cloudera Manager and go to Kafka >Configuration and search for Additional Broker Java Options. Replace the existing content with the following in the broker_java_opts text box:

    -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+DisableExplicitGC -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.useLocalHostname=true -Dcom.sun.management.jmxremote.rmi.port=9393

    Note

    In case your JMX port is not 9393, then you must change the JMX port accordingly.

    After applying all the changes, restart the services for the stale configuration.

Setting Kafka monitoring for external Kafka

To setup Unravel for external Kafka monitoring, do the following:

Note

In the case of multi-cluster Unravel deployment, run the following steps on the core node.

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. From the installation directory, create a new cluster entry.

    <Unravel installation directory>/unravel/manager config cluster add <cluster-name>

    For example: /opt/unravel/manager config add mycluster

  3. Create a Kafka cluster within that new cluster.

    <Unravel installation directory>/unravel/manager config kafka add <cluster-key> <kafka-key>

    For example: /opt/unravel/manager config kafka add mycluster mykafkacluster

  4. Add brokers to the Kafka cluster, which was created in Step 3.

    Note

    For CDH and CDP platforms, go to Cloudera Manager > Kafka > Instances to get the broker IDs. 

    <Unravel installation directory>/manager config kafka broker add <cluster-name> <kafka-key> <broker-id> <broker-host> <broker-port> <JMX-port>

    For example: /opt/unravel/manager config kafka broker add mycluster mykafka 15234 abc.example.com 9092 9393

  5. Set the Kafka properties using any of the following options:

    Refer to Kafka properties for the complete list and details of Kafka properties.

  6. Enable Kafka monitoring service on the Unravel UI.

    <Unravel installation directory>/unravel/manager config feature enable kafka-monitoring

  7. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  8. Start Unravel.

    <Unravel installation directory>/unravel/manager start

  9. Verify if the Kafka monitoring is enabled. 

    <Unravel installation directory> manager config feature show

    The following output is shown:

    -- Running: config feature show
kafka-monitoring     : Enabled
-- OK
In this section: