Skip to main content

Home

Connecting to a Kafka cluster

To connect Unravel server to a Kafka cluster, change the configuration of the Kafka cluster, and add properties. Refer to Editing Unravel properties.

  1. Change the configuration of the Kafka cluster.

    1. Export an available port for JMX_PORT. You need to export one port for each Kafka server.

      export JMX_PORT=jmx-port

      Tip

      The default JMX port for Kafka in CDH is 9393.

      In HDP you would export this parameter in Ambari Web UI under Advanced kafka-env | kafka-env template.

    2. Enable remote access for JMX monitoring by appending the following lines to KAFKA_JMX_OPTS in kafka_run_class.sh:

      -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT
-Dcom.sun.management.jmxremote.port=$JMX_PORT
-Djava.rmi.server.hostname=public-hostname
-Djava.net.preferIPv4Stack=true

      Tip

      Not required for HDP.

    3. Verify the configuration changes on the Kafka cluster.

    4. Restart the Kafka broker.

  2. Add properties to monitor the Kafka cluster. Refer to Editing Unravel properties.

    The Unravel daemon, unravel_km, relies on a list of Kafka servers to query and discover the entire cluster. You need to add this list of Kafka servers through properties.

    For complete descriptions of properties, see Kafka Monitoring Properties.

    com.unraveldata.ext.kafka.cluster.list=cluster-list
com.unraveldata.ext.kafka.cluster-id.bootstrap_servers=boot-servers
com.unraveldata.ext.kafka.cluster-id.jmx_servers=jmx-servers
com.unraveldata.ext.kafka.cluster-id.jmx.jmx-server-id.host=jmx-hosts
com.unraveldata.ext.kafka.cluster-id.jmx.jmx-server-id.port=jmx-port

    For example,

    com.unraveldata.ext.kafka.cluster.list=c1,c2 
com.unraveldata.ext.kafka.c1.bootstrap_servers=c1-broker-host-1:9092,c1-broker-host-2:9093 
com.unraveldata.ext.kafka.c2.bootstrap_servers=c2-broker-host-1:9092,c2-broker-host-2:9093 
com.unraveldata.ext.kafka.c1.jmx_servers=kafka-test1,kafka-test2 
com.unraveldata.ext.kafka.c2.jmx_servers=kafka-test1,kafka-test2 
com.unraveldata.ext.kafka.c1.jmx.kafka-test1.host=c1-broker-host-1 
com.unraveldata.ext.kafka.c1.jmx.kafka-test2.host=c1-broker-host-2 
com.unraveldata.ext.kafka.c2.jmx.kafka-test1.host=c2-broker-host-1 
com.unraveldata.ext.kafka.c2.jmx.kafka-test2.host=c2-broker-host-2 
com.unraveldata.ext.kafka.c1.jmx.kafka-test1.port=5005 
com.unraveldata.ext.kafka.c1.jmx.kafka-test2.port=5010 
com.unraveldata.ext.kafka.c2.jmx.kafka-test1.port=5105 
com.unraveldata.ext.kafka.c2.jmx.kafka-test2.port=5110
In this section: