Skip to main content

Home

Kafka security

You can improve the Kafka cluster security by having Kafka authenticate connections to brokers from clients using either SSL or SASL.

SSL + Kerberos for Kafka clients

Prerequisite: Kafka brokers are configured with SSL and Kerberos. Refer to your Hadoop provider's documentation for configuring SSL and Kerberos for Kafka brokers.

SSL+Kerberos is supported by new Kafka consumers and producers. The configuration is the same for consumers and producers. Replace items in red with values specific/relevant to your environment.

For single Kafka clients

  1. Create a file named consumerConfig.properties. Add the following properties. You can locate your SSL + Kerberos configuration.

    security.protocol=SASL_SSL
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
ssl.enabled.protocols=TLSvl.2,TLSvl.l,TLSvl
ssl.truststore.location=/usr/java/latest/jre/lib/security/jssecacerts
ssl.truststore.password=changeit
ssl.keystore.location=/opt/cloudera/security/pki/keystore.jks
ssl.keystore.password=changeit
sasl.jaas.config=\
com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true \
useTicketCache=true \
keyTab="/opt/unravel/kafka.keytab" \
principal="kafka/kafkavm.unraveldata.com@UNRAVELDATA.COM";

    Note

    Only if Kerberos is enabled, the security.protocol is SASL_PLAINTEXT and the SSL properties can be removed.

  2. Copy/move consumerConfig.properties file to <Unravel_installation_directory/unravel/data/conf.

  3. Check the Kafka cluster name you added against com.unraveldata.ext.kafka.cluster.list property while configuring Kafka connection.

  4. Use the manager command from the Unravel installation directory to set the com.unraveldata.ext.kafka.<ClusterName>.consumer.config property. 

    <Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.<ClusterName>.consumer.config <Unravel_installation_directory>/unravel/data/conf

    Replace <clustername> with the name provided in the com.unraveldata.ext.kafka.cluster.list property.

    <ClusterName> is the user-defined name of the Kafka cluster, which is used to determine the Kafka cluster on the Unravel UI.

  5. Apply changes. 

    <Unravel installation directory>/unravel/manager config apply

  6. Restart Unravel.

    <Unravel installation directory>/unravel/manager restart kafka_monitor
For multiple Kafka clients

Warning

Each cluster must have a separate consumerConfig.properties files.

  1. Check the Kafka cluster names you added against com.unraveldata.ext.kafka.cluster.list property while configuring Kafka connection. The property is defined with a comma-separated list. If there is only one cluster name see above.

  2. Create a file named consumerConfig<ClusterName>.properties for each cluster. Replace <clustername> with the names in the com.unraveldata.ext.kafka.cluster.list property. 

    security.protocol=SASL_SSL
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
ssl.enabled.protocols=TLSvl.2,TLSvl.l,TLSvl
ssl.truststore.location=/usr/java/latest/jre/lib/security/jssecacerts
ssl.truststore.password=changeit
ssl.keystore.location=/opt/cloudera/security/pki/keystore.jks
ssl.keystore.password=changeit
sasl.jaas.config=\
com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true \
useTicketCache=true \
keyTab="/opt/unravel/kafka.keytab" \
principal="kafka/kafkavm.unraveldata.com@UNRAVELDATA.COM";

    Note

    Only if Kerberos is enabled, the security.protocol is SASL_PLAINTEXT and the SSL properties can be removed.

  3. Copy/move each file to <Unravel_installation_directory/unravel/data/conf.

  4. Add the following property for each cluster. 

    <Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.ClusterName.consumer.config <Unravel_installation_directory>/unravel/data/conf/consumerConfigClusterName.properties

  5. Apply changes. 

    <Unravel installation directory>/unravel/manager config apply

  6. Restart the Kafka monitor daemon.

    <Unravel installation directory>/unravel/manager restart kafka_monitor
Kafka authorizations

Unravel consumes messages to topic __consumer_offsets using the consumer group UnravelOffsetConsumer.

Sentry authorization

The following privilege must be granted using sentry:

HOST=*->CONSUMERGROUP=UnravelOffsetConsumer→action=read
HOST=*->CONSUMERGROUP=UnravelOffsetConsumer→action=write
HOST=*->CONSUMERGROUP=UnravelOffsetConsumer→action=describe
HOST=*->TOPIC=__consumer_offsets→action=read
HOST=*->TOPIC=__consumer_offsets→action=write
HOST=*->TOPIC=__consumer_offsets->action=describe

For further details, see Using Kafka with Sentry Authorization in the Cloudera Distribution of Apache Kafka documentation.

Kafka with Ranger authorization

The following privilege must be granted using Ranger for the topic __consumer_offsets.

Publish
Consume
Describe

For further details, see Security - Create a Kafka Policy in the HDP Security Guide.

References

For further information, see Apache Kafka documentation chapter # 7 Security.

In this section: