Configuring internal hive metastore (Databricks)
This topic provides instructions to configure the Databricks internal hive metastore with Unravel.
Prerequisites
You must have admin privileges for Databricks workspace.
Enable Web Terminal access for the workspace. This is needed only once for getting the internal hive metastore details. To enable Web terminal access, do the following:
In your Databricks workspace, select the user profile dropdown from the top right and then select Admin Console > Workspace Settings.
Click the Web Terminal toggle.
To configure the Databricks internal hive metastore with Unravel, do the following:
Create a single-node cluster on Databricks and start it. This cluster is needed only once and can be deleted after you get the internal hive metastore configurations.
After the cluster is started, launch a web terminal for that cluster. On the Cluster page, click the Apps tab, and then click Launch Web Terminal.
After launching the web terminal, using a vi editor, open the
/databricks/hive/conf/hive-site.xml
file.Copy the values of the following properties from the
hive-site.xml
file and create a temporary.txt
or.prop
file.Following is a sample of such a file:
com.unraveldata.hive.metastore.list=<workspacename1>,<workspacename2> hive.metastore.cluster.ids=default hive.metastore.<workspacename1>.workspace.ids=<workspace1_ID> javax.jdo.option.<workspacename1>.ConnectionURL=jdbc:mariadb://<db_instance_name>:3306/organizationxxxxxxxxxx?useSSL=true&enabledSslProtocolSuites=TLSv1,TLSv1.1,TLSv1.2&serverSslCert=/tmp/mysql-ssl-ca-cert.crt javax.jdo.option.<workspacename1>.ConnectionDriverName=org.mariadb.jdbc.Driver javax.jdo.option.<workspacename1>.ConnectionUserName=XXXxxxXXXXX4UXXXXXXXxxxxXXXX javax.jdo.option.<workspacename1>.ConnectionPassword=XXXxxxXXXXX4UXXXXXXXxxxxXXXX hive.metastore.<workspacename2>.workspace.ids=<workspace2_ID> javax.jdo.option.<workspacename2>.ConnectionURL=jdbc:mariadb://<db_instance_name>:3306/organizationxxxxxxxxxx?useSSL=true&enabledSslProtocolSuites=TLSv1,TLSv1.1,TLSv1.2&serverSslCert=/tmp/mysql-ssl-ca-cert.crt javax.jdo.option.<workspacename2>.ConnectionDriverName=org.mariadb.jdbc.Driver javax.jdo.option.<workspacename2>.ConnectionUserName=XXXxxxXXXXX4UXXXXXXXxxxxXXXX javax.jdo.option.<workspacename2>.ConnectionPassword=XXXxxxXXXXX4UXXXXXXXxxxxXXXX
For the ConnectionURL, you must provide a valid SSL certificate path. Ensure to change the path of the SSL certificate in the JDBC URL to the path of the Unravel node where the SSL certificate is downloaded.
Note
SSL certificate for Azure Databricks can be downloaded from here.
Using a terminal from the installation directory, where Unravel binaries are installed, run the following command to import the temporary properties file you created.
<Unravel installation directory>/unravel/manager config properties import <path to the properties file> For example: /opt/unravel-install/unravel/manager config properties import /opt/properties.txt
Ensure to stop Unravel.
<Unravel installation directory>/unravel/manager stop
Apply the changes.
<Unravel installation directory>/unravel/manager config apply
Start Unravel.
<Unravel installation directory>/unravel/manager start