Skip to main content

Home

Databricks workspace setup guide

This section provides instructions to connect a Databricks cluster to Unravel SaaS.

  1. Create a Workspace token in Databricks.

    1. Go to Workspace > Admin Console > Access Control and enable Personal Access Tokens. For more details, refer to Manage personal access tokens.

      saas-azure-personal-access-token.png

    2. Go to Workspace > User Settings > Access Tokens and click Generate New Token. For more details, refer to Authentication using Databricks personal access tokens.

      saas-azure-personal-access-token1.png

  2. Register a new Databricks workspace or edit details of an existing Databricks workspace.

    1. Sign in to Unravel UI, and from the upper right, click manage-icon.png> Workspaces. The Workspaces Manager page is displayed.

    2. Click Add Workspace. The Add Workspace dialog box is displayed. Enter the following details:

      saas-new-workspace.png

      Field

      Description

      Workspace Id

      Databricks workspace ID, which can be found in the Databricks URL.

      The random numbers shown after o= in the Databricks URL become the workspace ID.

      For example, in this URL:https://<databricks-instance>/?o=987654321123456, the Databricks workspace ID is the random number after o=, which is 987654321123456.

      Workspace Name

      Databricks workspace name. A human-readable name for the workspace. For example, ACME-Workspace

      Instance (Region) URL

      Regional URL where the Databricks workspace is deployed. Specify the complete URL. For example: https://dbc-1dbx661f-a33e.cloud.databricks.com

      Tier

      Select a subscription option: Standard or Premium. For Databricks Azure, you can get the pricing information from the Azure portal. For Databricks AWS you can get detailed information about pricing tiers from Databricks AWS pricing.

      Token

      Use the personal access token to secure authentication to the Databricks REST APIs instead of passwords. You can generate the token from the workspace URL (Go to Settings > User Settings > Access Token > Generate New Token)

      See Authentication using Databricks personal access tokens to create personal access tokens.

      Note

      Users with admin or non-admin roles can create personal access tokens.

      Note

      After you click Add, it takes around 2-3 minutes to register the Databricks Workspace with Unravel.

  3. Configure Databricks cluster with Unravel.

    On the Databricks instance, go to Configure Cluster >  Advanced Options and update the following configurations. These configurations must be updated for every cluster (Automated /Interactive) in your workspace.

    saas-azure-workspace-adv-options.png

    • Spark

      Copy the following snippet to Spark > Spark Conf. Replace Unravel DNS or IP Address.

      spark.eventLog.enabled true
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
spark.unravel.server.hostport <Unravel DNS or IP Address>:4043
spark.unravel.shutdown.delay.ms 300
spark.executor.extraJavaOptions  
-Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3
spark.driver.extraJavaOptions  
-Dcom.unraveldata.client.rest.request.timeout.ms=1000 
-Dcom.unraveldata.client.rest.conn.timeout.ms=1000 
-javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3

    • Logging

      Select DBFS as Destination, and maintain the default Cluster Log Path.

      dbfs:/cluster-logs/

    • Init Scripts

      saas-azure-workspace-adv-options-init.png

      In the Init Scripts tab, select Destination as DBFS. Copy the following as the Init script path and click Add.

      dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh
In this section: