Microsoft Azure Databricks (Manual)

This topic explains how to deploy Unravel on Microsoft Azure Databricks walking you through the following procedures.

Verify you meet the prerequisites for installation of Azure Databricks

Create Azure components

Install Unravel

Configure and restart Unravel

Configure Databricks with Unravel

Configure Databricks with Unravel

For Reference:

Installing Databricks Sensors using Setup script

Uninstalling Unravel server and sensors on Databricks

Prerequisites

Platform

Each version of Unravel has specific platform requirements. See to Unravel's Databricks compatibility matrix confirm your Azure Databricks platform meets the requirements for the version of Unravel that you're installing.

Sizing

Minimum VM type suggested: Medium memory optimized such as Standard_E8s_v3

Important

You must have separate nodes for the Unravel server and the external MySQL database.

Unravel Server

Architecture: x86_64
vm.max_map_count is set to 262144
Minimum requirements for cores, RAM, and disks:
The table below lists the minimum requirements for cores, RAM, and disks for a typical environment with default data retention and lookback settings.
/usr/local/unravel is the storage location for Unravel binaries. /srv/unravel is used for Elasticsearch (ES) and the bundled database.
In production environments, put /usr/local/unravel and /srv/unravel on separate disks. Putting /srv/unravel on a separate high spin HDD with its own SATAIII (or equivalent) bus significantly increases IO bandwidth.
If /usr/local/unravel or /srv/unravel doesn't have the minimum free space shown in the table below, create symbolic links for them to another disk. To check the space on a volume use the df command. For example,
```
df -h /srv
```
Jobs per day
Cores
RAM
/usr/local/unravel
/srv/unravel
Less than
50,000
8
96 GB
8 GB free
500 GB free
50,000 to
100,000 to
8
128GB
8 GB free
500 GB free
Over 100,000
Contact Unravel Support
All volumes are mounted.
/tmp is mounted with executable permissions. To re-mount /tmp with executable permissions use the following command:
```
mount -o remount,exec /tmp
```
Maintain the nproc limit to unlimited.
Add unravel.conf file with the following settings to /etc/security//limits.d:
```
unravel    soft  nproc unlimited
```

Jobs per day	Cores	RAM	/usr/local/unravel	/srv/unravel
Less than 50,000	8	96 GB	8 GB free	500 GB free
50,000 to 100,000 to	8	128GB	8 GB free	500 GB free
Over 100,000	Contact Unravel Support

MySQL Server

Refer to Install > MySQL .

Permissions

You must already have an Azure account.
You must already have a resource group assigned to a region in order to group your policies, VMs, and storage blobs/lakes/drives.
A resource group is a container that holds related resources for an Azure solution. In Azure, you logically group related resources such as storage accounts, virtual networks, and virtual machines (VMs) to deploy, manage, and maintain them as a single entity.
You must have root privilege in order to perform some commands on the VM.

Network

Your virtual network and subnet(s) must be big enough to be shared by the Unravel VM and the target Databricks cluster(s).
You can use an existing virtual network or create a new one, but the virtual network must be in the same region and same subscription as the Azure Databricks workspace that you plan to create.
A CIDR range between /16 - /24 is required for the virtual network.
You must assign a public IP address to the Unravel Azure VM and open port 4043 for non-SSL and 4443 for unsecured SSL.
You must allow inbound SSH connections to the Unravel VM.
You must allow outbound Internet access and all traffic within the subnet (VSNET).

On the new node, open the following ports:

Port(s)	Direction	Description
3000	Both	Traffic to and from Unravel UI
4043	In	UDP and TCP ingest traffic from the entire cluster to Unravel Server(s)
22	In	Port for Secure Shell (SSH) communication and remote administration access.

Create Azure VM

Sign in to the Azure portal.
Select Virtual Machines > Add and enter the following information:
In the Basics tab (default) enter the following.
Project Details
- Subscription: Choose the applicable subscription.
- Resource group: Create a new group or choose an existing one.
Instance Details
- Virtual Machine Name:The Unravel server name.
- Region: Select the Azure region.
- Availability Options: Select No infrastructure is redundancy required.
- Image: Select the appropriate image. Both Centos-based 7.x+ and Red Hat Enterprise Linux 7.x+ are supported.
- Size: Click Change Size. In the modal select Memory optimized image with at least 128 GB memory and Premium Disk support, for example, E16s_v3 in East US 2)
Administrator account
- Authentication type: Select password or SSH Key.
- Username and Password: Enter your VM login information.
Inbound Port Rules
- Public inbound ports: Select Allow selected ports.
- Selected Inbound ports: Select both HTTPS and SSH.
Click Next: Disks >.
In the Disks tab enter the following information:
Disk Options
- OS disk type: Select Premium SSD.
Data Disk
- Click Create and attach a new disk.
  Note: This disk is formatted so don't choose Attach an existing disk.
  - Enter a Name.
  - Select Source typeNone (empty disk).
  - Set Size to at least 512 GiB.
  - Account type: Select premium SSD.
Click Next: Networking >
In the Networking tab enter the following information.:
- Virtual network: Create new or choose an existing one.
- Subnet: Create new or choose an existing one.
- Public IP: Create new or choose an existing one.
- Select Inbound ports: Select HTTPS and SSH.
Click Review + create.
Your deployment is now created.
Select Go to Resource > Networking > Inbound port rules > Add inbound port rule and include the following ports.
Rule Name
Destination
Destination IP Address
Destination Port Ranges
Unravel_3000
IP Addresses
NIC Private IP
3000
Unravel_4043
IP Addresses
NIC Private IP
4043
Click OK.

Rule Name	Destination	Destination IP Address	Destination Port Ranges
Unravel_3000	IP Addresses	NIC Private IP	3000
Unravel_4043	IP Addresses	NIC Private IP	4043

Install Database

You can either create an Azure database for MySQL or Install PostgreSQL.

Create Azure Database for MySQL

Select Create a Resource > Azure Database for MySQL. Click Create.
In the Basics tab (default) enter the following.
Project Details
- Subscription: Choose the applicable subscription.
- Resource group: Create a new group or choose an existing one.
Server Details
- Server name: Enter the MySQL server name.
- Data Source: Select None.
- Admin Username: Enter the MySQL admin name.
- Password/Confirm Password: Enter Admin password.
- Location: Select Azure region; it should be same region as the VM. (See Step 3 Create Azure VM, Instance Details.)
- Version: Select 5.7.
- Compute + storage: Click Configure Server. Select Memory Optimized, Compute Generation - Gen 5, 4 vCores, General Purpose Storage of 100GB with Auto-growth enabled. Click OK.
Click Review + Create.
Select Go to Resource > Connection Security > Add existing virtual network, enter the following information, and then select Enable:
- Subscription: Must be the same subscription as the VM. (See step 1 in Create VM.)
- Virtual Network: Must be the same virtual network as the VM. (See step 7 in Create VM.)
- Subnet: Create a new one if a default subnet doesn’t exist.
Select Go to Resource > Connection Security > SSL settings, and change the following:
- Enforce SSL connection: Select Disabled.
Click Save.

Select Server Parameters, change the following settings

Name	From	To
sort_buffer_size	524288	16777216 (32000000 and beyond or maximum allowed value)
query_cache_size	0	67108864 (64000000 and beyond or maximum allowed value)
max_connect_errors	100	2000000000 (2000000000 and beyond)
character_set_server	LATIN1	UTF8
innodb_file_per_table	OFF	ON
innodb_thread_concurrency	0	20
innodb_read_io_threads	4	16
innodb_io_capacity	200	4000
innodb_io_capacity_max	2000	4000

Click Save.

Installing PostgreSQL

Note

PostgreSQL is supported from Unravel version 4.6.1.6

You can either use the Unravel bundled PostgreSQL or external PostgreSQL version 12.

Installing Unravel bundled PostgreSQL on Databricks

Install Unravel. The Unravel bundled PostgreSQL is automatically installed.
Run the following commands in the given sequence to set up and connect the bundled PostgresSQL:
```
sudo /usr/local/unravel/bin/emdb_enable.sh
sudo /etc/init.d/unravel_all.sh restart.
```

Installing external PostgreSQL 12 on Databricks

Download PostgreSQL 12.

sudo yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

Run the following commands to install:

sudo yum install postgresql12-server
sudo /usr/pgsql-12/bin/postgresql-12-setup initdb
sudo systemctl start postgresql-12
sudo systemctl enable postgresql-12

Create Azure Databricks

Select Create a resource > Azure Databricks > Create. Go directly to step#3 if you already have workspaces.
Select Workspace name, Subscription, Resource group, Location, and Pricing tier.

Review VNET Peering options to connect Databricks with Unravel VM.

Databricks Workspace	Unravel VM	Region	VNET peering option
Deployed with VNET Injection	Deployed in the same VNET as Azure Workspace.	-	-
Deployed with VNET Injection	Deployed in different VNET from Azure Workspace.	Any Azure region	Create VNET Peering between the two VNETs
Deployed with Default VNET		Any Azure region	Create VNET Peering between the two VNETs

Install Unravel prerequisites on Azure VM

Install ntpd.

sudo su -
yum install ntp
ntpd -u ntp:ntp

Prepare the data disk with at least 500 GB that was added to Azure VM. Use fdisk -l to check any 500 GB disk without partition (usually /dev/sdc).

List disks and partitions.
```
fdisk -l
fdisk -l /dev/sdc
```
Format the disk /dev/sdc.
```
/usr/sbin/mkfs -t ext4 /dev/sdc
```

Mount the disk /dev/sdc on /srv.

mkdir -p /srv
DISKUUID=`/usr/sbin/blkid |grep ext4 |grep sdc | awk '{ print $2}' |sed -e 's/"//g'`
echo $DISKUUID
echo "${DISKUUID} /srv ext4 defaults 0 0" >> /etc/fstab
cat /etc/fstab
mount /dev/sdc /srv

Verify the disk space.
```
df -hT /srv
```

Set permissions for Unravel and symlink Unravel's directories to the /srv mount.

mkdir -p /srv/local/unravel
chmod -R 755 /srv/local
ln -s /srv/local/unravel /usr/local/unravel
chmod 755 /usr/local/unravel

Install MySQL client.
```
yum install mysql
```

Install DBFS CLI.

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install python-pip
sudo pip install databricks-cli

Note

Test connectivity using databricks-cli. In case there are any errors (Error: ValueError: Timeout value connect was Timeout) , then reinstall databricks-cli using python virtualenv as follows:

pip install databricks-cli
yum install python3
virtualenv -p /usr/bin/python3 mypy3
source mypy3/bin/activate
pip install databricks-cli

Install Unravel on Azure VM

Download the latest RPM for Cloud.

Install RPM.

sudo rpm -ivh latest-cloud-rpm 2> /tmp/rpm-install-log.txt

Configure Database

This topic provides information about configuring database:

Configure Unravel with MySQL

Using MySQL, create a database and user for Unravel. Enter MySQL admin login password when prompted:

mysql> CREATE DATABASE unravel_mysql_prod;
mysql> CREATE USER '<Unravel database user>'@'<Unravel host name>' IDENTIFIED BY '<Unravel database password>';
mysql> GRANT ALL PRIVILEGES ON unravel_mysql_prod.* TO '<Unravel database user>'@'<Unravel host name>';

Configure MySQL in /usr/local/unravel/etc/unravel.properties.

unravel.jdbc.username=username@hostname
unravel.jdbc.password=<Unravel database password>
unravel.jdbc.url=jdbc:mysql://<Unravel host name>:3306/unravel_mysql_prod
unravel.jdbc.url.params=useSSL=true&requireSSL=false

Install MySQL JDBC connector driver in Unravel classpath.

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz -O /tmp/mysql-connector-java-5.1.47.tar.gz

tar xvzf /tmp/mysql-connector-java-5.1.47.tar.gz
sudo mkdir -p /usr/local/unravel/share/java

sudo cp /tmp/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47.jar /usr/local/unravel/share/java
sudo chown unravel:unravel /usr/local/unravel/share/java/mysql-connector-java-5.1.47.jar

Create database and tables for Unravel.

/usr/local/unravel/dbin/db_schema_upgrade.sh

Configure Unravel with PostgreSQL

Note

PostgreSQL is supported from Unravel version 4.6.1.6

Run psql and create a database and user for Unravel. Enter psql admin login password when prompted:

sudo -u postgres psql

create database unravel;
create user unravel with encrypted password 'unraveldata';
grant all privileges on database unravel to unravel;
ALTER USER unravel WITH SUPERUSER;
alter user unravel with createdb createrole inherit replication bypassrls;
grant connect on database unravel to unravel;
grant usage on schema public to unravel;
grant all privileges on all tables in schema public to unravel;
grant all privileges on all sequences in schema public to unravel;
alter default privileges in schema public grant all privileges on tables to unravel;
alter default privileges in schema public grant all privileges on sequences to unravel;
grant pg_read_server_files to unravel;
grant pg_write_server_files to unravel;
grant pg_execute_server_program to unravel;

Allow user/server to connect to the database.
- Option 1: If PostgreSQL is installed on the same server as Unravel, add the following line in /var/lib/pgsql/12/data/pg_hba.conf at the first line of IPv4 local connections:
```
host    all             unravel         127.0.0.1/32            md5
```
- Option 2: If PostgreSQL is installed on a different server, do the following:
  1. Add the following line in /var/lib/pgsql/12/data/pg_hba.conf:
    host all unravel <Unravel Server Internal IP Address>/32 md5
  2. Update /var/lib/pgsql/12/data/postgresql.conf and ensure listen_addresses = '*' is set to allow PostgreSQL to listen to all the traffic.

Add following properties in /usr/local/unravel/etc/unravel.properties:

unravel.jdbc.username=unravel
unravel.jdbc.password=unraveldata
unravel.jdbc.url=jdbc:postgresql://127.0.0.1:5432/unravel

Restart PostgreSQL.

sudo systemctl restart postgresql-12.service

Install PostgreSQL JDBC connector driver in Unravel classpath.

sudo mkdir -p /usr/local/unravel/share/java
wget https://jdbc.postgresql.org/download/postgresql-42.2.18.jar -O /usr/local/unravel/share/java/postgresql-42.2.18.jar
sudo chown unravel:unravel /usr/local/unravel/share/java/postgresql-42.2.18.jar

Test and update db schema.

/usr/local/unravel/install_bin/db_access.sh*
/usr/local/unravel/dbin/db_schema_upgrade.sh

Configure Unravel with Workspace

Go to Workspace> Admin Console> Access Control and enable Personal Access Tokens. See Enable token-based authentication.
Go to Workspace> User Settings> Access Tokens and click Generate New Token. See Authenticate using Databricks personal access tokens. Choose the lifetime of the token as indefinite.

Install Unravel agents on the Workspace and update Unravel config with the Workspace details. See Running the Databricks_setup.sh script

Note

Run the following commands only if the Databricks command-line tool is installed using Python virtual environment:

sudo bash
virtualenv -p /usr/bin/python3 mypy3
source mypy3/bin/activate

usr/local/unravel/install_bin/databricks_setup.sh --add-workspace -i <Workspace id> -n <Workspace name> -t <Workspace token> -r https://<Workspace location>.azuredatabricks.net -p <Workspace_tier> -u <Unravel DNS or IP address>:4043

Restart Unravel

Restart all Unravel services
```
service unravel_all.sh restart
```
Using a supported web browser, (see Unravel's Databricks compatibility matrix), navigate to http://unravel-host:3000 and log in with username admin with password unraveldata.

Configure Databricks with Unravel

In your Databricks workspace, update the following tabs under Advanced Options for every cluster (Automated/Interactive) that you want to monitor:

Spark

Copy the following snippet to Spark > Spark Conf. Replace <Unravel DNS or IP Address>. This snippet is also generated by the Databricks setup script on Unravel.

Note

For spark-submit jobs, click Configure spark-submit and copy the following snippet in the Set Parameters > Parameters text box as spark-submit parameters. Replace <Unravel DNS or IP Address>.

"--conf", "spark.eventLog.enabled=true",
"--conf", "spark.eventLog.dir=dbfs:/databricks/unravel/eventLogs/",
"--conf", "spark.unravel.shutdown.delay.ms=300",
"--conf", "spark.unravel.server.hostport=<Unravel DNS or IP Address>:4043",
"--conf", "spark.executor.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3",
"--conf", "spark.driver.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3"

spark.eventLog.enabled true
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
spark.unravel.server.hostport <Unravel DNS or IP Address>:4043
spark.unravel.shutdown.delay.ms 300
spark.executor.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3
spark.driver.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3

Logging

Select DBFS as Destination, and copy the following as Cluster Log Path.

dbfs:/cluster-logs/

Init Scripts

In the Init Scripts tab, set Destination to DBFS. Copy the following as the Init script path and click Add.

dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh

Installing Databricks sensors using Setup script

Description

This topic explains how to configure Unravel for Databricks using the /usr/local/unravel/install_bin/databricks_setup.sh script on the Unravel server. This script deploys Unravel agent binaries and prints Databricks cluster configuration that is required for Unravel monitoring. Repeat this script for each workspace you want to initialize.

Important

Run this script as the same username that you used to install the Unravel server.

Syntax

usage:  
databricks_setup.sh --add-workspace -i <workspace-id> -n <workspace-name> -r <workspace-instance> -t <workspace-token> -u <unravel_server:port> [options]  databricks_setup.sh --print-spark-conf -u <unravel_server:port> [options]  databricks_setup.sh --help

Options:

Option	Description
`--add-workspace \| -a`	Sets up or updates a Databricks workspace for monitoring by Unravel. Valid values:
	`-i`	Id of the workspace to be configured.
	`-n`	Workspace name.
	`-r`	Workspace instance. Must start with `https://`. For example, `https://eastus.databricks.com`
	`-t`	Workspace access token.
	`-u`	Unravel LR Endpoint. (e.g: 10.0.0.4:4043)
	`-p`	(Optional) Workspace tier. Accepted values are - premium, standard. The default is premium.
	`-e`	Enables/disables SSL for Databricks sensor and agent. Valid values: `true`, `false`. Default: `false`.
	`-c`	(Optional) Enables/disables SSL connections to Unravel endpoints without certificates. This option is only in effect if `-e` is set to `true`. Valid values: `true`, `false`. Default: `false`.
	`-v`	(Optional) Spark version to be used. Default: `2.3`.
	`-d`	(Optional) Enables debug logs for Unravel Databricks sensor installation. Default: `false`.
	`-m`	(Optional) Specifies the frequency in seconds in which to poll cluster metrics. Default: `30`.
`--print-spark-conf \| -p`	Print the minimal Spark configuration required to monitor cluster using Unravel.
	`-u`	Unravel server URL. For example, `0.0.0.1:4043`
	`-e`	Enables/disables SSL for Databricks sensor and agent. Valid values: `true`, `false`. Default: `false`.
	`-c`	(Optional) Enables/disables SSL connections to Unravel endpoints without certificates. This option is only in effect if `-e` is set to `true`. Valid values: `true`, `false`. Default: `false`.
	`-v`	(Optional) Spark version to be used. Default: `2.3`.
	`--help \| -h`	Prints the usage of this script.

Note

If you generate new tokens, re-run this script to update Unravel Server.

Examples

To add/edit a workspace:

/usr/local/unravel/install_bin/databricks_setup.sh --add-workspace -i 1234567890 -n DemoWorkspace -t ***** -r https://eastus.azuredatabricks.net -u 10.1.2.3:4043 -p premium
Deleting directory - dbfs:/databricks/unravel/unravel-agent-pack-bin
Deleted directory successfully
Deleting directory - dbfs:/databricks/unravel/unravel-db-sensor-archive
Deleted directory successfully
Creating directory - dbfs:/databricks/unravel/logs
Created directory successfully
Creating directory - dbfs:/databricks/unravel/eventLogs
Created directory successfully
Copying /tmp/unravel_db.properties to dbfs:/databricks/unravel/unravel-db-sensor-archive/etc/unravel_db.properties
Copied file successfully
Copying /tmp/agent-pack to dbfs:/databricks/unravel/unravel-agent-pack-bin
Copied file successfully
Copying /tmp/sensor_pack to dbfs:/databricks/unravel/unravel-db-sensor-archive
Copied file successfully

-----------------------------------
Cluster Spark Configuration
-----------------------------------
spark.executor.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3
spark.eventLog.enabled true
spark.unravel.server.hostport 10.1.2.3:4043
spark.driver.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
spark.unravel.shutdown.delay.ms 300

-----------------------------------
Spark Submit Parameters
-----------------------------------
"--conf", "spark.executor.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3",
"--conf", "spark.eventLog.enabled=true",
"--conf", "spark.unravel.server.hostport=10.1.2.3:4043",
"--conf", "spark.driver.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3",
"--conf", "spark.eventLog.dir=dbfs:/databricks/unravel/event

To print the configuration:

/usr/local/unravel/install_bin/databricks_setup.sh -p -u 10.1.2.3:4043

-----------------------------------
Cluster Spark Configuration
-----------------------------------
spark.executor.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3
spark.eventLog.enabled true
spark.unravel.server.hostport 10.1.2.3:4043
spark.driver.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
spark.unravel.shutdown.delay.ms 300

-----------------------------------
Spark Submit Parameters
-----------------------------------
"--conf", "spark.executor.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3",
"--conf", "spark.eventLog.enabled=true",
"--conf", "spark.unravel.server.hostport=10.1.2.3:4043",
"--conf", "spark.driver.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3",
"--conf", "spark.eventLog.dir=dbfs:/databricks/unravel/eventLogs/",
"--conf", "spark.unravel.shutdown.delay.ms=300"

-----------------------------------
Databricks Cluster Init Script
-----------------------------------
dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh

Uninstalling Unravel Server and Sensors on Azure Databricks

Delete the Unravel installation location on DBFS for each workspace where Unravel is deployed using DBFS CLI:

dbfs rm -r dbfs:/databricks/unravel

For a list of Databricks workspaces configured, see /usr/local/unravel/etc/unravel.properties.

In this section:

Home

Microsoft Azure Databricks (Manual)

Prerequisites

Important

Create Azure VM

Install Database

Note

Create Azure Databricks

Install Unravel prerequisites on Azure VM

Note

Install Unravel on Azure VM

Configure Database

Note

Configure Unravel with Workspace

Note

Restart Unravel

Configure Databricks with Unravel

Note

Installing Databricks sensors using Setup script

Important

Note

Uninstalling Unravel Server and Sensors on Azure Databricks

Search results