Compute (Databricks)
Note
This tab is visible only for Databricks clusters.
The Compute tab displays the list of Databricks clusters tracked by Unravel. Each cluster has a separate tab that contains information about the cluster's metadata, KPIs, configurations, trends, and Unravel's analysis. In addition, you can filter the clusters based on the status, duration, user, workspace, cluster type, and tags. The Compute tab consists of the following tabs:
All: Lists all the clusters and their details.
Finished: Lists the clusters where the jobs have completed the runs.
Running: Lists the clusters where the jobs are in the running status.
Inefficient: Lists the inefficient clusters.
The cluster list displays the following columns:
Columns | Description |
|---|---|
Status | Status of the cluster. |
Cluster Name | Name of the Databricks cluster. Click the link of the cluster name to view the details of the cluster session. |
Cluster Type | Type of the Databricks cluster.
|
User | Name of the user running the jobs on the cluster. |
Insights | Displays the ( |
Workspace | Name of the Databricks workspace. |
Start time | The start time and day of the cluster. |
Duration | Total time taken by the cluster to process the jobs. |
Cost | An estimated cost incurred in running the apps on the cluster. This cost is calculated based on VM and DBU prices. |
Actions | Click to access the Application Details page, where more details of a specific job run on the cluster are displayed. |
Filtering clusters
You can filter clusters by:
Selecting a date and time range of the running clusters.
Using ID or keyword.
Using cluster name
The left sidebar lets you filter the clusters by:
Status: Status of the cluster, success, failed, or killed.
Duration: You can use the slider to set the time range.
Cost: You can use the slider to filter database clusters by cost (based on the dollar). If the maximum cost is less than one dollar ($1), you cannot apply the Cost filter.
User: Name of user, who runs the jobs on the cluster. Click in the text box for a list of valid values.
Workspace: Databricks workspaces. Click in the text box for a list of valid values. Click the value to select it.
Cluster Type: The type of cluster; Automated or Interactive. Click in the text box for a list of valid values. Click the value to select it.
Tags: List of created tags. Click the checkbox next to the tag and then click in the text box for the valid values.
Events: This filter is only available in the Inefficient tab. You can filter the inefficient apps based on the associated events.
Viewing a cluster session
On the Unravel UI, navigate to Compute > All/Finished/Running/Inefficient tab.
Click the link in the Cluster Name column for the cluster you want to view. The cluster session page is displayed.
From the Cluster session page, you can get a quick reference of the following:
Name of the cluster
Type of the cluster
The cost incurred for the cluster
Status of the job run on the cluster
Start time of the job on the cluster
End time of the job on the cluster
Workspaces on the cluster
Platform of the cluster
The following tabs are shown for the cluster session.
From this tab, you can view the cluster insights. Unravel's analysis results show where the efficiencies of your app can be improved at the cluster level.
The number of efficiencies found is noted in the tab. When Efficiency leads to a suggested change, those suggestions are listed as Recommendations.
Efficiency: Suggest areas to examine to improve efficiency. For example, inefficient usage of resources. Expand the Efficiency of reading the in-depth analysis. While some are only configuration property suggestions, others point to areas for examination. For example, inefficient storage, problems with a query, and so on.
You also view the insights when a high possibility of cost savings may be available.
Viewing insights:
The insights are automatically displayed for a Job when any recommendations are available in the Analysis tab of the Compute > App Summary page to start the analysis.
Utilize these insights to achieve cost-effective resource allocation. You can click Update Insights to generate updated insights at any time.
Note
Update Insights feature is applicable only for cost-saving insights for job or automated clusters.
Recommendations: Suggestions to update a property or configuration parameter. Click to view the current and recommended value.
Bottlenecks: Suggestions to make your apps run faster.
Whitelisting Insights
You can configure the system to display insights for only specific instances by using the whitelisting property. This feature is available for the for the node downsizing and node rightsizing insights.
To whitelist the insights, perform the following steps:
Stop Unravel.
<unravel installation directory>/unravel/manager stop
Add the instances to be whitelisted through the property.
<Unravel installation directory>/unravel/manager config databricks set com.unraveldata.databricks.instances.whitelist=\ Standard_E4as_v4, Standard_E8as_v4, Standard_E16as_v4, Standard_E32as_v4, \ Standard_E64as_v4, Standard_E4ds_v5, Standard_E8ds_v5, Standard_E16ds_v5, Standard_E32ds_v5, \ Standard_E64ds_v5, Standard_D4a_v4, Standard_D8a_v4, Standard_D16a_v4, Standard_D32a_v4, \ Standard_D64a_v4, Standard_D4ds_v4, Standard_D8ds_v4, Standard_D16ds_v4, Standard_D32ds_v4, \ Standard_D64ds_v4, Standard_D4ds_v5, Standard_D8ds_v5, Standard_D16ds_v5, Standard_D32ds_v5, Standard_D64ds_v5
The above command will allow insights for the mentioned instances. You can remove any instances that are not required.
Apply the changes. If you have not stopped using Unravel services, you will be prompted to do so. Run manager stop command to stop Unravel services.
<Unravel installation directory>/unravel/manager config apply
Start Unravel.
<Unravel installation directory>/unravel/manager start
Execute the following scripts:
For updates to node rightsizing, execute insights_upgrade_v1.sh
For updates to node downsizing, execute insights_downSizing_upgrade_v1.sh
The following example illustrates the whitelisting functionality. Please refer to the images below for a visual representation.
Node Rightsizing
The following image represents the default settings with no node rightsizing whitelisting properties applied.
Example whitelisting properties applied. You must execute the insights_upgrade_v1.sh script after applying the properties.
The following image is an illustration of the screen with whitelist instance type Standard_L4s applied:
Node Downsizing
The following image represents the default settings with no node downsizing whitelisting properties applied.
Example whitelisting properties applied. You must execute the insights_downSizing_upgrade_v1.sh script after applying the properties.
The following image is an illustration of the screen with whitelist instance type Standard_D4s_v3 applied:
Cluster Detail
The Cluster detail tab shows cluster details such as the Cluster Mode, Databricks Runtime Version, Autopilot Options, Worker Type, Driver Type, and so on.
Trends
From the Trends tab, based on the type of the cluster, Interactive, Automated, or Automated Light, you can view job trends in the cluster and the trends of the resources used for all the jobs.
Interactive cluster
Automated cluster
Automated Light cluster
Configurations
From this tab, you can view all the configurations set for the chosen cluster. In the upper-right corner, click the Copy icon to copy the cluster configuration, and click the Download icon to download the configuration in a JSON file format.
