v4.7.7.0 Release notes

Software version

Release date: 09/December/2022

See 4.7.7.0 for download information.

Software upgrade support

The following upgrade paths are supported:

v4.7.0.x, v4.7.1.x, v4.7.2.x, v4.7.3.x, v4.7.4.x, v4.7.5.x, v4.7.6.x → v4.7.7.0
v4.6.1.9 → v4.7.7.0
v4.6.1.8 or earlier → v4.6.1.9 → v4.7.7.0

For instructions to upgrade to Unravel v4.6.1.9, see Upgrading Unravel server.

For instructions to upgrade to Unravel v4.7.7.x, see Upgrading Unravel.

For fresh installations, see Installing Unravel.

Certified platforms

The following platforms are tested and certified in this release:

Google Cloud Platform (BigQuery, Dataproc)

Review your platform's compatibility matrix before you install Unravel.Compatibility Matrix

Updates to Unravel's configuration properties

See 4.7.x - Updates to Unravel properties.

Updates to upgrading Unravel to v4.7.7.0

An existing license for any previous version (before 4.7.7.x) does not work with the newer version of Unravel. Therefore, before upgrading Unravel you must obtain a license file from Unravel Customer Support. For information about setting the license, see Upgrading Unravel from version 4.7.x to 4.7.7.x section in Upgrading Unravel.
In this release, Unravel supports the pull model, along with the push model, to get data from BigQuery projects. After the upgrade, the push model, which was used in previous releases, will be continued. However, the pull model can be implemented only with a fresh install of Unravel.
When you have enabled both Bigquery and Dataproc before upgrading to Unravel v4.7.7.0, then after the upgrade, you must enable Hadoop daemons (hive_worker, kafka_monitor, yarn_jc_worker, oozie_sensor) manually. see Upgrading Unravel from version 4.7.x to 4.7.7.x section in Upgrading Unravel for complete instructions.
For best performance, enable message ordering in GCP Cloud Pub/Sub. The Message ordering setting lets you receive the messages in the order they are published. Refer to Ordering messages for more details. After you upgrade, you cannot directly update the Message ordering setting.
To enable the Message ordering setting after an upgrade, do the following:
1. From the GCP console, delete the configured subscription for the respective project.
2. Create a new subscription with the Message ordering setting on.
  Note
  The Message ordering setting can be configured only once when configuring the Pub/Sub service for the first time.

New features

License

Replaced the license key with the license file for expandability and human readability. A license file contains product licensing information, the license validity, and the expiration date.

Unravel no longer supports an automatic trial period. You must obtain a license file from Unravel Customer Support.

For information about the Licensing documentation, see the following topics:

Topics	Guide
Setting Unravel license Verifying Unravel license	Configuration Guide
License: new property License FAQ	Reference Guide
Licensing error messages and troubleshooting	Troubleshooting Guide
Install Unravel on GCP Dataproc using Interactive precheck Install Unravel on GCP Dataproc manually Installing Unravel for GCP BigQuery Manually installing Unravel for GCP BigQuery Use the Interactive Precheck utility Upgrading Unravel for Cloud platforms (v4.7.x to v4.7.7.0)	Installation Guide

AutoActions for BigQuery
Unravel now supports AutoActions for BigQuery. An updated user interface is provided for AutoActions in GCP with improved usability.
For more information, refer to AutoActions > AutoActions (GCP - BigQuery, Dataproc) topic in User Guide.
BigQuery pull model
Unravel now supports getting data about BigQuery from GCP with the pull model. Previously, only the push model was supported. The pull model is available only for fresh installations.
Refer to the Installation Guide > Google Cloud platform - BigQuery installation section for more details.
Secure connection with Cloud SQL Auth proxy to external MySQL and PostgreSQL database
For the Google Cloud Platform (GCP), you can enable a secure connection between Unravel server and an external MySQL or PostgreSQL database. This can be done using the Cloud SQL Auth proxy, which is a Cloud SQL connector that provides secure access to your instances without needing Authorized networks for configuring SSL. Refer to Enabling secure connection from Unravel GCP to external MySQL database with Cloud SQL Auth proxy section in the Configuration Guide.

Improvements and enhancements

Installation
- Added support for the .xz archive format when configuring Unravel managed database with the --extra setup option. (INSTALL-2773)
- When installing for BigQuery, the Interactive Precheck utility now prompts for the push and pull options. (INSTALL-2817)
BigQuery enhancements
- Enhancements to the BigQuery Application Details page:
  - Added support for different job types displayed on the BigQuery details page. The supported job types are QUERY, LOAD, COPY, and EXTRACT. The new Type KPI shows the type of job run in BigQuery. (UIX-5003)
  - On the Tables tab, the tables are now categorized into Input and Output columns. (UIX-4939)
  - Added a new Child Jobs tab to display child jobs of the selected parent job. You can also view parent details in the child app. (ASP-1552)
  - Added a Close icon next to the selected stage. Click the icon to return to the App Summary page. (UIX-5045)
  For information, see the BigQuery Details page section in User Guide.
- On-demand analysis pricing is configurable for different regions
  - You can now configure the on-demand analysis pricing for different regions, including custom prices, using the com.unraveldata.bigquery.ondemand.analysis.pricing.per.tb.usd.{region} property.
    For more information about this property, see BigQuery properties in Reference Guide.
    To configure on-demand pricing, see the Configuring on-demand pricing to process queries section in Configuration Guide.
Other enhancements
- Upgraded the Node.js Javascript to the 16.18.0 version. (UIX-4461)

Unsupported

Data

In GCP - BigQuery, for the Data page, a count of more than 100 projects is not supported.

BigQuery pricing

For BigQuery pricing, Unravel only supports On-demand analysis pricing. Flat-rate analysis pricing and Storage pricing (Active and Long Term storage) is not supported.

Bug fixes

BigQuery
- Incorrect column count for SelectStartEvent. (BIGQ-79)
Insights
- The Elasticsearch cache issue is observed on the Insights Overview dashboard. (CDI-660)
- After clicking the READ operator link on the Analysis tab, no action is triggered. (UIX-5067)

Known issues

Applications

Workflow/Job page displays empty for Analysis, Resources, Daggraph, and Errors tabs. (DT-1093)
Event logs and YARN logs are not loaded for some applications in Google Dataproc clusters. (ASP-1372)

AutoActions

When multiple AutoActions policies are created with the Overlapping ruleset and scopes, only one of the AutoAction policies is triggered. (AA-498)
AutoActions stop responding due to an invalid or unsupported HTTP URL or webhook. (AA-575)

App store

App store tasks fail to start with SSL. (APP-614)
Workaround
To resolve this issue, do the following:
1. Stop Unravel.
```
<Unravel installation directory>/unravel/manager stop
```
2. Use an editor to open <Installation_directory>/unravel/data/conf/unravel.yaml file.
3. In the unravel.yaml file, under the database > advanced > python_flags block, enter the path to the trusted certificates. For example, if Unravel is installed at /opt/unravel, you must edit the unravel.yaml file as follows:
```
unravel:
...snip...
  database:
...snip...
    advanced:
      python_flags:
        ssl_ca: /opt/unravel/data/certificates/trusted_certs.pem
```
4. Use the manager utility to upload the certificates.
```
<Unravel installation directory>/manager config tls trust add --pem /path/to/certificate
```
  For example: /opt/unravel/manager config tls trust add --pem /path/to/certificate
5. Enable the Truststore.
```
<Unravel installation directory>/manager config tls trust enable
```
6. Apply the changes and restart Unravel.
```
<Unravel installation directory>/unravel/manager config apply --restart
```

BigQuery

On the Application details page, the original query link is missing for some cached queries due to the parallel processing of original and cached queries. (BIGQ-61)
Issue: Sometimes when you process a large number of BigQuery projects with the manager config bigquery integrate command, you may see the following error:
Provider produced inconsistent result after apply
Workaround: Wait for few minutes and re-run the command. (INSTALL-2860, INSTALL-2934)

Data page

If tables are created with the same name and are accessed, deleted, and re-created, and if those tables are accessed again, then their query and apps count does not match.(DATAPAGE-502)
For Hive metastore 3.1.0 or earlier versions, the create time of partitions is not captured if a partition is created dynamically. Therefore, in Unravel, the “Last Day” KPIs for the partition section are not shown. (DATAPAGE-473)

Databricks

Errors and Logs data are missing in specific scenarios for a failed Spark application, such as an application failing with OutOfMemoryError. (ASP-1624)
On the Chargeback page, when you group by clusters, Unravel has a limitation of only grouping a maximum of 1000 clusters. (SUPPORT-1570)
On the Cost > Trends and Cost > Chargeback pages, the tooltip for the Last <number of> days field includes more days than the displayed days. (UIX-5042)
When the Interactive cluster is restarted, the cluster count is increased on the Databricks Cost > Trends page. (DT-1275)
The duplicate job runs (with the same run IDs) are generated on the Job Runs page. (DT-1190)
After navigating from Trends and Chargeback pages with Tag filters, the No data available message is displayed on the Compute page. (DT-1094)
Inconsistent data is displayed for the cluster Duration and Start Time on the Compute page. (ASP-1636)
The DriverOOME and ExecutorOOME events are not generated for the Databricks notebook task. (DT-533)
When a job fails to submit a Spark application, the failed DataBricks job is missing from the Unravel UI. (ASP-1427)
In Databricks, when a job in a workflow fails and a new job is launched instead of a new attempt, the new job cannot be part of the same workflow. (PG-269)
In the Databricks view, the application is shown in a running state, even though the corresponding Spark application is marked as finished. (ASP-1436)

Dataproc

Google Cloud Dataproc: Executor Logs are not loaded for Spark applications. (ASP-1371)

EMR

The workflow of multiple transient clusters (EMR) is not supported. (ASP-1424, EMR-460)
Unable to run Spark applications on all the master nodes after Unravel bootstrap for high availability clusters. (EMR-49)

Email

Unravel node fails to send email notifications. (INSTALL-1694)

Installation

Issue: You can encounter a NoIndexFound exception for fresh installations of Unravel on GCP-BigQuery. (BIGQ-104)
Workaround: Run the following CURL command on the Unravel node after the installation.
```
curl -XPUT http://localhost:4171/app-19700101_07
```
An exception occurs when installing Unravel version 4.7.6.0 with the Azure MySQL database (SSL Enabled). (INSTALL-2799)
During precheck and healthcheck, the Hadoop check fails for the MapR cluster. You can ignore the messages. (INSTALL-2603)

Insights Overview

The Insights Overview tab uses UTC as the timezone, while other pages use local time. Hence, the date and time shown on the Insights Overview tab and the other pages after redirection can differ. (UIX-4176)

Kerberos

Kerberos can only be disabled manually from the unravel.yamlfile.
```
 kerberos:
      enabled: False
```

Migration

WorkloadFit report
- A large number of tags can cause the Workload Fit report to fail. (PG-265, CUSTOMER-2084)
- WorkloadFit report > Heatmap: The job count has data, but Vcore and memory are empty. (MIG-262)

Reports

Cluster discovery
- The On-prem Cluster Identity might show an incorrect Spark version on CDH. The report may incorrectly show Spark 1 when Spark 2 is installed on the CDH cluster. (REPORT-1702)
TopX report
- The TopX report email links to the Unravel TopX report instead of showing the report content in the email as in the old reports.
Queue analysis:
- The log file name (unravel_us_1.log) displayed in the error message is incorrect. The correct name of the log file is unravel_sensor.log. (REPORT-1663)
Cloud Mapping Per Host report scheduled in v4.6.1.x does not work in v4.7.1.0. Users must organize a new report. (REPORT-1886)
When using PostgreSQL, the % sign is duplicated and displayed in the Workload Fit report > Map to single cluster tab. (MIG-42)

Spark

A blank page is displayed on the Databricks Run Details page for Spark structured streaming applications. (ASP-1629, UIX-5124)
If the Spark job is not running for Databricks, the values for the Duration and End time fields are not updated on the Databricks Run Details page. (ASP-1616)
You can see a lag for SQL Streaming applications. (PLATFORM-2764)

Spark insights

For PySpark applications, the processCPUTime and processCPULoad values are not captured properly. (ASP-626)

Tez

SQL events generator generates SQL Like clause event if the query contains a like pattern even in the literals. (TEZLLAP-349)

Upgrade

After upgrading from v4.7.1.1 to v4.7.5.0, the Hive jobs running with the Tez application as an execution engine are not linked. (EMR-406)
After upgrading to v4.7.1.0, Notebooks do not work. You can configure them separately. (REPORT-1895)
After upgrading from v4.6.x to v4.7.1.0, the Tez application details page does not initially show DAG data. The DAG data is visible only after you refresh the page. (ASP-1126)

UI

You can access the new user interface (UI) only from Chrome.
In the App summary page for Impala, the Query> Operator view is visible after scrolling down. (UIX-3536).
Issue: When you return from the application details > SQL tab> Stage page to the application details > Attempt page, the Duration, Data I/O, and Jobs Count fields are not displayed. (UIX-5048)
Workaround: You can reload or refresh the application details > Attempt page to restore all the KPIs.

Workflow

Jobs are falsely labeled as a Tez App for Oozie Sqoop and Shell actions. (PLATFORM-2403)

Support

For support issues, contact Unravel Support.

In this section:

Home