v4.7.4.3 Release notes
Software version
Release date: 08/July/2022
See 4.7.4.3 for download information.
Software upgrade support
Fresh installations are supported along with the following upgrade paths:
v4.7.0.x, v4.7.1.x, v4.7.2.x, v4.7.3.x, v4.7.4.0, v4.7.4.1, v4.7.4.2 → v4.7.4.3
v4.6.1.9 → v4.7.4.3
v4.6.1.8 or earlier → v4.6.1.9 → v4.7.4.3
For instructions to upgrade to Unravel 4.6.1.9 version, see Upgrading Unravel server.
For instructions to upgrade to Unravel 4.7.4.x version, see Upgrading Unravel.
For fresh installations, see Installing Unravel.
Sensor upgrade
A sensor upgrade is recommended.
Sensor upgrade is mandatory for upgrading from all versions before v4.7.4.0 to v4.7.4.3.
See Upgrading Sensors.
Certified platforms
The following platforms are tested and certified in this release for new installations and upgrade to Unravel version 4.7.4.3
Cloudera Distribution of Apache Hadoop (CDH)
Cloudera Data Platform (CDP)
Hortonworks Data Platform (HDP)
Databricks (Azure)
Review your platform's compatibility matrix before you install Unravel.
Updates to Unravel's configuration properties
See 4.7.x - Updates to Unravel properties.
New features
None
Improvements and enhancements
Log receiver
You can now specify the retention period, in hours, for messages stored in the DocStorage by using the logreceiver.msg_retention.hours configuration property. (CDI-562)
Unsupported
Unravel does not support Billing for on-prem platforms.
On the Data page, File Reports, Small File reports, and file size information are not supported for MapR, EMR, and Dataproc clusters.
Impala jobs are not supported on the HDP platform.
Monitoring the expiration of the SSL Certificates and Kerberos principals in Unravel multi-cluster deployments.
The following features are not supported for MapR:
Impala applications
Kerberos
The following features are supported on the Data page:
Forecasting
Small Files
File Reports
The following reports are not supported on MapR:
File Reports
Small Files Report
Capacity Forecasting
Migration Planning
The Tuning report is supported only for MR jobs.
Migration Planning
AutoAction is not supported for Impala applications
Migration
Billing
Insights Overview
Migration planning is not supported for the following regions for Azure Data Lake:
Germany Central (Sovereign)
Germany Northeast (Sovereign)
Forecasting and Migration: In a multi-cluster environment, you can configure only a single cluster at a time. Hence, reports are generated only for that single cluster.
Migration planning is not supported for MapR.
Unravel does not support multi-cluster management of combined on-prem and cloud clusters.
Unravel does not support apps belonging to the same pipeline in a multi-cluster environment but is sourced from different clusters. A pipeline can only contain apps that belong to the same cluster.
All the reports, except for the TopX report, are not supported on Databricks and EMR.
In Jobs > Sessions, the feature of applying recommendations and then running the newly configured app is not supported.
Pig and Cascading applications are not supported.
Bug fixes
Document storage
The following issues are fixed with Document Storage (CDI-562):
A
NullPointerException
is shown when the Document is missing.The
read-idle-timeout
property is not propagated correctly to DocStorageClient configuration.
Hive query
The Hive query application details page shows a blank space in the header. (UIX-4610)
Installation
Support Databricks source files generator for older versions (4.7.4.0 and earlier). (CDI-553)
The manager utility must use the bundled PostgreSQL even in case of an external database connection. (INSTALL-2651)
Log receiver
DocStorage messages are deleted by default with 1-hour retention and are not available. (CDI-562)
Monitoring service
The
nashorn
library dependency and Java script rules are removed from the monitoring service. (CDI-568)
Spark
Disable processing of stage and job data for streaming applications based on a configuration property. (ASP-1507)
The API call for some of the metrics returns data as null even though the metrics details are displayed on the Resources tab of the App Summary page. (UIX-4601)
Missing job runs on the Unravel UI. (SUPPORT-1346)
Upgrade
After upgrading Unravel version to 4.7.4.2, Unravel services do not start because the Kerberos package was not included in the JVM export parameters. (CDI-567)
After upgrading Unravel from version 4.7.4.2 to 4.7.4.3, the custom JDK version 1.8.0_112 reverted to the bundled JDK version 17.0.3. (INSTALL-2660)
After upgrading Unravel from version 4.7.4.2 to 4.7.4.3, the Spark jobs are not displayed on the Jobs tab. (DT-1170)
UI
Data is not loaded on the Daemons page. (UIX-4602)
Workflow/Job page displays empty for Analysis, Resources, Daggraph, and Errors tabs. (DT-1093)
Event logs and YARN logs are not loaded for some applications in Google Dataproc clusters. (PG-170)
After upgrading Unravel from version 4.7.2.3 to 4.7.4.3 with Maria DB, the App Store daemon fails to start. (REPORT-2048)
Workaround: Drop the
kombu_queue
database table for the schema mismatch issue and start the On-demand and App Store services.
Incorrect data is displayed in the Number of Queries KPI/Trend graph on the Overview page. (DATAPAGE-502)
Create time of partitions is not captured in hive metastore if the partition is created dynamically. This limits Unravel to show Last Day KPIs for the partition section.
Wrong data displayed for Number of Partitions Created KPI/trend graph under Partitions KPIs - Last Day section in theData page. (DATAPAGE-473)
When a job fails to submit a spark application, the failed DataBricks job is missing from Unravel UI. (ASP-1427)
In Databricks, when a job in a workflow fails and a new job is launched instead, as a new attempt, the new job cannot be part of the same workflow. (PG-269)
DataBricks jobs are being missed intermittently in Unravel. (PG-232)
In the Databricks view, the application is shown in a running state, even though the corresponding Spark application is marked as finished. (ASP-1436)
Google Cloud Datapro: Executor Logs are not loaded for spark applications. (PG-229)
Exception: Problem when retrieving bootstrap actions for cluster is seen in the aws_worker daemon logs.
Workaround: While creating an AWS account for EMR Chargeback/Insights overview feature, you must include an additional entry in the
Policy JSON
file for"elasticmapreduce:ListBootstrapActions"
, as follows:{ “Version”: “2012-10-17", “Statement”: [ { “Effect”: “Allow”, “Action”: [ “pricing:GetProducts”, “elasticmapreduce:ListClusters”, “elasticmapreduce:DescribeCluster”, “elasticmapreduce:ListInstanceFleets”, “elasticmapreduce:ListInstanceGroups”, “elasticmapreduce:ListBootstrapActions“, “elasticmapreduce:ListInstances”, “ec2:DescribeSpotPriceHistory” ], “Resource”: “*” } ] }
Even though the AWS account was already created without this entry (
elasticmapreduce:ListBootstrapActions
), you can always include this policy later.The workflow of multiple transient clusters (EMR) is not supported. (ASP-1424)
Unravel node fails to send email notifications. (INSTALL-1694)
During precheck and healthcheck, the Hadoop check fails for the MapR cluster. You can ignore the messages. (INSTALL-2603)
When you upgrade from Unravel version 4.6.1.9 to 4.7.4.2 and the database is an externally managed MySQL version 8, you will face a JAVA connector issue. (INSTALL-2647)
The Insights Overview tab uses UTC as the timezone, while other pages use local time. Hence, the date and time that are shown on the Insights Overview tab and the other pages after redirection can be different. (UIX-4176)
Kerberos can only be disabled manually from the
unravel.yaml
file.kerberos: enabled: False
WorkloadFit report
A large number of tags can cause the Workload Fit report to fail. (PG-265)
WorkloadFit report > Heatmap: The job count has data, but Vcore and memory are empty. (MIG-262)
On-prem playground: In Clusters > Hbase > Tables, the settings button does not respond. (PG-291)
Cluster discovery
If the metric retrieval for a host fails, then the CPU and memory capacity/usage graphs and heatmaps are not displayed.
This happens on a CDH cluster when the Cloudera Manager agent of a host does not send any heartbeats to the Cloudera Manager server. Such a host is shown as Bad Health in Cloudera Manager. (REPORT-1706)
Workaround: Ensure that the Cloudera Manager agent sends heartbeats to the Cloudera Manager on all hosts and that none of the hosts are shown as Bad Health.
The On-prem Cluster Identity might show an incorrect Spark version on CDH. The report may incorrectly show Spark 1 when Spark 2 is installed on the CDH cluster. (REPORT-1702)
When using PostgreSQL, the % sign is duplicated and displayed in the Workload Fit report > Map to single cluster tab. (MIG-42)
Cloud Mapping Per Host report scheduled in v4.6.1.x will not work in v4.7.1.0. Users must organize a new report. (REPORT-1886)
The TopX report email contains a link to the Unravel TopX report instead of showing the report content in the email as in the old reports.
Queue analysis: The log file name (
unravel_us_1.log
) displayed in the error message is incorrect. The correct name of the log file isunravel_sensor.log
. (REPORT-1663)
You can see a lag for SQL Streaming applications. (PLATFORM-2764)
If the customer uses an active directory for Kerberos and the samAccountName and principal do not match, this can cause errors when accessing HDFS. (DOC-755)
In AAD login mode when external logout happens, the user continues to have access to their current logged-in UI. (UIX-4125)
For PySpark applications, the processCPUTime and the processCPULoad are not captured properly. (ASP-626)
SQL events generator generates SQL Like clause event if the query contains a like pattern even in the literals. (TEZLLAP-349)
After upgrading to v4.7.1.0, Notebooks do not work. You can configure them separately. (REPORT-1895)
If you have configured a single cluster deployment for Unravel and the cluster name is not default, then the Datapage feature might not work correctly.
To avoid this issue, you must explicitly set the following property after upgrading. (INSTALL-2151)
<Unravel installation directory>/unravel/manager stop <Unravel installation directory>/unravel/manager config properties set hive.metastore.cluster.ids=
<cluster-name>
<Unravel installation directory>/unravel/manager apply <Unravel installation directory>/unravel/manager startAfter upgrading from v4.6.x to v4.7.1.0, the Tez application details page does not initially show DAG data. The DAG data is visible only after you refresh the page. (ASP-1126)
You can access the new user interface (UI) only from Chrome.
In the App summary page for Impala, the Query> Operator view is visible after scrolling down. (UIX-3536).
Jobs are falsely labeled as a Tez App for Oozie Sqoop and Shell actions. (PLATFORM-2403)