v4.7.2.0 Release notes
Software version
Release Date: 23/Sept/2021
See 4.7.2.0 for download information.
Software upgrade support
The following upgrade paths are supported:
4.6.1.8 or earlier → 4.6.1.9 → 4.7.2.0
4.6.1.9 → 4.7.2.0
4.7.0.0 → 4.7.2.0
4.7.0.1 → 4.7.2.0
4.7.1.0 → 4.7.2.0
4.7.1.1 → 4.7.2.0
Sensor upgrade
Sensor upgrade is mandatory.
Certified platforms
The following platforms are tested and certified in this release:
Hortonworks Data Platform (HDP)
Cloudera Distribution of Apache Hadoop (CDH)
Cloudera Data Platform (CDP)
Amazon Elastic MapReduce (EMR)
Databricks (Azure Databricks and AWS Databricks)
Google Cloud Dataproc
Review your platform's compatibility matrix before you install Unravel.
Updates to Unravel's configuration properties
Refer to 4.7.x - Updates to Unravel properties.
New features
Insights Overview
The Insights Overview tab is added on the Unravel UI, which provides a single location to show a summary of the insights that Unravel aggregates across your clusters. The insights are designed to monitor and reduce cost on the cloud, improve resource efficiency, and enable app acceleration. This is a preview feature and is available only for EMR, Databricks, and on-prem (CDH, CDP, and HDP) platforms.
Google Cloud Dataproc
Google Cloud Dataproc platform is now a certified platform for Unravel. Dataproc version 1.5 and 2.0 are supported in this release.
Azure Active Directory (AAD)
AAD sign in is fully supported with roles based on AAD groups.
Databricks REST API can be used with AAD token as well as with Personal Access Token (PAT).
Improvements and enhancements
Apps
YarnRMConnector: reload RM list after failed RM request. (CDI-190)
Capture all YARN Application types in RM polling. (CDI-272)
Support for generic yarn applications (resource, logs). (ASP-1150)
Capture all critical log types for individual YARN applications. (ASP-1105)
Failure reasons for Impala queries are no longer missing on Unravel UI. (ASP-1078)
Logs and Errors tab for all attempts of a spark application will be populated. (ASP-849)
Databricks
Nested Notebooks are supported for Databricks.
Role-based Access Control (RBAC) is fully supported for Databricks.
Basic support for YARN-based app types
All YARN-based application types (e.g., SASGrid) outside of Spark, Tez, MapReduce get basic support including log loading, chargeback, and AutoAction applicable to YARN.
Migration
The heatmap is generated 24*7 for the Cluster Discovery reports. (MIG-15)
Ability to specify custom prices for instance types in the migration assessment reports. (MIG-64)
Access to fine-grained host metrics for accuracy in the Cloud Migration reports. (MIG-50)
Sum up cores/memory in instances summary in the Cloud Mapping Per Host report. (MIG-55, MIG-107)
The data for timeline graphs in the Cluster Discovery reports are averaged on an hourly basis. (MIG-100)
The heatmap for the Cluster discovery reports is averaged on an hourly basis. (MIG-101)
Updated prices for Google Cloud Platform in the Cloud Migration reports. (MIG-130)
The heatmap for Cluster Discovery reports shows the usage instead of ranks. (MIG-147)
Monthly and annual prices are shown for Cloud Mapping Per Host and Workload Fit reports. (MIG-110 MIG-111)
The Cluster Discovery reports can be viewed by CPU hours and Memory hours besides Job count. (MIG-54)
The Cloud Mapping Per Host reports are generated based on percentiles of resource usage. Only applicable for CDH and CDP platforms. (CUSTOMER-1741, MIG-98)
The heatmap for the Cluster Discovery report shows hourly data regardless of the date period selected. (CUSTOMER-1752)
RBAC
LDAP user group names are no longer case-sensitive. (RBAC-44)
UI
Enhanced menu bar for the Search box, Support, Documentation, About, and Logout options.
Added Finished tab in the app list to show the finished apps that ended in the selected period.
The job links are displayed for untracked clusters on the Cluster Insights page. (UIX-4121)
Unsupported
Billing is not supported on EMR and on-prem platforms.
On the Data page, File Reports, Small File reports, and file size information are not supported for EMR and Databricks clusters.
Impala jobs are not supported on the HDP platform.
Monitoring the expiration of the SSL Certificates and Kerberos principals in Unravel multi-cluster deployments.
Migration Planning is not supported for the following regions for Azure Data Lake:
Germany Central (Sovereign)
Germany Northeast (Sovereign)
Forecasting and Migration: In a multi-cluster environment, only a single cluster can be configured at a time; hence reports are generated only for that single cluster.
Unravel does not support multi-cluster management of combined on-prem and cloud clusters.
Unravel does not support apps belonging to the same pipeline in a multi-cluster environment but is sourced from different clusters. A pipeline can only contain apps that belong to the same cluster.
Reports are not supported on Databricks and EMR.
In Jobs > Sessions, the feature of applying recommendations and then running the newly configured app is not supported.
Pig and Cascading applications are not supported.
Upgrade to Unravel version 4.7.2.0 from Unravel version 4.6.1.9 TAR installation is not supported.
Bug fixes
Insights
SqlNoFilterEvent generated for a query having filter condition that is a subquery in the WHERE clause. (INSIGHTS-155)
Apps
App Summary > Changed Configuration tab for Hive on Tez app shows the wrong configuration for hive query. (ASP-1145)
The details page for running applications (vcore/memory/container) plots column charts instead of line charts, even if the number of series data is more. (ASP-1148)
Application not found message is not shown if app hitdoc is missing. (ASP-1163)
Scrolling of table showing user, jobs count, etc., is not working appropriately for Impala Chargeback UI. (ASP-1203)
EMR
The Cluster insights page is empty after upgrading from Unravel v4.6.1.9 to V4.7.2.0. (UIX-4167)
UI is not rendering the data even though it has the expected data from the backend in the Rest API response. (UIX-4174)
Migration
The Cloud Mapping Per Host report fails with an error: list index out of range: Cloud mapping per host report generation failed. (MIG-96)
A misleading error message is shown for the Workload Fit report when the credentials are not set properly. (MIG-141)
In the Service version Compatibility report, the version numbers are incorrect for CDP services. (MIG-151)
Reports
Duration of MapSlotDuration and ReduceSlotDuration is not getting updated in TopX reports (REPORT-1757)
Duplicate apps with the same cluster UID entry are found in Elastic Storage. (ASP-1044)
Event logs and YARN logs are not loaded for some applications in Google Dataproc clusters. (PG-170)
In the Spark apps details page, under the Resources tab, the label denoting attempts in resource graphs appears distorted. (PG-169)
Create time of partitions does not get captured in hive metastore if the partition is created dynamically. This limits Unravel to show Last Day KPIs for the partition section.
Wrong data displayed for Number of Partitions Created KPI/trend graph under Partitions KPIs - Last Day section in theData page. (DATAPAGE-473)
In Jobs>Runs tab and Cloud Clusters tab when you change the selected time interval to another interval and you are on a page that is not included in the new interval, the tab does not display any data. (UIX-4205)
Exception: Problem when retrieving bootstrap actions for cluster is seen in the aws_worker daemon logs.
Workaround: While creating AWS account for EMR Chargeback/Insights overview feature, you must include an additional entry in the
Policy JSON
file for"elasticmapreduce:ListBootstrapActions"
, as follows:{ “Version”: “2012-10-17", “Statement”: [ { “Effect”: “Allow”, “Action”: [ “pricing:GetProducts”, “elasticmapreduce:ListClusters”, “elasticmapreduce:DescribeCluster”, “elasticmapreduce:ListInstanceFleets”, “elasticmapreduce:ListInstanceGroups”, “elasticmapreduce:ListBootstrapActions“, “elasticmapreduce:ListInstances”, “ec2:DescribeSpotPriceHistory” ], “Resource”: “*” } ] }
Even though the AWS account was already created without this entry (
elasticmapreduce:ListBootstrapActions
), you can always include this policy later.
Unravel node fails to send email notifications. (INSTALL-1694)
The Insights Overview tab uses UTC as the timezone while other pages use local time. Hence, the date and time shown on the Insights Overview tab and the other pages after redirection can be different. (UIX-4176)
Kerberos can only be disabled manually from the
unravel.yaml
file.kerberos: enabled: False
Cluster discovery
If the metric retrieval for a host fails, then the CPU and memory capacity/usage graphs and heatmaps are not displayed.
This happens on a CDH cluster when the Cloudera Manager agent of a host does not send any heartbeats to the Cloudera Manager server. Such a host is shown as Bad Health in Cloudera Manager. (REPORT-1706)
Workaround: Ensure that the Cloudera Manager agent sends heartbeats to the Cloudera Manager on all hosts and that none of the hosts are shown as Bad Health.
The On-prem Cluster Identity may show an incorrect Spark version on CDH. The report may incorrectly show Spark 1 when Spark 2 is installed on the CDH cluster. (REPORT-1702)
When using PostgreSQL, the % sign is duplicated and displayed in the Workload Fit report > Map to single cluster tab. (MIG-42)
Cloud Mapping Per Host report scheduled in v4.6.1.x will not work in v4.7.1.0. Users must schedule a new report. (REPORT-1886)
The TopX report email contains a link to the Unravel TopX report instead of showing the report content in the email as in the old reports.
Queue analysis: The log file name (
unravel_us_1.log
) displayed in the error message is incorrect. The correct name of the log file isunravel_sensor.log
. (REPORT-1663)
The sensor setup script fails with unrecognized arguments. (INSTALL-1667)
There is a lag seen for SQL Streaming applications. (PLATFORM-2764)
If the customer uses an active directory for Kerberos and the samAccountName and principal do not match, this can cause errors when accessing HDFS. (DOC-755)
In AAD login mode when external logout happens, the user still has access to his current logged-in UI. (4125)
For PySpark applications, the processCPUTime and the processCPULoad are not captured properly. (ASP-626)
SQL events generator generates SQL Like clause event if the query contains like pattern even in the literals. (TEZLLAP-349)
Notebooks will not work after upgrading to v4.7.1.0. You can configure them separately. (REPORT-1895)
After you upgrade from v4.6.x to v4.7.1.0, the Tez application details page does not initially show DAG data. The DAG data is visible only after you refresh the page. (ASP-1126)
On the Manage page, the DB Stats are not displayed for untracked clusters. (UIX-4171)
The new user interface (UI) can be accessed only from Chrome.
In the App summary page for Impala, the Query> Operator view is visible after scrolling down. (UIX-3536).
Jobs getting falsely labeled as a Tez App for Oozie Sqoop and Shell actions. (PLATFORM-2403)