v4.6.1.3 Release notes
Software version
Release Date: 09/01/2020
See v4.6.1.3 for download information
Software upgrade support
v4.5.3.1 tarball upgrade. The following upgrade path is supported in v4.6.1.3:
4.5.3.3 to 4.6.1.3
Sensor upgrade
A sensor upgrade is mandatory.
Certified platforms
Review your platform's compatibility matrix before you upgrade or install.
Updates to Unravel's configuration properties
Refer to v4.6.x - Updates to Unravel properties.
Unsupported
AutoAction
ADF and Airflow for interactive clusters
Notebooks on interactive clusters
Spark Program / Query Graph for Notebook and Python tasks
Chargeback view by custom tags
Cost feature for cluster type that includes spot instances. For clusters including spot instances, the predicted cost does not reflect the purchase price and outbidding events.
Unravel's APIs
Sessions
Role-based Access Control (RBAC)
Data Insights for:
Workload
Spark
Reports
Small files
Cluster optimization
Notebooks
Top-X
Forecasting
Migration planning
Queue analysis
Datapage
Size created of the table
Total size
Accessed partitions
Size created of the partitions
Datapage
Size created of the table.
Total size
Size created of the partitions.
Oozie on EMR
Missing table and column statistics events.
Migration Planning is not supported for the following regions for Azure Data Lake:
US DoD East
US DoD Central
Germany Central (Sovereign)
Germany Northeast (Sovereign)
New features
Fully supported Unravel tarball installation.
Upgrade instructions
This section provides instructions to upgrade to Unravel v4.6.1.3.
Assumptions
The following assumptions are made in the upgrade instructions:
Unravel is installed to
/unravel-ro. You can verify the correct directory name by looking forsoftwaredirectory in that location. For example:/unravel-ro/software.Unravel is configured with
/unravel-rwas the writable location. You can verify the correct directory name by looking for a data directory. For example:/unravel-rw/data.
Upgrading to Unravel v4.6.1.3
To install Unravel from a tar file, do the following:
Change directories to
/unravel-ro.curl -v https://preview.unraveldata.com/unravel/RPM/4.6.1/unravel-4.6.1.3-CDH.tar.gz -o unravel-4.6.1.3-CDH.tar.gz -u
username:passwordUntar
unravel-4.6.1.3.tar zxf unravel-4.6.1.3.tar.gz -C /unravel-ro
Stop Unravel.
/unravel-rw/service/manager.sh stop
Run the setup script.
/unravel-ro/unravel/versions/4.6.1.3.333/setup --migrate-ms /unravel-rw/data
(Optional) Refresh the cluster configuration.
/unravel-rw/manager config auto
Start unravel.
/unravel-rw/manager start
Check the status.
/unravel-rw/manager report
Improvements and enhancements
Workflow
ID is added to the workflows that return from /api/v1/workflows. (CUSTOMER-1524)
Date time filter added in /api/v1/workflows/{workflow Id}/annotation endpoint to return a specific range of time. (CUSTOMER-1524)
Bug fixes
Reports
In Reports> Datasight, the table search is functioning appropriately. (CUSTOMER-1528)
Impala
In some cases, the unravel_tw daemon failed to deserialize Impala queries. This issue is resolved. (CUSTOMER-1518)
Job Run cost is estimated based on the total number of nodes present in the cluster and irrespective of whether all the nodes are in use by the Job Run or not.
AWS Databricks cost supports Standard and Premium tiers only. Enterprise and Dedicated are not supported.
Operations dashboards do not support multi-cluster and have incorrect aggregations.
Operations Nodes Dashboard does not capture cluster inactivity in graphs.
Spark Application with the same application ID is captured as one. It is captured correctly in the Billing Service.
Default Databricks Spark extraJavaOptions are overwritten by Unravel for spark-submit tasks.
DriverOOME and ExecutorOOME events are not generated for the Databricks notebook task.
Recommended Azure instances available in Cluster page but not at run time.
Recommended Azure instances could be in Beta mode only.
Instance recommendation is missing when EMDB is used.
The Violation Badge functionality for AutoAction is not working for Impala queries (Running, Killed). (AA-44)
EMR: Hive metrics are not published in the RUNNING state. (HIVE-135)
INSERT statements are not included in the App count on HDI. (DATAPAGE-256)
Accessed partition does not support Hive on Tez etc. It only supports Hive on MR. (DATAPAGE-250)
Latency in fetching the data for MR jobs. (PLATFORM-1613)
API connection error while Polling impalad metrics from CM. (PLATFORM-1567)
conflicted ephemeral node' or 'Corrupt index found'(PLATFORM-702)
gc load metric sensor for MR application will not load on EMR.
All files deleted except fsimage.txt (parsed fsimage) after parse fsimage is processed. (REPORT-1195)
Certain users and queues are missing in the Cluster Compare report. (REPORT-1342)
In the Cluster Compare report, the trends of the same values are highlighted instead of trends with different values.(REPORT-1479)
FSImage process fails with an error message. (REPORT-1635)
Workaround: Before running the etl fsi image process, check if the
/opt/unravel/data/tmp/reports/spark_tmpdirectory exists. If it does not exist, manually create/opt/unravel/data/tmp/reports/spark_tmpdirectory.
For PySpark applications, the processCPUTime and the processCPULoad are not captured properly. (USPARK-626)
Partition size 0 is shown in the insight message on the timings tab. (USPARK-647)
The number of Apps is incorrect. (PLATFORM-2403)
Spark apps running in Oozie workflow is not visible under Oozie workflow in Unravel UI. (PLATFORM-2746)
The Session apply feature does not function in a kerberized environment in Unravel v4.6.1.3 version. However, both Session creation and Session analyze features work as expected. (SESS-329)
Workaround: Pick the recommendations from the session and apply them to the application manually.