Skip to main content

Home

Upgrading Unravel from version 4.7.x to 4.7.6.x

This topic provides instructions to upgrade Unravel from version v4.7.0.x, v4.7.1.x, v4.7.2.x, 4.7.3.x, 4.7.4.x, v4.7.5.x to v4.7.6.x for the following platform:

  • On-prem platforms (single cluster and multi-cluster deployments): Cloudera Distribution of Apache Hadoop (CDH), Cloudera Data Platform (CDP), and Hortonworks Data Platform (HDP)

  • Cloud platforms: Amazon EMR, Databricks (Azure, AWS), GCP - Dataproc, GCP - BigQuery

The following upgrade paths are supported:

  • 4.7.5.x → 4.7.6.x

  • 4.7.4.x → 4.7.6.x

  • 4.7.3.x → 4.7.6.x

  • 4.7.2.x → 4.7.6.x

  • 4.7.1.x → 4.7.6.x

  • 4.7.0.x → 4.7.6.x

Cloud platforms upgrade (v4.7.x to v4.7.6.0)
Amazon EMR, Databricks, GCP-Dataproc, GCP-Bigquery

This section provides information about upgrading Unravel for cloud platforms (Amazon EMR, Databricks, GCP-Dataproc, GCP-Bigquery)

Important notice - Amazon EMR platform

  • For the Amazon EMR platform, the data collected for Clusters/Cost tabs in the previous versions of Unravel (v4.7.3.0, v4.7.4.0) will not be retained after you upgrade to Unravel v4.7.6.0.

  • Unravel does not support the Insights Overview tab on the UI for the EMR platform.

Before you upgrade Unravel for Cloud platforms (v4.7.x to v4.7.6.0)

Before you upgrade to Unravel (Cloud), do the following:

Backup the data directory

Take a backup of the data directory and the external database if you use one. To back up the data, refer to Backing-up and recovering Unravel

Note

If an upgrade fails, you can roll back the upgrade to the release from which you had upgraded. For information, see Rollback after a failed upgrade. For this rollback to work, you must ensure to take a backup of the data directory and the external database before you upgrade.

Configure custom JDK, if you have a different version of JDK installed

Unravel requires Java Runtime Environment (JRE), and hence it is shipped with OpenJDK version 17.0.1. If you have a different version of JDK installed, you must configure Unravel to access the corresponding jre directory in that JDK. For instructions, refer to Configuring custom JDK.

Check workspaces

Run the following command to verify all the Databricks tokens are valid and fix them before running the upgrade

<Unravel installation directory>/unravel/manager refresh databricks
Reset global init scripts (For Databricks platform only)

The global init script is not backward compatible, and therefore, before you upgrade, ensure to disable or remove the existing global init scripts and set the new global init scripts. Also, the following tokens/passcodes will be encrypted. For this encryption to take effect too, you must disable or remove all the previously set up global init scripts and set up the new global init scripts.

  • DBX Personal Access Token (PAT)

  • LR Authentication password

  • Unravel trust store password

  • Azure Active Directory token

Do the following to reset the global init scripts.

Upgrade Unravel for Cloud platforms (v4.7.x to v4.7.6.0)

  1. Go to the Download section. The complete list of Unravel downloads is available in this section.

  2. Download the Unravel version that you want to upgrade. The section provides the details and necessary instructions to download Unravel.

  3. Extract the Unravel binaries of the Unravel version you want to upgrade.

    tar zxf unravel-<version>tar.gz -C </path/to/installation/directory>

    For example: tar zxf unravel-4.7.0.0.tar.gz -C /opt

  4. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  5. Activate to Unravel version 4.7.6.0.

    <Unravel installation directory>/unravel/manager activate <version>

  6. Apply the changes.

    <unravel_installation_directory>/unravel/manager config apply

  7. Start all the services.

    <unravel_installation_directory>/unravel/manager start

  8. Check the status of the services.

    <unravel_installation_directory>/unravel/manager report

    The following service statuses are reported:

    • OK: Service is up and running.

    • Not Monitored: Service is not running. (Has stopped or has failed to start)

    • Initializing: Services are starting up.

    • Does not exist: The process unexpectedly disappeared. A restart will be attempted 10 times.

  9. Validate the JAVA version that Unravel is using now.

  10. Delete the previous installation directory from unravel/versions/<THE.OLD.VERSION>.

Important

Ensure to perform the post-upgrade steps. Refer to Post upgrade steps for Cloud platforms (v4.7.x to v4.7.6.0).

Post upgrade steps for Cloud platforms (v4.7.x to v4.7.6.0)

After you upgrade Unravel, set the following:

Update Log Receiver (LR) endpoint for Databricks

Update the Log Receiver (LR) endpoint if the Unravel LR is not accessible with the default hostname and port from Databricks.

  1. Set LR hostname and LR port number to default based on the following scenarios:

    • TLS is NOT enabled, run the following:

      <Unravel installation directory>/unravel/manager config databricks set-lr-endpoint <hostname> 4043

    • TLS is enabled, run the following:

      <Unravel installation directory>/unravel/manager config databricks set-lr-endpoint <FQDN> 4443

    • TLS is enabled, but if you want to keep using HTTP, run the following:

      <Unravel installation directory>/unravel/manager config databricks set-lr-endpoint <FQDN> 4043 --no-tls

    If TLS is enabled, the LR uses HTTPS protocol and listens on the 4443 port (by default). You must ensure that port 4443 to the unravel server is open.

  2. Apply changes.

    <Unravel installation directory>/unravel/manager config apply

  3. Update Workspaces.

    <Unravel installation directory>/unravel/manager refresh databricks

  4. Restart Unravel.

    <Unravel installation directory>/unravel/manager start
Examples:

  • TLS is NOT enabled

    /opt/unravel/manager config databricks set-lr-endpoint 10.2.1.4 4043
/opt/unravel/manager stop then config apply then refresh databricks then start

  • TLS is enabled

    /opt/unravel/manager config databricks set-lr-endpoint myhost.unraveldata.com 4443
/opt/unravel/manager stop then config apply then refresh databricks then start

  • TLS is enabled, but you want to continue using HTTPS

    /opt/unravel/manager config databricks set-lr-endpoint myhost.unraveldata.com 4043 --no-tls
/opt/unravel/manager stop then config apply then refresh databricks then start
Delete the Bootstrap files in the /tmp directory

In the previous releases, the bootstrap file which was generated during interactive precheck installation was placed in the /tmp/unravel-interactive-precheck directory. In this release, for better security, this file is moved to the $HOME directory of the Unravel user.

The $HOME directory will be moved for new installations. However, when you upgrade to v4.7.6.0, you must ensure to run the following command and delete the unravel-interactive-precheck directory from the /tmp directory.do the following:

rm -rf /tmp/unravel-interactive-precheck
Empty the JSON file created for API tokens

If you have created a JSON file for handling the API tokens persistently, then you must empty this JSON file after the upgrade. This is because the JWT secret has changed after the upgrade, so the earlier generated tokens are no longer valid.

Caution

Ensure that the JSON file is NOT placed in the /tmp directory or any other directory that is accessed by users other than Unravel users. In case, the JSON file is in a directory accessed by other users, then move it to the $HOME directory of the Unravel user.

Also, after you move the JSON file to the $HOME directory, configure the persistent API authorization tokens and update the new file path. Refer to Storing persistent API authorization tokens.

Configure Log Receiver (LR) authentication for Databricks (Optional)

Refer to Configuring LR authentication.

Set up Unravel to receive BigQuery data automatically

Refer to Set up Unravel to receive BigQuery data automaticallyInstalling Unravel for GCP BigQuery

Note

The plain text service account keys on Unravel server are encrypted when you upgrade to Unravel v4.7.6.0.

On-prem platforms upgrade (v4.7.x to v4.7.6.0)
Cloudera Distribution of Apache Hadoop (CDH), Cloudera Data Platform (CDP), and Hortonworks Data Platform (HDP)

This section provides information about upgrading Unravel for on-prem platforms (CDH, CDP, HDP, and MapR)

Before you upgrade Unravel for on-prem platforms

  • Backup the data directory

    Take a backup of the data directory and the external database if you use one. To back up the data, refer to Backing-up and recovering Unravel

    Note

    If an upgrade fails, you can roll back the upgrade to the release from which you had upgraded. For information, see Rollback after a failed upgrade. For this rollback to work, you must ensure to take a backup of the data directory and the external database before you upgrade.

  • Configure custom JDK, if you have a different version of JDK installed

    Unravel requires Java Runtime Environment (JRE), and hence it is shipped with OpenJDK version 17.0.1. If you have a different version of JDK installed, you must configure Unravel to access the corresponding jre directory in that JDK. For instructions, refer to Configuring custom JDK.

    In a multi-cluster deployment, you must configure the jre directory both on the edge node and the core node.

Upgrade Unravel for on-prem platforms (v4.7.x to v4.7.6.0)

You can upgrade Unravel for the single cluster as well as multi-cluster deployments. Refer to the following:

Important

Ensure to perform the post-upgrade steps. Refer to Post upgrade steps for on-prem platforms (v4.7.x to v4.7.6.0).

Upgrade Unravel in a single cluster deployment (v4.7.x to v4.7.6.0)

  1. Go to the Download section. The complete list of Unravel downloads is available in this section.

  2. Select and download the Unravel version that you want to upgrade. The section provides the details and necessary instructions to download Unravel.

  3. Extract the Unravel binaries of the Unravel version you want to upgrade.

    tar zxf unravel-<version>tar.gz -C </path/to/installation/directory>

    For example: tar zxf unravel-4.7.0.0.tar.gz -C /opt

    Note

    The default installation directory is /usr/local for RPM installation. This might have changed during the initial deployment. Make sure that you use the same installation directory when you upgrade Unravel.

  4. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  5. Activate to Unravel version 4.7.6.0.

    <Unravel installation directory>/unravel/manager activate <version>

    Note

    For on-prem platforms (CDH, CDP, HDP or MapR), If you have used an automatic Hadoop configuration, run auto-configuration and refresh to reload the updates automatically.

    <unravel_installation_directory>/unravel/manager config auto --refresh

  6. Apply the changes.

    <unravel_installation_directory>/unravel/manager config apply

  7. Start all the services.

    <unravel_installation_directory>/unravel/manager start

  8. Check the status of the services.

    <unravel_installation_directory>/unravel/manager report

    The following service statuses are reported:

    • OK: Service is up and running.

    • Not Monitored: Service is not running. (Has stopped or has failed to start)

    • Initializing: Services are starting up.

    • Does not exist: The process unexpectedly disappeared. A restart will be attempted 10 times.

  9. Validate the JAVA version that Unravel is using now.

  10. Delete the previous installation directory from unravel/versions/<THE.OLD.VERSION>.

Upgrade Unravel in multi-cluster deployment (v4.7.x to v4.7.6.0)

To upgrade Unravel in a multi-cluster environment, you must upgrade all the edge nodes, upgrade the core node, and pull all the edge node updates to the core node.

  1. Edge nodes

    Run the following steps to upgrade Unravel on the edge nodes involved in Unravel monitoring.

    1. Go to the Download page.

    2. Click the Unravel version you want to upgrade to and run the commands provided to download Unravel.

    3. Extract the Unravel binaries of the Unravel version you want to upgrade.

      tar zxf unravel-<version>tar.gz -C </path/to/installation/directory>

      For example: tar zxf unravel-4.7.0.0.tar.gz -C /opt

      Note

      The default installation directory is /usr/local for RPM installation. This might have changed during the initial deployment. Make sure that you use the same installation directory when you upgrade Unravel.

    4. Stop Unravel.

      <Unravel installation directory>/unravel/manager stop

    5. Activate the version that you want to upgrade. 

      <Unravel installation directory>/unravel/manager activate <version>

      Caution

      You may get a Network port failed precheck error if you run the manager activate command immediately after executing the manager stop command.

      To avoid this precheck error, stop Unravel and wait for a minute or two before executing the manager activate command.

      Retry the manager activate command in case of the Network port failed precheck error.

    6. If you have used an automatic Hadoop configuration, run auto-configuration and refresh to reload the updates automatically.

      <unravel_installation_directory>/unravel/manager config auto --refresh

    7. Apply the changes. 

      <unravel_installation_directory>/unravel/manager config apply

    8. Start all the services.

      <unravel_installation_directory>/unravel/manager start

    9. Check the status of services.

      <unravel_installation_directory>/unravel/manager report

      The following service statuses are reported:

      • OK: Service is up and running.

      • Not Monitored: Service is not running. (Has stopped or has failed to start)

      • Initializing: Services are starting up.

      • Does not exist: The process unexpectedly disappeared. Restarts will be attempted 10 times.

    10. Validate the JAVA version that Unravel is using now.

  2. Core node

    Run the following steps only on the core node.

    1. Download Unravel.

    2. Extract the Unravel binaries of the Unravel version you want to upgrade.

      tar zxf unravel-<version>tar.gz -C </path/to/installation/directory>

      For example: tar zxf unravel-4.7.0.0.tar.gz -C /opt

    3. Stop Unravel.

      <Unravel installation directory>/unravel/manager stop

    4. Activate the version that you want to upgrade.

      <Unravel installation directory>/unravel/manager activate <version>

    5. Run auto-configuration on the core node and refresh to reload the updates automatically. Run this command only if you have previously run the config auto command on the cluster. This is an optional step. 

      <unravel_installation_directory>/unravel/manager config auto --refresh

      Caution

      If you have NOT run the manager config auto earlier on the core node, an error will be displayed, which you can ignore.

    6. Find the edge key using the manager config edge show command, keep it handy and run the following command for each configured edge node. This refreshes the configurations on the edge nodes. Provide the edge key when prompted: 

      <unravel_installation_directory>/unravel/manager config edge auto --refresh

      For example:

      /opt/unravel/manager config edge auto --refresh
-- Running: config edge auto --refresh
2021-09-22 03:32:25 Archiving configuration ... Ok
Edge key: edge-tnode39

    7. The Hive metastore database password can be recovered automatically only for a cluster manager with an administrative account. Otherwise, you must manually set the password as follows:

      1. Run the manager config edge show command to get the <EDGE_KEY>, <HIVE_KEY>, and <CLUSTER_KEY>, which must be provided when you set the Hive metastore password.

        • <EDGE_KEY> is the label you provide to identify the edge node when you add the edge node in Step 3.

        • CLUSTER_KEY is the name of the cluster where you set the Hive configurations.

        • <HIVE_KEY> is the definition of the Hive service. In the output of the manager config edge show command, this is shown as the <SERVICE_KEY> for Hive. 

        -- Running: config edge show
------------ | ---------------------------------------- | ------------
    EDGE KEY | - edge-a                                 | Enabled
             |     Cluster manager:                     | Enabled
             |     Clusters:                            | 
 CLUSTER KEY |       - Cluster_Name                     | Enabled
             |           HBASE:                         | 
 SERVICE KEY |             - hbase                      | Enabled
             |           HDFS:                          | 
 SERVICE KEY |             - hdfs                       | Enabled
             |           HIVE:                          | 
 SERVICE KEY |             - hive                       | Enabled
 SERVICE KEY |             - hive2                      | Enabled
             |           IMPALA:                        | 
 SERVICE KEY |             - impala                     | Enabled
 SERVICE KEY |             - impala2                    | Enabled
             |           KAFKA:                         | 
 SERVICE KEY |             - kafka                      | Enabled
 SERVICE KEY |             - kafka2                     | Enabled
             |           SPARK_ON_YARN:                 | 
 SERVICE KEY |             - spark_on_yarn              | Enabled
             |           YARN:                          | 
 SERVICE KEY |             - yarn                       | Enabled
             |           ZOOKEEPER:                     | 
 SERVICE KEY |             - zookeeper                  | Enabled
------------ | ---------------------------------------- | ------------
-- OK

      2. In a multi-cluster deployment, where edge nodes are monitoring, set the password on the core node as follows:

        <Unravel installation directory>/unravel/manager config edge hive metastore password <EDGE_KEY> <CLUSTER-KEY> <HIVE-KEY> <password> 
##Example: /opt/unravel/manager config edge hive metastore password local-node cluster1 hive password

        In case, the core node is monitoring the Hadoop cluster directly, run the following command from the core node. 

        <Unravel installation directory>/unravel/manager config hive metastore password <CLUSTER_KEY> <HIVE_KEY> <password> 
##Example: /opt/unravel/unravel/manager config edge hive metastore password cluster1 hive P@SSw0rd

    8. Apply the changes. 

      <unravel_installation_directory>/unravel/manager config apply

    9. Start all the services.

      <unravel_installation_directory>/unravel/manager start

    10. Check the status of services.

      <unravel_installation_directory>/unravel/manager report

    11. Delete the old install directory from unravel/versions/<THE.OLD.VERSION>.

Important

Ensure to perform the post-upgrade steps. Refer to Post upgrade steps for on-prem platforms (v4.7.x to v4.7.6.0).

Post upgrade steps for on-prem platforms (v4.7.x to v4.7.6.0)

After you upgrade Unravel, set the following:

Upgrade Unravel sensors

Refer to Upgrading sensors.

Important

In a multi-cluster deployment, upgrading the core node is mandatory before upgrading the sensors on the edge node.

Delete the Bootstrap files in the /tmp directory

In the previous releases, the bootstrap file which was generated during interactive precheck installation was placed in the /tmp/unravel-interactive-precheck directory. In this release, for better security, this file is moved to the $HOME directory of the Unravel user.

After you upgrade Unravel to v4.7.6.0, ensure to run the following command and delete the unravel-interactive-precheck directory from the /tmp directory.

rm -rf /tmp/unravel-interactive-precheck
Empty the JSON file created for API tokens

If you have created a JSON file for handling the API tokens persistently, then you must empty this JSON file after the upgrade. This is because the JWT secret has changed after the upgrade, so the earlier generated tokens are no longer valid.

Caution

Ensure that the JSON file is NOT placed in the /tmp directory or any other directory that is accessed by users other than Unravel users. In case, the JSON file is in a directory accessed by other users, then move it to the $HOME directory of the Unravel user.

Also, after you move the JSON file to the $HOME directory, configure the persistent API authorization tokens and update the new file path. Refer to Storing persistent API authorization tokens.

Enable Migration reports (Optional)

If you had migrations reports on CDH or CDP clusters, you must run the following steps to continue using the migration reports.

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Enable the Migration reports as an admin user. Run the following command:

    <Unravel installation directory>/unravel/manager config ondemand cloud-migration enable

  3. Apply changes. 

    <Unravel installation directory>/unravel/manager config apply

  4. Start Unravel

    <Unravel installation directory>/unravel/manager start

    The Migration tab will be visible on the Unravel UI. Also, refer to the Migrations topics for more details.

Enable SSL/TLS support for external MySQL

After you upgrade, you can set the SSL/TLS support for external MySQL.

Important

You can enable the SSL/TLS support only for MySQL version 8.

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Use an editor to open <Installation_directory>/unravel/data/conf/unravel.yaml file.

  3. In the unravel.yaml file, change tls value to true. 

    database:
    external: true
    hostname: <hostname>
    port: 'port'
    schema: unravel_db_prod
    tls: true
    type: mysql
    username: <username>
    password: <encrypted password>

  4. Apply the changes. 

    <Unravel installation directory>/unravel/manager config apply

  5. Start Unravel.

    <Unravel installation directory>/unravel/manager start
Validate the JAVA version of Unravel

  1. Go to Unravel node and login as unravel user. In the case of multi-cluster, go to Unravel edge node.

  2. Run the following command:

    ps -ef | grep unravel | grep java

    For example, the following output is shown:

    unravel 26871 1 1 Jul05 ? 00:30:43 /tmp/jdk1.8.0_112/jre/bin/java -server -Dident=unravel_taw_1 -Dunravel.log.dir=/opt/unravel/logs -Dhadoop.version= -Djdbc.driver.jar=com.mysql.jdbc.Driver -Dhadoop.conf.dir=/etc/hadoop/conf -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dnetworkaddress.cache.ttl=30 -Dsun.net.inetaddr.ttl=30 -Xmx6g -Xms1g -Dvertx.cacheDirBase=/opt/unravel/tmp/ver

    In the above example, Unravel has selected the Java version /tmp/jdk1.8.0_112/jre/bin/java.

  3. You can further validate the Java version. For example, run the following command: 

    /tmp/jdk1.8.0_112/jre/bin/java -version
In this section: