Skip to main content

Home

Upgrading Unravel from version 4.7.x to 4.7.4.x

This topic provides instructions to upgrade from Unravel version v4.7.0.x, v4.7.1.x, v4.7.2.x, 4.7.3.x to 4.7.4.x

Caution

Before upgrading Unravel, you must take a backup of the data directory and external database (if using). Perform the following steps:

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Archive the entire data directory.

    For example,

    tar czf /archive/destination/unravel-backup.tar.gz /opt/unravel/data

  3. If using an external database, perform a backup of the database.

    Note

    For the details on how to back up the database, refer to the documentation for your corresponding database.

After upgrading, based on your requirement, you can set the following:

The following upgrade paths are supported:

  • 4.7.0.x → 4.7.4.x

  • 4.7.1.x → 4.7.4.x

  • 4.7.2.x → 4.7.4.x

  • 4.7.3.x → 4.7.4.x

Upgrade Unravel in a single cluster deployment (v4.7.0.x, v4.7.1.x, v4.7.2.x, 4.7.3.x to v4.7.4.x)

Note

Unravel requires Java Runtime Environment (JRE) and hence it is shipped with OpenJDK version 17.0.1. If you have a different version of JDK installed, you must configure Unravel to access the corresponding jre directory in that JDK. For instructions, refer to Configuring custom JDK.

  1. Go to the Download section. The complete list of Unravel downloads is available in the section.

  2. Click the Unravel version that you want to upgrade. The section provides the details and necessary instructions to download Unravel.

  3. Download and extract the Unravel binaries of the Unravel version you want to upgrade.

    Note

    The default installation directory is /usr/local for RPM installation. This might have changed during the initial deployment. Make sure that you use the same installation directory when you upgrade Unravel.

  4. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  5. Activate the version that you want to upgrade.

    <Unravel installation directory>/unravel/manager activate <Unravel-version>

  6. Based on your platform, do the following:

    • On-prem platforms

      If you have used an automatic Hadoop configuration, run auto-configuration and refresh to reload the updates automatically.

      <unravel_installation_directory>/unravel/manager config auto --refresh

      The Hive metastore password may need to be set message is displayed.

      Example: 

      ATTENTION: Hive metastore password may need to be set.
Use: manager config hive metastore password HDPfc137 HIVE

      You can use the command after Use as it is followed by the <password> and press Enter.

    • Databricks

      • If you have used manager config properties set to force set Databricks Workspace properties, then you must unset the Workspace properties. Refer to Unset Workspace properties.

        Note

        You are no longer required to unset the workspace properties from Unravel version 4.7.4.2 onwards.

      • If Unravel Log Receiver is not accessible with the default hostname and port from Databricks, you must update Log Receiver (LR) endpoint for Databricks. Refer to Update Log Receiver (LR) endpoint.

  7. Apply the changes.

    <unravel_installation_directory>/unravel/manager config apply

  8. Start all the services.

    <unravel_installation_directory>/unravel/manager start

  9. Check the status of the services.

    <unravel_installation_directory>/unravel/manager report

    The following service statuses are reported:

    • OK: Service is up and running.

    • Not Monitored: Service is not running. (Has stopped or has failed to start)

    • Initializing: Services are starting up.

    • Does not exist: The process unexpectedly disappeared. A restart will be attempted 10 times.

  10. Validate the JAVA version that Unravel is using now:

  11. Delete the previous installation directory from unravel/versions/<THE.OLD.VERSION>.

Upgrade Unravel in multi-cluster deployment (v4.7.0.x, v4.7.1.x, v4.7.2.x, 4.7.3.x to v4.7.4.x)

To upgrade Unravel in a multi-cluster environment, you must upgrade all the edge nodes, upgrade the core node, and pull all the edge node updates to the core node.

Note

Unravel requires Java Runtime Environment (JRE) and hence it is shipped with OpenJDK version 17.0.1. If you have a different version of JDK installed, you must configure Unravel to access the corresponding jre directory in that JDK. For instructions, refer to Configuring custom JDK.

In a multi-cluster deployment, you must configure the jre directory both on the edge node as well as on the core node.

  1. Edge nodes

    Run the following steps to upgrade Unravel on the edge nodes involved in Unravel monitoring.

    1. Go to the Download page.

    2. Click the Unravel version you want to upgrade to and run the commands provided to download Unravel.

    3. Extract the Unravel binaries.

      Note

      The default installation directory is /usr/local, which might be changed during the initial deployment. Make sure that you use the same installation directory when you upgrade Unravel.

    4. Stop Unravel.

      <Unravel installation directory>/unravel/manager stop

    5. Activate the version that you want to upgrade to. 

      <Unravel installation directory>/unravel/manager activate <Unravel-version>

      Caution

      You may get a Network port failed precheck error if you run the manager activate command immediately after executing the manager stop command.

      To avoid this precheck error, stop Unravel and wait for a minute or two before executing the manager activate command.

      Retry the manager activate command in case of the Network port failed precheck error.

    6. If you have used an automatic Hadoop configuration, run auto-configuration and refresh to reload the updates automatically.

      <unravel_installation_directory>/unravel/manager config auto --refresh

    7. Apply the changes. 

      <unravel_installation_directory>/unravel/manager config apply

    8. Start all the services.

      <unravel_installation_directory>/unravel/manager start

    9. Check the status of services.

      <unravel_installation_directory>/unravel/manager report

      The following service statuses are reported:

      • OK: Service is up and running.

      • Not Monitored: Service is not running. (Has stopped or has failed to start)

      • Initializing: Services are starting up.

      • Does not exist: The process unexpectedly disappeared. Restarts will be attempted 10 times.

    10. Validate the JAVA version that Unravel is using now.

  2. Core node

    Run the following steps only on the core node.

    1. Download and extract the Unravel binaries of the Unravel version you want to upgrade.

    2. Stop Unravel.

      <Unravel installation directory>/unravel/manager stop

    3. Activate the version that you want to upgrade.

      <Unravel installation directory>/unravel/manager activate <Unravel-version>

    4. Run auto-configuration on the core node and refresh to reload the updates automatically. Run this command only if you have previously run the config auto command on the cluster. This is an optional step. 

      <unravel_installation_directory>/unravel/manager config auto --refresh

      Caution

      If you have NOT run the manager config auto earlier on the core node, an error will be displayed, which you can ignore.

    5. Find the edge key using the manager config edge show command, keep it handy and then run the following command for each of the configured edge nodes. This refreshes the configurations on the edge nodes. Provide the edge key when prompted: 

      <unravel_installation_directory>/unravel/manager config edge auto --refresh

      For example:

      /opt/unravel/manager config edge auto --refresh
-- Running: config edge auto --refresh
2021-09-22 03:32:25 Archiving configuration ... Ok
Edge key: edge-tnode39

    6. The Hive metastore database password can be recovered automatically only for a cluster manager with an administrative account. Otherwise, you must manually set the password as follows:

      1. Run the manager config edge show command to get the <EDGE_KEY>, <HIVE_KEY>, and <CLUSTER_KEY>, which must be provided when you set the Hive metastore password.

        • <EDGE_KEY> is the label you provide to identify the edge node when you add the edge node in Step 3.

        • CLUSTER_KEY is the name of the cluster where you set the Hive configurations.

        • <HIVE_KEY> is the definition of the Hive service. In the output of the manager config edge show command, this is shown as the <SERVICE_KEY> for Hive. 

        -- Running: config edge show
------------ | ---------------------------------------- | ------------
    EDGE KEY | - edge-a                                 | Enabled
             |     Cluster manager:                     | Enabled
             |     Clusters:                            | 
 CLUSTER KEY |       - Cluster_Name                     | Enabled
             |           HBASE:                         | 
 SERVICE KEY |             - hbase                      | Enabled
             |           HDFS:                          | 
 SERVICE KEY |             - hdfs                       | Enabled
             |           HIVE:                          | 
 SERVICE KEY |             - hive                       | Enabled
 SERVICE KEY |             - hive2                      | Enabled
             |           IMPALA:                        | 
 SERVICE KEY |             - impala                     | Enabled
 SERVICE KEY |             - impala2                    | Enabled
             |           KAFKA:                         | 
 SERVICE KEY |             - kafka                      | Enabled
 SERVICE KEY |             - kafka2                     | Enabled
             |           SPARK_ON_YARN:                 | 
 SERVICE KEY |             - spark_on_yarn              | Enabled
             |           YARN:                          | 
 SERVICE KEY |             - yarn                       | Enabled
             |           ZOOKEEPER:                     | 
 SERVICE KEY |             - zookeeper                  | Enabled
------------ | ---------------------------------------- | ------------
-- OK

      2. In a multi-cluster deployment, where edge nodes are monitoring, set the password on the core node as follows:

        <Unravel installation directory>/unravel/manager config edge hive metastore password <EDGE_KEY> <CLUSTER-KEY> <HIVE-KEY> <password> 
##Example: /opt/unravel/manager config edge hive metastore password local-node cluster1 hive password

        In case, the core node is monitoring the Hadoop cluster directly, run the following command from the core node. 

        <Unravel installation directory>/unravel/manager config hive metastore password <CLUSTER_KEY> <HIVE_KEY> <password> 
##Example: /opt/unravel/unravel/manager config edge hive metastore password cluster1 hive P@SSw0rd

    7. Apply the changes. 

      <unravel_installation_directory>/unravel/manager config apply

    8. Start all the services.

      <unravel_installation_directory>/unravel/manager start

    9. Check the status of services.

      <unravel_installation_directory>/unravel/manager report

    10. Delete the old install directory from unravel/versions/<THE.OLD.VERSION>.

Note

If an upgrade fails, you can roll back the upgrade to the release from which you upgraded. For information, see Rolling back after a failed upgrade.

Upgrade Unravel sensors

Refer to Upgrading sensors.

Important

In a multi-cluster deployment, it is mandatory to upgrade the core node before you upgrade the sensors on the edge node.

Unset Workspace properties for Databricks

Caution

Additional caution should be exercised if you have used manager config properties set to force set Databricks Workspace properties. You can skip this step and proceed only if you have used manager config databricks commands or the UI to configure workspaces.

Do the following to unset the Workspace properties:

  1. Extract the affected properties:

    grep com.unraveldata.databricks. /opt/unravel/data/conf/unravel.yaml > /tmp/unset-ws

  2. Review the properties in /tmp/unset-wsIf. if you have force set Databricks properties using manager config properties set and want to maintain these properties, you must remove such properties from /tmp/unset-ws. Else, these properties will get unset.

  3. Edit /tmp/unset-ws to remove the values, for example with sed:

    sed -i 's/:.*//' /tmp/unset-ws

    /tmp/unset-ws should now only have the Databricks workspace properties you want to unset.

    cat /tmp/unset-ws 
        com.unraveldata.databricks.lr.endpoint
        com.unraveldata.databricks.workspaces
        com.unraveldata.databricks.my-workspace-1.instance
        com.unraveldata.databricks.my-workspace-1.name
        com.unraveldata.databricks.my-workspace-1.tier
        com.unraveldata.databricks.my-workspace-1.token
        com.unraveldata.databricks.my-workspace-2.instance
        com.unraveldata.databricks.my-workspace-2.name
        com.unraveldata.databricks.my-workspace-2.tier
        com.unraveldata.databricks.my-workspace-2.token

  4. Use the Unravel manager utility to unset the properties:

    <Unravel installation directory>/unravel/manager config properties import /tmp/unset-ws
    The following properties have been unset:
- com.unraveldata.databricks.lr.endpoint
- com.unraveldata.databricks.workspaces
- com.unraveldata.databricks.my-workspace-1.instance
- com.unraveldata.databricks.my-workspace-1.name
- com.unraveldata.databricks.my-workspace-1.tier
- com.unraveldata.databricks.my-workspace-1.token
- com.unraveldata.databricks.my-workspace-2.instance
- com.unraveldata.databricks.my-workspace-2.name
- com.unraveldata.databricks.my-workspace-2.tier
- com.unraveldata.databricks.my-workspace-2.token
-- OK

  5. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply
Update Log Receiver (LR) endpoint for Databricks

Update the Log Receiver (LR) endpoint if the Unravel LR is not accessible with the default hostname and port from Databricks.

  1. Set LR hostname and LR port number to default based on the following scenarios:

    • TLS is NOT enabled, run the following:

      <Unravel installation directory>/unravel/manager config databricks set-lr-endpoint <hostname> 4043

    • TLS is enabled, run the following:

      <Unravel installation directory>/unravel/manager config databricks set-lr-endpoint <hostname> 4443

    • TLS is enabled, but you want to keep using HTTP, run the following:

      <Unravel installation directory>/unravel/manager config databricks set-lr-endpoint <hostname> 4043 --no-tls

    If TLS is enabled, the LR uses HTTPS protocol and listens on the 4443 port (by default). You must ensure that port 4443 to the unravel server is open.

  2. Apply changes.

    <Unravel installation directory>/unravel/manager config apply

  3. Update Workspaces.

    <Unravel installation directory>/unravel/manager refresh databricks

  4. Restart Unravel.

    <Unravel installation directory>/unravel/manager stop then config apply then start
Examples:

  • TLS is NOT enabled

    /opt/unravel/manager config databricks set-lr-endpoint 10.2.1.4 4043
/opt/unravel/manager stop then config apply then refresh databricks then start

  • TLS is enabled

    /opt/unravel/manager config databricks set-lr-endpoint 10.2.1.4 4443
/opt/unravel/manager stop then config apply then refresh databricks then start

  • TLS is enabled, but you want to continue using HTTPS

    /opt/unravel/manager config databricks set-lr-endpoint 10.2.1.4 4043 --no-tls
/opt/unravel/manager stop then config apply then refresh databricks then start
Configure Log Receiver (LR) authentication for Databricks (Optional)

Refer to Configuring LR authentication.

Enable Migration reports (Optional)

If you had migrations reports on CDH or CDP clusters, you must run the following steps to continue using the migration reports.

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  2. Enable the Migration reports as an admin user. Run the following command:

    <Unravel installation directory>/unravel/manager config ondemand cloud-migration enable

  3. Apply changes. 

    <Unravel installation directory>/unravel/manager config apply

  4. Start Unravel

    <Unravel installation directory>/unravel/manager start

    The Migration tab will be visible on the Unravel UI. Also, refer to the Migrations topics for more details.

Enable Impala monitoring

After the upgrade, if you want to monitor Impala applications, you must enable the impala_worker daemon. For instructions to enable the impala_worker daemon, refer to Impala monitoring.

Validate the JAVA version that Unravel is using

  1. Go to Unravel node and login as unravel user. In the case of multi-cluster, go to Unravel edge node.

  2. Run the following command:

    ps -ef | grep unravel | grep java

    For example, the following output is shown:

    unravel 26871 1 1 Jul05 ? 00:30:43 /tmp/jdk1.8.0_112/jre/bin/java -server -Dident=unravel_taw_1 -Dunravel.log.dir=/opt/unravel/logs -Dhadoop.version= -Djdbc.driver.jar=com.mysql.jdbc.Driver -Dhadoop.conf.dir=/etc/hadoop/conf -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dnetworkaddress.cache.ttl=30 -Dsun.net.inetaddr.ttl=30 -Xmx6g -Xms1g -Dvertx.cacheDirBase=/opt/unravel/tmp/ver

    In the above example, Unravel has selected the Java version /tmp/jdk1.8.0_112/jre/bin/java.

  3. You can further validate the Java version. For example, run the following command: 

    /tmp/jdk1.8.0_112/jre/bin/java -version
In this section: