Skip to main content

Home

Install Unravel in Amazon Elastic MapReduce (EMR)

Before installing Unravel on EMR, check and ensure that the Unravel installation requirements are completed and follow the instructions to install and configure Unravel:

1. Create and configure an EC2 instance

2. Download Unravel

3. Deploy Unravel binaries

4. Install Unravel

5. Connect a new or existing EMR cluster to Unravel

6. Add AWS account details in Unravel for EMR chargeback data and cluster insights

1. Create and configure an EC2 instance

Run the following steps to create an EC2 instance:

Run the following steps to configure the EC2 instance:

  1. Disable selinux.

    sudo setenforce Permissive

  2. Edit /etc/selinux/config to make sure the setting persists after reboot and make sure SELINUX=permissive.

    sudo vi /etc/selinux/config

  3. Install libaio.x86_64, lzop.x86_64, and ntp.x86_64.

    sudo yum install -y libaio.x86_64 (##Only required if you use Unravel managed MySQL)
sudo yum install -y lzop.x86_64
sudo yum install -y ntp.x86_64

  4. Start ntpd and check the system time.

    sudo service ntpd start
sudo ntpq -p

  5. Create a new user named hadoop.

    sudo useradd hadoop
2. Download Unravel
3. Deploy Unravel binaries
4. Install Unravel using interactive precheck

You can install Unravel with Interactive precheck. When you run the Interactive Precheck utility, a bootstrap configuration file is generated for installation.

The Interactive Precheck utility validates the required configurations before installing Unravel. When you run the Interactive Precheck utility, various checks are prompted for gathering configuration information. The responses you provide for these checks generate a bootstrap configuration file. This file, which contains the configuration information, is then used to install Unravel.

Do the following to install and configure Unravel with Interactive Precheck.

  1. After you download and deploy the Unravel, run the precheck.sh script from unravel/versions/X.Y.Z/healthcheck/.

    For example:

    /opt/unravel/versions/X.Y.Z/healthcheck/precheck.sh

  2. Enter the necessary details when you are prompted for the following configuration information:

    Note

    For more information, refer to Using the Interactive Precheck utility.

    The responses that you have provided for the configuration information are used to generate a configuration file. You can use this configuration file when you run the setup command to install and configure Unravel.

  3. After you have completed the responses, you are prompted to confirm if you want to generate the bootstrap configuration file. Press ENTER if you want to generate the bootstrap configuration file. 

    -- Unravel bootstrap configuration

   Generate a unravel bootstrap configuration file? (y/n) [Yes]:

    The bootstrap configuration file is generated and located at $HOME/unravel-interactive-precheck/unravel-bootstrap.yaml.

  4. Install Unravel with the bootstrap configuration file. 

    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --bootstrap $HOME/unravel-interactive-precheck/unravel-bootstrap.yaml

  5. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  6. Start all the services.

    <unravel_installation_directory>/unravel/manager start

  7. Check the status of services.

    <unravel_installation_directory>/unravel/manager report

    The following service statuses are reported:

    • OK: Service is up and running.

    • Not Monitored: Service is not running. (Has stopped or has failed to start)

    • Initializing: Services are starting up.

    • Does not exist: The process unexpectedly disappeared. A restart will be attempted ten times.

    You can also get the status and information for a specific service. Run the manager report command as follows:

    <unravel_installation_directory>/unravel/manager report <service>

    For example: /opt/unravel/manager report auto_action

You can also install Unravel manually. Refer to Installing Unravel manually . Manually install Unravel in Amazon Elastic MapReduce (EMR)

5. Connect a new or existing EMR cluster to Unravel

This topic explains how to set up and configure your EMR cluster, so Unravel can begin monitoring jobs running on the cluster.

Assumptions

  • The EC2 instance for Unravel is created.

  • Unravel services are running.

  • The nodes in the EMR cluster allow all traffic from the Unravel EC2 instance. This implies either of the following configurations:

  • Network ACL on VPC allows all traffic.

Warning

If you encounter any EMR cluster configuration issues, see the Troubleshooting guide to resolve the issues.

Connect to a new EMR cluster

Follow the steps below to run Unravel's bootstrap script, unravel_emr_bootstrap.py, on all nodes in the cluster. The bootstrap script makes the following changes:

  • On the master node:

    • On Hive clusters, it updates /lib/hive/conf/hive-site.xml

    • On Spark clusters, it updates /lib/spark/conf/spark-defaults.conf

    • It updates /lib/hadoop/etc/hadoop/mapred-site.xml

    • It updates /lib/hadoop/etc/hadoop/yarn-site.xml

    • If Tez is installed, it updates /etc/tez/conf/tez-site.xml

    • It installs and starts the unravel_es daemon in /usr/local/unravel_es

    • It installs the Spark and MapReduce sensors in /usr/local/unravel-agent/jars

    • It installs the Hive Hook sensor in /usr/lib/hive/lib/.

  • On all other nodes:

    • It installs the Spark and MapReduce sensors in /usr/local/unravel-agent/jars.

Run the following steps to connect Unravel to a new EMR cluster. Simultaneously, you can watch the following video tutorial:

  1. Download Unravel's bootstrap script, unravel_emr_bootstrap.py.

    curl https://s3.amazonaws.com/unraveldatarepo/unravel_emr_bootstrap.py -o /tmp/unravel_emr_bootstrap.py

  2. Upload the bootstrap script to an S3 bucket.

    Permissions needed

    You need to write access to the S3 bucket if you want to upload the bootstrap script to. Also, the AWS account you use to create the EMR cluster must have read access to the bootstrap script to execute its directives.

    To upload the bootstrap script to the default EMR logging bucket, s3://aws-logs-account_number-region/elasticmapreduce, execute the following command:

    aws s3 cp unravel_emr_bootstrap.py s3://aws-logs-account_number-region/elasticmapreduce

  3. On the AWS console, select the EMR service and click Create cluster.

  4. On the Create Cluster - Quick Options page, click Go to advanced options.

  5. In Step 1: Software and Steps, select emr-6.2 release.

  6. In Step 2: Hardware, enter the following configuration for your EMR cluster and click Next.

    Settings

    Action

    Instance group configuration

    By default, the Uniform instance groups option is selected.

    Network

    EC2 Subnet

    The default configuration works for Network and EC2 Subnet if you have a virtual machine hosted in AWS in the same subnet.

    If your virtual machine is hosted on a different subnet in AWS or on GCP or Azure, then the virtual machine and the cloud platform must have access to the public IP.

  7. In Step 3: General Cluster Settings, specify the following settings in Bootstrap Actions > Add Bootstrap Action, click Custom Action Add and then click Configure and add.

  8. In the Add Bootstrap Action window, enter the following details and click Add:

    Settings

    Action

    Name

    Select Custom action.

    Script location

    Specify the following bootstrap location:

    s3://unraveldatarepo/unravel_emr_bootstrap.py

    Note

    If you want to monitor MR jobs, then you must pass an additional optional argument --all

    Optional arguments

    Enter the following:

    --unravel-server UNRAVEL-INSTANCE-IP --all --bootstrap

    The Amazon EMR cluster starts with this bootstrap action.

  9. In Step 4: Security, edit the configuration for the cluster as required. For example:

    • Choose the EC2 key pair.

    • Select the EC2 security groups. AWS EMR service automatically applies additional rules that are required for EMR nodes.

  10. Click Create cluster. Your new EMR cluster finishes the bootstrap process and is in the Waiting state.

Connect to an existing EMR cluster

To connect the Unravel EC2 instance to an existing EMR cluster, follow the steps below to run the Unravel EMR Ansible playbook on either the EMR master node or your Mac/Linux workstation.

Important

The following process is for existing clusters created without Unravel bootstrap. Only those clusters of this type that do not have auto-scaling enabled are currently supported.

Repeat the steps below to upgrade Unravel Sensors whenever you upgrade Unravel Server.

Note

In case you have to run unravel_emr_bootstrap.py manually, you must run it with the full path of the system default python, that is /usr/bin/python, as follows:

sudo /usr/bin/python unravel_emr_bootstrap.py
6. Add AWS account details in Unravel

Important

This is a mandatory step. If you do not add the AWS account details in Unravel, you cannot view the EMR clusters.

Configuring CloudWatch agent

You can configure the CloudWatch agent for Unravel monitoring of your EMR clusters.

  • If you already have CloudWatch agent setup in your environment, then you must add metrics and dimensions to the CloudWatch agent configuration file.

  • If you do not have the CloudWatch agent setup, then you can run the Unravel bootstrap script, which will set up the CloudWatch agent in your environment.

  • If CloudWatch agent setup is available in your environment

    Add the following metrics and dimensions to the CloudWatch agent configuration file:

    {
  "agent": {
    "metrics_collection_interval": 60
  },
  "metrics": {
    "append_dimensions": {
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}",
      "InstanceName": "${aws:InstanceName}"
    },
    "metrics_collected": {
      "mem": {
        "measurement": [
          "mem_used_percent"
        ],
        "metrics_collection_interval": 60
      },
      "cpu": {
        "measurement": [
          "cpu_usage_idle",
          "cpu_usage_iowait",
          "cpu_usage_user",
          "cpu_usage_system"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "*"
        ],
        "totalcpu": False
      }
    }
  }
}

  • If CloudWatch agent setup is NOT available in your environment

    CloudWatch agent is packaged as part of the Unravel bootstrap script so to enable the setup of the CloudWatch agent, you must add the following optional argument within the bootstrap action:

    1. Edit the bootstrap action and provide the following location to the bootstrap script:

      s3://unraveldatarepo/unravel_emr_bootstrap.py

    2. In the Optional arguments text box, add the following arguments as shown in the image:

      -cwa or --cloud-watch-agent
      emr-cloudwatch-newusers-bootstrap.png

      Unravel bootstrap script will automatically deploy the CloudWatch Agent to all the EMR cluster nodes.

In this section: