Download unravel-emr-ansible.zip:

curl https://s3.amazonaws.com/unraveldatarepo/unravel-emr-ansible.zip --output unravel-emr-ansible.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11708  100 11708    0     0  66541      0 --:--:-- --:--:-- --:--:-- 66902

Unzip unravel-emr-ansible.zip:

Archive:  unravel-emr-ansible.zip
  inflating: unravel_emr_ansible/emr_ansible_inventory
  inflating: unravel_emr_ansible/unravel_emr_bootstrap.py
  inflating: unravel_emr_ansible/prepare_inventory.py
  inflating: unravel_emr_ansible/README.md
  inflating: unravel_emr_ansible/emr_ansible_playbook.yaml

Run prepare_inventory.py:

Enter the following values either interactively at the prompts or through their command-line options:

For example,

python prepare_inventory.py 
Please Enter Unravel host IP: 172.31.62.27
Please Enter ssh key path: /home/hadoop/id_rsa

Ansible Inventory updated

sudo python3 -m pip install --upgrade pip

Install Ansible on the EMR master node:

sudo pip install ansible

(Optional) Determine what directory Ansible was installed in, and add that directory to the $PATH variable in ~/.bashrc, if it isn't there already.

which ansible
/usr/local/bin/ansible

In ~/.bashrc, update this line:

export PATH=/usr/local/bin/:$PATH

Run the Unravel Ansible playbook:

$ cd unravel_emr_ansible
$ ANSIBLE_HOST_KEY_CHECKING=false
$ ansible-playbook -i emr_ansible_inventory emr_ansible_playbook.yaml
 
PLAY [nodes] *******************************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [172.31.109.7]
ok: [172.31.109.251]
ok: [172.31.97.203]
 
TASK [Run emr bootstrap script] ************************************************
changed: [172.31.109.7]
changed: [172.31.109.251]
changed: [172.31.97.203]
 
TASK [Check Unravel sensor version] ********************************************
changed: [172.31.109.7]
changed: [172.31.109.251]
changed: [172.31.97.203]
 
TASK [Print sensor version] ****************************************************
ok: [172.31.109.7] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
ok: [172.31.109.251] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
ok: [172.31.97.203] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
 
PLAY RECAP *********************************************************************
172.31.109.251             : ok=4    changed=2    unreachable=0    failed=0  
172.31.109.7               : ok=4    changed=2    unreachable=0    failed=0  
172.31.97.203              : ok=4    changed=2    unreachable=0    failed=0

Set up AWS CLI.

Make sure AWS CLI has permission to list EMR clusters.

aws emr list-instances --cluster-id cluster id

Also, ensure that the region stored in ~/.aws/config must match the region of the EMR cluster.

Download unravel-emr-ansible.zip:

wget https://s3.amazonaws.com/unraveldatarepo/unravel-emr-ansible.zip

Unzip unravel-emr-ansible.zip:

unzip unravel-emr-ansible.zip 
Archive:  unravel-emr-ansible.zip
  inflating: unravel-emr-ansible/README.md  
  inflating: unravel-emr-ansible/emr_ansible_inventory  
  inflating: unravel-emr-ansible/emr_ansible_playbook.yaml  
  inflating: unravel-emr-ansible/prepare_inventory.py  
  inflating: unravel-emr-ansible/unravel_emr_bootstrap.py

Run prepare_inventory.py:

Enter the following values either interactively at the prompts or through their command-line options:

For example,

python prepare_inventory.py --use-public
Please Enter Unravel host IP: 172.31.62.27
Please Enter ssh key path: /home/hadoop/id_rsa

Ansible Inventory updated

Install Ansible:

sudo pip install ansible

Run the Unravel Ansible playbook:

$ cd unravel-emr-ansible
$ ANSIBLE_HOST_KEY_CHECKING=false
$ ansible-playbook -i emr_ansible_inventory emr_ansible_playbook.yaml
 
PLAY [nodes] *******************************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [18.61.10.17]
ok: [18.61.10.21]
ok: [18.61.10.20]
 
TASK [Run emr bootstrap script] ************************************************
changed: [18.61.10.17]
changed: [18.61.10.21]
changed: [18.61.10.20]
 
TASK [Check Unravel sensor version] ********************************************
changed: [18.61.10.17]
changed: [18.61.10.21]
changed: [18.61.10.20]
 
TASK [Print sensor version] ****************************************************
ok: [18.61.10.17] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
ok: [18.61.10.21] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
ok: [18.61.10.20] => {
    "msg": "Unravel Version: 4.4.3.0b0005"
}
 
PLAY RECAP *********************************************************************
18.61.10.21             : ok=4    changed=2    unreachable=0    failed=0  
18.61.10.17             : ok=4    changed=2    unreachable=0    failed=0  
18.61.10.20             : ok=4    changed=2    unreachable=0    failed=0

After you connect the Unravel EC2 instance to your EMR cluster, run some jobs on the EMR cluster and monitor the information displayed in Unravel UI (http://unravel_ec2_node_public_IP:3000).

Set additional configurations if required.

Optionally, you can run healthcheck, at this point, to verify that all the configurations and services are running successfully.

<unravel_installation_directory>/unravel/manager healthcheck

Healthcheck is run automatically on an hourly basis in the backend. You can set the healthcheck intervals and email alerts to receive the healthcheck reports.