Configuring CloudWatch agent
You can configure the CloudWatch agent for Unravel monitoring of your EMR clusters.
If you already have CloudWatch agent setup in your environment, then you must add metrics and dimensions to the CloudWatch agent configuration file.
If you do not have the CloudWatch agent setup, then you can run the Unravel bootstrap script, which will set up the CloudWatch agent in your environment.
If CloudWatch agent setup is available in your environment
Add the following metrics and dimensions to the CloudWatch agent configuration file:
{ "agent": { "metrics_collection_interval": 60 }, "metrics": { "append_dimensions": { "AutoScalingGroupName": "${aws:AutoScalingGroupName}", "ImageId": "${aws:ImageId}", "InstanceId": "${aws:InstanceId}", "InstanceType": "${aws:InstanceType}", "InstanceName": "${aws:InstanceName}" }, "metrics_collected": { "mem": { "measurement": [ "mem_used_percent" ], "metrics_collection_interval": 60 }, "cpu": { "measurement": [ "cpu_usage_idle", "cpu_usage_iowait", "cpu_usage_user", "cpu_usage_system" ], "metrics_collection_interval": 60, "resources": [ "*" ], "totalcpu": False } } } }
If CloudWatch agent setup is NOT available in your environment
CloudWatch agent is packaged as part of the Unravel bootstrap script so to enable the setup of the CloudWatch agent, you must add the following optional argument within the bootstrap action:
Edit the bootstrap action and provide the following location to the bootstrap script:
s3://unraveldatarepo/unravel_emr_bootstrap.py
In the Optional arguments text box, add the following arguments as shown in the image:
-cwa or --cloud-watch-agent
Unravel bootstrap script will automatically deploy the CloudWatch Agent to all the EMR cluster nodes.