Setting up VPC peering (optional)
Note
Follow these steps only if you have an Amazon EMR cluster located in a different VPC than the Unravel EC2 instance.
This topic explains how to resolve connectivity issues when you have an Amazon EMR cluster located in a different VPC than your Unravel EC2 instance or your Unravel EC2 instance connects to an RDS instance created on a different VPC in the same region.
The VPC where Unravel EC2 is located is in the same region as the EMR cluster.
The subnet used by Unravel EC2 does not overlap the IP block range of the subnet used in the EMR cluster.
Network ACLs on both the VPC for Unravel EC2 and the EMR cluster is the default and allow all traffic. The security group is the only security enforcement on network access.
The examples below show both the Unravel EC2 instance and the EMR cluster in the us-east-1 region but configured with different VPC and subnet. There is no network access allowed between Unravel EC2 and EMR cluster by default.
Resources | Internal IP address | Subnet ID | Subnet IP block | VPC ID (name) | IP block in VPC | Security group ID (name) |
|---|---|---|---|---|---|---|
Unravel EC2 node | 10.10.0.7 | subnet-03b82c56b2c26dbd1 | 10.10.0.0/24 | vpc-0b0e17b01c4a3b54a (Unravel_VPC) | 10.10.0.0/16 | sg-0e0a03084398287c9 (Unravel-EC2_SG) |
EMR cluster master node | 10.11.0.53 | subnet-0294cc17a42a9acfd | 10.11.0.0/24 | vpc-c3d079a4 (VPC_for_VPC Peering) | 10.11.0.0/16 | sg-0a73c3aea9340ae49 (EMR_VPC_SG) |
EMR cluster core nodes | 10.11.0.76 10.11.0.130 | subnet-0294cc17a42a9acfd | 10.11.0.0/24 | vpc-c3d079a4 (VPC_for_VPC Peering) | 10.11.0.0/16 | sg-0a73c3aea9340ae49 (EMR_VPC_SG) |
1. Create VPC peering in VPC dashboard
From the AWS console | VPC | Peering Connections, click Create Peering Connection.
Enter the name tag.
For example,
EMR_to_Unravel.In the VPC (Requester) field, select the VPC of the EMR cluster.
In the VPC (Accepter) field, select the VPC of Unravel Server.
Click Create Peering Connection.
A success message should appear in the screen.
Click OK.
2. Accept the VPC peering request
In the VPC Dashboard, the new VPC peering connection has the status Pending Acceptance.
Select this connection, click Action, and select Accept Request.
Click Yes Accept in the prompt screen. You will see a message regarding "Modify my route tables". Click Close.
3. Create routes between peered VPC
To create the routes between peered VPCs (Unravel Server on Unravel_VPC and the EMR cluster on Test_EMR_VPC):
Go to VPC Dashboard | Route Tables.
Use the VPC name to locate the route tables for the two peered VPCs and then for each route table add another route.
Repeat Step 3 to Step 13 for each route table. Refer to Updating your Route tables for a VPC peering connection for more details.
Click Edit | Add another route.
Find the Unravel_VPC route table.
In the Destination field, enter the IP block of the EMR VPC.
For example,
10.11.0.0/16
In the Target field, select or enter the VPC peer connection ID.
For example, pcx-0a57a978ef9a525e2.
Click Save .
Find the Test_EMR_VPC route table.
Set the Destination to the IP block of Unravel_VPC.
For example,
10.10.0.0/16.In the Target field, select the VPC peer connection ID.
For example, pcx-0a57a978ef9a525e2.
Click Save.
In the Target field, select the connection ID.
For example, pcx-0a57a978ef9a525e2.
Click Save.
4. Update security groups
Go to VPC Dashboard | Security Group.
Click Add another rule.
Repeat step-2 to step-7 for each of the security group. Refer to Updating your security groups to reference peer VPC groups for more details.
Set Type to inbound ALL traffic and Protocol to ALL.
Locate the security group used on Unravel EC2 node. Enter the EMR VPC IP block in the Source field.
For example,
10.11.0.0/16.
Click Save.
Locate the security group used on EMR cluster node and enter the Unravel VPC IP block.
For example,
10.10.0.0/16.Click Save.
5. Verify the connection between Unravel and the EMR master node
Open SSH sessions to both Unravel EC2 node and EMR master node.
Since the above example allows all traffic from both VPC IP blocks, you should be able to ping the IP address of EMR master node from Unravel Server.
On the Unravel EC2 instance, open a telnet session to the EMR master node port 8082 (the namenode port).
On the EMR master node, open telnet sessions to the Unravel EC2 instance, ports 3000 and 4043.
If telnet port tests are positive, the VPC peering connection is set up correctly. If not, troubleshoot the configuration on network ACL, security groups, and route tables used on both VPCs.