Skip to main content

Home

Setting up Amazon RDS (optional)

Unravel's default installation uses a bundled database for part of its storage. For better performance and ease of management, we recommend replacing the bundled database with an Amazon RDS instance.

1. Create an RDS security group, subnet group, and parameter group

  1. Create an RDS security group on the VPC of Unravel Server and allow access from Unravel Server security group.

  2. Create a new database subnet group.

    A new database subnet group is required for "multiple availability zone" (multi-AZ) deployment. The VPC should at least contains two subnets in at least two availability zones in a given region, in the same VPC. For more information, see AWS documentation.

    aws-emr-subnet.png

  3. Create a new database parameter group with custom settings. Choose a MySQL version compatible with Unravel.

    aws-emr-parameters-grp.png
    Custom database parameters
     key_buffer_size = 268435456 
 max_allowed_packet = 33554432 
 table_open_cache = 256 
 read_buffer_size = 262144 
 read_rnd_buffer_size = 4194304 
 max_connect_errors=2000000000 
 net-read-timeout = 300 
 net-write-timeout = 600 
 open_files_limit=9000 
 innodb_open_files=9000 
 character_set_server=utf8 
 collation_server = utf8_unicode_ci 
 innodb_autoextend_increment=100 
 innodb_additional_mem_pool_size = 20971520 
 innodb_log_file_size = 134217728 
 innodb_log_buffer_size = 33554432 
 innodb_flush_log_at_trx_commit = 2 
 innodb_lock_wait_timeout = 50
    Required database parameters
    Database name: unravel_mysql_prod 
Port: 3306 
DB parameter group: unravel

    Caution

    It is not possible to modify a parameter group that is associated with a running RDS instance. To modify a parameter group that is in use, you can do either one of the following actions:

    • Create a copy of your desired parameter group, make the required modifications, and assign this new parameter group to your RDS instance.

    • Stop the RDS instance associated with your parameter group, make the change, and start the RDS instance again.

2. Create an Amazon RDS instance

  1. In the Amazon RDS portal, click Create database.

    aws-emr-rds.png

  2. Select MySQL and click Next.

  3. Select Production - MySQL.

    aws-emr-chose-case.png

  4. Change the following properties, and leave all others with default values.

    • License model: generic-public-license

    • DB engine version: 5.5.46

    • DB instance class: db.r3.xlarge (vCPU, 30.5 GiB RAM)

    • Multi-AZ deployment: Create replica in different zone

    • Storage type: Provisioned IOPS (SSD)

    • Allocated storage: 500GB (or more depending on number of jobs and clusters the unravel node will monitor)

    • Provisioned IOPS: 1000

    aws-emr-instance-spec.png

  5. Specify the database instance ID, username, and password.

    • DB Instance identifier: unravel_mysql_prod

    • Master username: unravel

    • Master password: Change_Password

    aws-emr-settings.png

  6. Click Next.

  7. In the Advanced Settings page change the following settings.

    You can leave all other settings with default values or specify values suitable to your requirements.

    Network & Security Settings

    • Virtual Private Cloud: Select the VPC that contains minimally two subsets and on the same region that you plan to deploy Unravel and the EMR cluster.

    • Subnet group: Select the new database subnet group you created, named unravel.

    • Public accessibility: No

    • Availability zone : No Preference

    • VPC security group: Select the new VPC security group you created.

    aws-emr-vpc.png

  8. Click Create database.

    You should see the following message.

    aws-emr-instance-created.png
3. Connect Unravel to the RDS instance

By default, the security group created for the unravel RDS has no network access granted on port 3306 on the subnet connected. You must modify the security group applied on Unravel RDS.

  1. In the Amazon RDS dashboard, locate the MySQL database endpoint.

    aws-emr-connect-endpoint.png

  2. Look for the security group used for the Unravel RDS instance.

    aws-emr-security-group.png

  3. Modify the inbound rule of the security group by adding a new rule to allow connections from either:

    • The Unravel EC2 instance's security group.

    • The subnet IP block in which the Unravel EC2 instance located.

    Either the security group or IP block works, provided the RDS instance is located on the same region as the VPC.

    aws-emr-security-group-inbound.png

  4. On Unravel Server, verify the MySQL connection.

    For example,

    /usr/local/unravel/mysql/bin/mysql -h unravelmysqlprod.csfw1hkmlpgh.us-east-1.rds.amazonaws.com -u unravel -p
    aws-emr-cli-mysql.jpg

  5. Verify that the database unravel_mysql_prod has been created; if not create it.

    CREATE DATABASE IF NOT EXISTS unravel_mysql_prod;
4. Create a schema for the RDS instance

  1. Stop Unravel Server.

    sudo /etc/init.d/unravel_all.sh stop

  2. Set or add the following properties in /usr/local/unravel/etc/unravel.properties so that Unravel Server knows about the database.

    If the property isn't found, add it. Use the actual values you set in the steps above.

    You can use a hostname; but to avoid DNS lookups use an IP address.

    The database password can be encrypted.

    unravel.jdbc.username=unravel
unravel.jdbc.password=unraveldata
unravel.jdbc.url=jdbc:mysql://unravelmysqlprod.csfw1hkmlpgh.us-east-1.rds.amazonaws.com:3306/unravel_mysql_prod

  3. Ensure the schema is up to date using the schema upgrade utility provided by Unravel Server. The script step connects to the database and applies schema deltas, in order, until the schema is up to date. The success or failure of the update is noted.

    sudo /usr/local/unravel/dbin/db_schema_upgrade.sh

    Note

    If table creation privilege is not granted because an internal DBA support group provides the external database, request that they apply the schemas in /usr/local/unravel/sql/mysql/ in numerical order. The schema deltas assume the database name is already picked with a 'use' statement. The schema_migrations table keeps track of what schemas have been applied.

  4. Create the default user admin with the SQL statement emitted by

    /usr/local/unravel/install_bin/db_initial_inserts.sh | /usr/local/unravel/install_bin/db_access.sh
5. Start Unravel Server

  1. Disable the bundled database on Unravel Server. Only one of these commands is needed, depending on your exact version of Unravel. The unnecessary command produces an error that you can ignore.

    sudo chkconfig unravel_db off
sudo chkconfig unravel_pg off

  2. Start Unravel services.

    sudo /etc/init.d/unravel_all.sh start
In this section: