Skip to main content

Home

Advanced Spark worker configurations

You can set the following advanced configurations to fine-tune the Spark worker for efficient memory utilization.

Set Spark consumers

Unravel supports the processing of multiple records in parallel in a single Spark daemon. The number of Spark consumers defines how many records are processed simultaneously.

  1. Check the number of Spark workers that are configured. 

    /<unravel_installation_directory>/unravel/manager config worker show

  2. Check the status of memory. Ensure that you have sufficient memory before you set Spark consumers. 

    /<unravel_installation_directory>/unravel/manager config memory show

  3. Set the count for Spark consumers. In <count> specify the number of Spark consumers. 

    /<Unravel installation directory>/opt/unravel/manager config worker set spark_worker consumer_count <count>

    For example:

    /opt/unravel/manager config worker set spark_worker consumer_count 4

    Check again for the new memory requirements and adjust accordingly. 

    /<unravel_installation_directory>/unravel/manager config memory show

  4. Apply the changes and restart.

    <Unravel installation directory>/unravel/manager config apply --restart
Set applications to be stored in the cache

Caution

This is an advanced configuration that must be done in collaboration with Unravel support team only.

The number of applications that you want to be stored in the cache can be set to speed up the processing. However, this can affect memory requirements. The default number of applications that you can set is 10.

  1. Set the number of applications that you want to be stored in the cache. The default is 10: 

    <Unravel installation directory>/unravel/manager config worker set spark_worker max_concurrent_app <count>

    For example:

    /opt/unravel/manager config worker set spark_worker max_concurrent_app 11

  2. Apply the changes and restart.

    <Unravel installation directory>/unravel/manager config apply --restart
Set memory for Spark worker per consumer

You can set the memory of the Spark worker for each consumer. The default memory is 2048 MB

  1. Set the memory of the Spark worker for each consumer. <memory> can be specified in GB. 

    /<Unravel installation directory>/unravel/manager config worker set spark_worker memory_per_consumer <memory>

    For example:

    /opt/unravel/manager config worker set spark_worker memory_per_consumer 5

  2. Check and confirm the memory status.

    /<unravel_installation_directory>/unravel/manager config memory show

  3. Apply the changes and restart.

    <Unravel installation directory>/unravel/manager config apply --restart
In this section: