Skip to main content

Home

Enabling multiple daemon workers for high-volume data

Tip

These instructions apply to single-host Unravel deployments; contact Unravel Support for multi-host deployments.

Note

Run the <Unravel/installation/directory>/manager config worker --help to view the commands menu for scaling up daemons for high-volume data.

You can increase the workers for the following daemons only:

  • bigquery_worker

  • event_worker

  • impala_worker

  • insights_worker

  • metrics_analyzer

  • spark_worker

  • task_worker

  • unravel_sensor

  • yarn_jc_worker

  1. Stop Unravel.

    <Unravel/installation/directory>/manager stop

  2. Check the list of workers that are already customized.

    <Unravel/installation/directory>/manager config worker show

  3. Set the number of processes for a service type.

    <Unravel/installation/directory>/manager config worker count SERVICE_TYPE NUMBER

    In the following example, you add more event workers by setting four processes that are event_worker_1 to event_worker_4:

    <Unravel/installation/directory>/manager config worker count event_worker 4

  4. Apply the changes.

    <Unravel/installation/directory>/manager config apply

  5. Start Unravel.

    <Unravel/installation/directory>/manager start

Note

A bug can prevent you from setting the number of workers for the metrics_analyzer daemon. Only in such a case, stop Unravel and manually increase the number of workers from <Unravel installation directory>/unravel/data/conf/unravel.yaml file. In the following example, the unravel.yaml is edited to increase the number of workers for the metrics_analyzer daemon to 4. 

services:
  metrics_analyzer:
    instance_count: 4

Ensure to apply the changes and restart Unravel. 

<Unravel/installation/directory>/manager config apply
<Unravel/installation/directory>/manager start
In this section: