Airflow
Property/Description | Set by user | Unit | Default |
|---|---|---|---|
com.unraveldata.airflow.available Notes if the airflow is currently available.
| boolean | false | |
airflow.look.back.num.days The number of days to look back. The look-back days can be specified as a positive or negative number. For instance -5 or 5 sets the look back number of days to 5. | count | 1 | |
airflow.look.back.num.hours The look-back time window in hours, which can be either a positive or negative integer. If present, it takes precedence over airflow.look.back.num.days to have finer granularity. Suggested value for large clusters: 2 | count | 24 | |
com.unraveldata.airflow.http.max.body.size.byte Set maximum number of bytes Unravel fetches data from Airflow Web UI. Default unlimited. | bytes | 0 | |
com.unraveldata.airflow.login.name Airflow UI login username. You must set this if airflow.server.url = https. | Required | string | - |
com.unraveldata.airflow.login.password Password for Airflow UI com.unraveldata.airflow.login.name. You must set set this if airflow.login.name is set. | Required | string | - |
com.unraveldata.airflow.protocol Type of connection, e.g., HTTPS or HTTP. You must set the airflow.login.name and airflow.login.password when this value is https. | https | ||
com.unraveldata.airflow.server.url Full URL of the airflow server, starting with | url | http://localhost:10080 | |
com.unraveldata.airflow.status.timeout.sec Set Airflow workflow status timeout in Unravel. | sec | 3600 | |
com.unraveldata.airflow.task.log.parsing.enabled Controls whether to parse the Airflow Task logs. These logs are used to populate the Workflow Instance entities in the "Jobs - Workflows" page. | boolean | true | |
com.unraveldata.airflow.task.log.parsing.operators Controls the Task logs to parse based on the Operator that produced it. Since Unravel only derives insights for Hive, Spark, and MR applications, it is set to only analyze operators that can launch those types of jobs. The values are delimited using a "," and it treats "*" as a wildcard to many any or no characters. Any special characters like "\" or "." will be removed. | string | BashOperator, PythonOperator, *Hive*, *Spark* | |
com.unraveldata.airflow.task.thread.pool.size Controls whether to process the Airflow Task logs sequentially or in parallel. Process the logs in parallel improves performance. This config takes effect only if airflow.task.log.parsing.enabled=true. Possible values are 1-16. If one, logs are processed sequentially. If greater than one, logs are processed in parallel using a Thread Pool with of the size specified value. | count | 1 |