Skip to main content

Home

Detect anomalies in the cost of Databricks job runs

You can use this API to detect whether a particular Databricks job is of high cost or low cost compared to the other job runs. You can pass the job ID and identify cost anomalies for a Databricks job.

Specify only one job ID at a time for which the Run Count is greater than 1. Log in to the Unravel UI to get the Job ID and Run Count value. (Navigate to the Jobs tab inside the Jobs Page).

Request

The request produces an application/json payload.

GET http://<unravel-host>/api/v1/app_store/api/v1/anomalies/jobs/<databricks_jobid>/cost

Example: http://<unravel_host>/api/v1/app_store/api/v1/anomalies/jobs/272404321223542/cost

Path parameters

None

Query parameters

All the parameters are optional.

Name

Type

Description

from

to

string

The date-time filter. Specify the from and to date and time values to find anomalies for a particular date and time range when the jobs are run. Specify the values in the ISO date-time format as follows:

Format: %Y-%m-%dT%H:%M:%S.%fZ (date with timestamp) or %Y-%m-%d (date with no timestamp)

Example: 2022-03-07T14:10:26:054Z or 2022-03-07

Default value: None

Note

An exception occurs if the from and to values are not specified in the ISO date-time format.

tolerance

integer

Controls the anomaly boundaries—the lower the tolerance, the higher the anomalies.

Default value: 0.95

Valid values: 0 to 1

lowerBoundary

upperBoundary

float

The lowerBoundary and upperBoundary are the user-defined parameters to control the anomaly range. You must apply these parameters as additional filters on the anomalies returned by the algorithm.

For example, if the algorithm returns $0.5 and $2 costs as anomalies, and if you do not want to see these anomalies, then define a upperBoundary greater than $2 so that the response marks these as not anomalies.

Examples:

Requirement

Values

Result

If you do not want to view anomalies between 10 and 100 dollars

Specify lowerBoundary=10 and upperBoundary=100,

All values between 10 and 100 are returned as not anomalies.

If you do not want to view anomalies greater than five dollars

Specify lowerBoundary=5 and upperBoundary=None

All values greater than five are returned as not anomalies.

If you do not want to view anomalies lesser than 100 dollars

Specify upperBoundary=100 and lowerBoundary=None

All values lesser than 100 are returned as not anomalies.

If you want to view all anomalies

Specify lowerBoundary=None and upperBoundary=None

Returns all anomalies reported by the algorithm.

Default: None

topNAnomalies

integer

Displays the highest number of anomalies based on the significance (score). N indicates any number you want to specify to return the top anomalies.

Note

If you have specified topNAnomalies=10 and if the algorithm returns less than ten anomalies, this parameter is ignored.

Default: None

returnAnomaliesOnly

boolean

  • True: Returns data that has only anomalies and not all the data points.

  • False: Returns all data points.

Default: False

suppressNegativeAnomalies

boolean

  • True: Suppresses negative anomalies. These negative anomalies are not returned in the output.

  • False: Returns all anomalies.

Default: False

Examples

Request 

curl -X GET -H "Authorization: JWT <token>" -H "Content-Type:application/json" -H "Accept: application/json" http://<unravel-host>/api/v1/app_store/api/v1/anomalies/jobs/<databricks_jobid>/cost

Response

[
    {
        "value": 50.33,
        "anomaly": 0,
        "datetime": "2022-02-22T03:58:15.944000Z",
        "lowerBoundary": -151.5689863110632,
        "upperBoundary": 309.6075651211625,
        "score": 0.0
    },
    {
        "value": 51.12,
        "anomaly": 0,
        "datetime": "2022-02-22T14:23:44.369000Z",
        "lowerBoundary": -196.4820692181124,
        "upperBoundary": 274.988031900136,
        "score": 0.0
    },
    {
        "value": 57.06,
        "anomaly": 0,
        "datetime": "2022-02-22T15:25:51.078000Z",
        "lowerBoundary": -179.66430889182902,
        "upperBoundary": 285.0467646923019,
        "score": 0.0
    },
    {
        "value": 69.71,
        "anomaly": 0,
        "datetime": "2022-02-22T18:58:03.523000Z",
        "lowerBoundary": -166.13682141126742,
        "upperBoundary": 309.65327527682035,
        "score": 0.0
    },
    {
        "value": 32.76,
        "anomaly": 0,
        "datetime": "2022-02-23T20:49:45.484000Z",
        "lowerBoundary": -195.8326199493043,
        "upperBoundary": 267.5704782275939,
        "score": 0.0
    },
    {
        "value": 47.54,
        "anomaly": 0,
        "datetime": "2022-03-04T15:34:48.189000Z",
        "lowerBoundary": -225.808921942789,
        "upperBoundary": 248.08816652995216,
        "score": 0.0
    },
    {
        "value": 50.69,
        "anomaly": 0,
        "datetime": "2022-03-04T15:57:40.994000Z",
        "lowerBoundary": -156.7418278886897,
        "upperBoundary": 303.3146308606596,
        "score": 0.0
    },
    {
        "value": 41.12,
        "anomaly": 0,
        "datetime": "2022-03-05T01:27:51.496000Z",
        "lowerBoundary": -176.68871413153553,
        "upperBoundary": 271.1775045154612,
        "score": 0.0
    },
    {
        "value": 65.81,
        "anomaly": 0,
        "datetime": "2022-03-07T14:10:26.054000Z",
        "lowerBoundary": -145.4064201546787,
        "upperBoundary": 294.55383723341055,
        "score": 0.0
    },
    {
        "value": 62.19,
        "anomaly": 0,
        "datetime": "2022-03-07T18:58:27.724000Z",
        "lowerBoundary": -185.20942550542986,
        "upperBoundary": 301.62357198114006,
        "score": 0.0
    },
    {
        "value": 72.33,
        "anomaly": 0,
        "datetime": "2022-03-07T21:51:45.795000Z",
        "lowerBoundary": -143.64130130498756,
        "upperBoundary": 307.1662054911512,
        "score": 0.0
    },
    {
        "value": 76.9,
        "anomaly": 0,
        "datetime": "2022-03-08T01:27:10.392000Z",
        "lowerBoundary": -163.7108291889344,
        "upperBoundary": 305.01130817100415,
        "score": 0.0
    },
    {
        "value": 3.74,
        "anomaly": -1,
        "datetime": "2022-03-08T02:59:06.260000Z",
        "lowerBoundary": 209.8489573150019,
        "upperBoundary": 681.1699932092157,
        "score": 0.9821776574549002
    },
    {
        "value": 892.88,
        "anomaly": 1,
        "datetime": "2022-03-08T03:04:06.170000Z",
        "lowerBoundary": 193.72941058005716,
        "upperBoundary": 685.9399035641256,
        "score": 0.3016883773062612
    },
    {
        "value": 45.68,
        "anomaly": 0,
        "datetime": "2022-03-08T14:43:58.895000Z",
        "lowerBoundary": -157.15185634049396,
        "upperBoundary": 300.7481991778867,
        "score": 0.0
    }
]
In this section: