Choosing a polling and authentication method to install (GCP) BigQuery
Before you install Unravel for BigQuery, you must evaluate and determine an appropriate method for polling and authentication that aligns with your organization's needs.
In Unravel, the jobs, project, billing, reservation, and job timeline data are fetched from BigQuery for analytical and monitoring purposes. For all these types of data, Unravel uses the API polling (pull/push) method. However, for fetching the jobs data, you can choose one of the following polling methods:
INFORMATION_SCHEMA method
API Push/Pull method
Note
To convert from one polling method to another, Contact Unravel Support team.
BigQuery includes INFORMATION_SCHEMA, which contains read-only, system-defined views that provide metadata details about different BigQuery objects. Using a SELECT query, Unravel can access this metadata information and use it for the purpose of monitoring or generating insights.
Refer to the BigQuery installation for the complete steps provided for each authentication method.
The following table provides details of the pros and cons of using this polling method.
Pros | Cons |
|---|---|
INFORMATION_SCHEMA based jobs polling is faster than the API Pull/Push method. | INFORMATION_SCHEMA based polling method incurs some minimal cost. |
You can access past job metadata for up to 180 days. The API Pull/Push method does not support this capability. | The Location property should be updated every time a new location is configured in GCP. |
Does not require the creation of GCP resources like pub/sub topic, logs router, and subscription since audit logs are not required. | Insights on scheduled queries are not supported. |
In this method, API requests are used to fetch the job data. The following table provides details of the pros and cons of using this polling method.
Pros | Consrn-4820 |
|---|---|
API is almost free and incurs negligible cost | In cases of large job workloads, APIs start throttling. Thereby increasing latency in getting job data to Unravel UI. |
You can install Unravel for GCP BigQuery and set Unravel to receive data from BigQuery using one of the following authentication methods:
VM based authentication | Single key-based authentication | Multiple key-based authentication |
|---|---|---|
 |  |  |
This is the recommended and more secure way to connect Bigquery with Unravel. Instead of downloading the service account key, you simply attach the service account to the Unravel VM, which works well when the VM is on the GCP compute engine service and hosted in the same project as Unravel. | Single key-based authentication In this authentication method, the Unravel VM is located outside the GCP environment. You use a single service account key that covers all the projects that you download and use. This approach is suitable for Unravel SaaS or when the VM is hosted in a different cloud or on-premises environment. | Multiple key-based authentication In this authentication method, you can host the Unravel VM on either the GCP environment or any other cloud or on-premises setup. For each project integrated with Unravel, you simply download a single service account key. This is an ideal option when your VM is hosted in different cloud or on-premises environments. |