BigQuery setup guide

Home

BigQuery setup guide

The following table presents information about the total number of projects and billing accounts that can be integrated with Unravel Pro (BigQuery) for monitoring in Unravel SaaS.

Items	No of projects
Monitoring projects	100 projects
Admin project	1
Billing account	1

Prerequisites for BigQuery SaaS setup

The integration of GCP projects with BigQuery consists of the following significant aspects:

Exporting the GCP billing data of the projects that you want to integrate with Unravel
Creating resources on GCP that are required for the integration.

This topic provides instructions on creating resources on GCP and setting the export of billing data, which are the prerequisites for integrating Unravel.

Setting exporting of the GCP billing data for Unravel

You need the following to set the exporting of the GCP billing data for Unravel:

Either the Billing Accounts Costs Manager role or the Billing Account Administrator role on the target Cloud Billing account.
BigQuery user role for the project in which you want to export the billing data.

Run the following steps to set up the exporting of the GCP billing data for the projects monitored by Unravel.

From the GCP console, choose a project to export the GCP billing data.
Note
Ensure this is also the same project that you want Unravel to monitor.
Export the GCP billing data of the billing account you want to integrate with Unravel to the chosen project. Do the following to export the billing data:
1. Search for the Billing Exports page and select the Cloud billing account you want to integrate with Unravel.
2. In the Detailed Usage Cost section, click the Edit Settings button and choose the project you selected to export the GCP billing data.
3. From the Dataset ID drop-down, select a dataset to export the billing data or create a new dataset.
4. Confirm that the project and the dataset in which you want to export the billing data are correct, and then click Save.
5. In the Billing export page > Detailed usage cost section, check and ensure that the Detailed usage cost is shown as Enabled and your selected project ID and dataset are displayed.
The table creation process in the chosen dataset will take a few minutes.
Note the following billing export details and keep them handy:
- Dataset ID
- Table ID
- Monitoring project IDs
- Admin project ID
  Note
  The administrator project manages the BigQuery Reservations resources and is the primary billing source for these resources. This project need not be the same project with BigQuery jobs. Google recommends creating a dedicated project for Reservations resources.

Creating GCP resources for projects monitored by Unravel

Terraform scripts are used to facilitate the smooth integration of Unravel with BigQuery projects. These scripts help you set up various components and resources.

Note

Unravel recommends creating GCP resources for integration using the Terraform script. You can fork the repository using a GCP cloud storage bucket to store the Terraform state and push changes to the fork every time you do the integration.

The following IAM permissions are required for creating the GCP resources. Whoever executes the Terraform script for creating GCP resources for integration must have these permissions.

iam.roles.create	iam.serviceAccountKeys.create	logging.sinks.create	pubsub.subscriptions.create	resourcemanager.projects.get
iam.roles.delete	iam.serviceAccountKeys.delete	logging.sinks.delete	pubsub.subscriptions.delete	resourcemanager.projects.getIamPolicy
iam.roles.get	iam.serviceAccountKeys.get	logging.sinks.get	pubsub.subscriptions.get	resourcemanager.projects.setIamPolicy
iam.roles.undelete	iam.serviceAccounts.create		pubsub.topics.attachSubscription
iam.roles.update	iam.serviceAccounts.delete		pubsub.topics.create
	iam.serviceAccounts.get		pubsub.topics.delete
	iam.serviceAccounts.getIamPolicy		pubsub.topics.get
	iam.serviceAccounts.getIamPolicy		pubsub.topics.get
	iam.serviceAccounts.setIamPolicy		pubsub.topics.getIamPolicy
			pubsub.topics.setIamPolicy

Follow the instructions to create GCP resources using Terraform scripts for projects that you want to be integrated with Unravel for monitoring:

Use a terminal and clone the following repository:
https://github.com/unraveldata-org/unravel-terraform-scripts/tree/release/4.8.2.1/
Go to unravel-terraform-scripts/saas/bigquery folder.
Duplicate the example input file named input.tfvars.example. Rename it to input.tfvars.
```
cp input.tfvars.example input.tfvars
```
Configure Terraform backend. Unravel recommends configuring a central storage for your Terraform state file.
1. Copy the backend.tf.example file using this command:
```
cp backend.tf.example backend.tf
```
2. Update the Google Storage Path, to which the user who runs Terraform has access, in the backend.tf file.
Using a VM editor, update the following in the input.tfvars file:
- Project IDs
  Enter the new project IDs that you had specified for Setting exporting of the GCP billing data for Unravel in the monitoring_projects_list variable. You can specify up to 100 projects.
- Administrator project ID
  Enter your administrator project ID in the admin_project_id variable.
- Service account
  Enter the service account that you received in the email from Unravel in the service_account variable.
  Tip
  The service account is also available from the Unravel UI > Manage icon > BigQuery configuration, which you can copy.
Navigate to the Terraform directory and run the following commands to log in to the gcloud application. A user with the essential IAM permissions can only log on to the gcloud application.
```
cd unravel-terraform-scripts/saas/bigquery
gcloud init
gcloud auth-application-default login
```
After the login, run the following commands:
```
terraform init
terraform plan --var-file=input.tfvars
terraform apply --var-file=input.tfvars
```
The GCP resources are created. You can now login to the Unravel BigQuery account using the login credentials that you have received in the email.

Integrating Unravel with BigQuery projects

Do the following to integrate BigQuery with Unravel:

After your Unravel account is set, you will receive an email with the following details. Keep these handy to access your Unravel UI and integrate the BigQuery projects.
- URL to access Unravel (BigQuery) SaaS
- Login credentials
- Service account
Run the following steps to integrate your BigQuery projects for Unravel monitoring:
1. Complete the prerequisites to create GCP resources and set up the exporting of billing data.
2. Click the URL that you have received in the email and access the Unravel UI.
3. Enter the login credentials received by email on the login page. The Unravel UI is displayed.
4. From the top right, click and select BigQuery configuration. The BigQuery account setting page is displayed.
5. Specify the following details:
  - In the Monitored projects section, specify the Project IDs you have integrated using Terraform.
  - In the Administrator Projects section, specify the admin project ID.
  - In the Google Cloud Billing Export Data section, specify the Project ID, Dataset ID, and Table ID. Refer to Prerequisites.
6. Click the Save button. The BigQuery projects are successfully integrated, and you can monitor these projects from the Unravel UI. In case any errors are shown, you must fix it and click the Save button again.