Skip to main content

BigQuery

BigQuery is a fully-managed, serverless data warehouse that enables scalable analysis over massive datasets. Boltic leverages BigQuery to store and fetch data, creating powerful pipes and streams. These pipes include actions such as source integration, database selection, and destination setup. Additionally, you can schedule pipes to run at fixed intervals or specific dates and times.

Getting Started​

Requirements​

To integrate BigQuery with Boltic, ensure you have the following:

  • Access to a BigQuery project with one or more datasets containing at least one table.

  • A Service Account Key (Private Key) for authentication.

  • A Google Service Account with the required permissions:

    BigQuery Permissions​

    PermissionDescription
    BIGQUERY_DATASETS_CREATEbigquery.datasets.create
    BIGQUERY_DATASETS_GETbigquery.datasets.get
    JOBS_CREATEbigquery.jobs.create
    TABLES_GETbigquery.tables.get
    TABLES_GET_DATAbigquery.tables.getData
    TABLES_CREATEbigquery.tables.create
    TABLES_UPDATE_DATAbigquery.tables.updateData
    TABLES_UPDATEbigquery.tables.update
    TABLES_UPDATE_DELETEbigquery.tables.delete

    Resource Manager Permissions​

    PermissionDescription
    RESOURCEMANAGER_PROJECTS_GETresourcemanager.projects.get

    Storage Permissions​

    PermissionDescription
    STORAGE_BUCKETS_CREATEstorage.buckets.create
    STORAGE_BUCKETS_GETstorage.buckets.get
    STORAGE_BUCKETS_LISTstorage.buckets.list
    STORAGE_BUCKETS_UPDATEstorage.buckets.update
    STORAGE_MULTIPARTUPLOADS_ABORTstorage.multipartUploads.abort
    STORAGE_MULTIPARTUPLOADS_CREATEstorage.multipartUploads.create
    STORAGE_MULTIPARTUPLOADS_LISTstorage.multipartUploads.list
    STORAGE_MULTIPARTUPLOADS_LISTPARTSstorage.multipartUploads.listParts
    STORAGE_OBJECTS_CREATEstorage.objects.create
    STORAGE_OBJECTS_GETstorage.objects.get
    STORAGE_OBJECTS_LISTstorage.objects.list
    STORAGE_OBJECTS_UPDATEstorage.objects.update

Service Account​

To enable Boltic to sync data from BigQuery, you need to configure a Service Account with the above permissions. We recommend creating a dedicated Service Account for Boltic to simplify permission management and auditing. If you already have a suitable Service Account, you can use that.

Creating a Service Account:

  1. Follow our guide on Creating a Service Account.
  2. Note the Service Account ID, which typically follows the format <account-name>@<project-name>.iam.gserviceaccount.com.

Granting Permissions:

  1. Add the Service Account as a Member in your Google Cloud Project. For instructions, see Granting Access. The email address of the member is the Service Account ID.

Service Account Key​

To authenticate as the Service Account, you need to provide its Service Account Key.

Creating and Managing Keys:

  1. Refer to Google's documentation on Service Account Keys.
  2. Follow the guide on Creating and Managing Service Account Keys to generate a key. Boltic supports only JSON Keys.
  3. Download the key immediately, as Google will only show it once. After configuring BigQuery in Boltic, delete the key from your computer for security.

Setup Guide to Integrate BigQuery With Boltic​

This guide will give you a brief idea of what steps you need to follow to integrate BigQuery into Boltic.

  1. Integration Name: Enter a unique name for this BigQuery integration.

    Integration Name

  2. Select Service Account: Choose the Service Account option if not already selected.

  3. Attach Key: Upload the Service Account Key (Private Key) JSON file you created.

    Service Account Key

  4. Project ID: Enter the Project ID associated with the Service Account.

Additional Info​

Enhance your data management with metadata:

FieldDetails
TagsUse tags for easy data discovery for you and your team.
DescriptionProvide a detailed and meaningful description of your dataset.

Tags & Description

Test & Save​

Validate your configuration by clicking Test & Save. This ensures that the connection is successfully established.

Test & Save

Congratulations! You've successfully integrated BigQuery with Boltic.

🤓 Have Questions?​

We're here to help! Reach out to us via email at 📧 [email protected].