Skip to main content

Google Cloud Storage (GCS)

GCS iis a scalable and secure object storage service provided by Google Cloud. It allows you to store and retrieve any amount of data at any time, with features like high durability, accessibility, and integration with other Google Cloud services. GCS is ideal for storing unstructured data such as backups, media files, and large datasets, with flexible access control and data management options.

Getting Started​

Requirements​

Before proceeding, ensure the following:

  • Ensure that the Boltic server can connect to your GCS cluster, especially if they are located in separate VPCs.
  • Provide a GCP bucket with the necessary credentials (for the COPY strategy).

Step-by-Step Guide​

  • GCS Bucket Name
    Follow the instructions here to create a GCS bucket. Ensure the bucket does not have a retention policy and that the Protection Tools are set to "None" or "Object Versioning."

  • GCS Bucket Region
    [Specify region here]

  • HMAC Key Access ID
    Instructions on how to generate an access key can be found here. For more details on HMAC keys, refer to the GCP documentation.
    We recommend creating a dedicated Boltic user or service account. This account will need the following permissions for the bucket:

    • storage.multipartUploads.abort
    • storage.multipartUploads.create
    • storage.objects.create
    • storage.objects.delete
    • storage.objects.get
    • storage.objects.list

    You can set these permissions by navigating to the permissions tab in the GCS bucket and adding the service account or user email with the appropriate permissions.

  • Secret Access Key
    This is the key corresponding to the above Access ID.

Ensure your GCS bucket is accessible from the machine running Boltic. This depends on your network configuration. To verify if Boltic can connect to the GCS bucket, use the check connection tool in the Boltic UI.

Best Practices​

  • Adopt Clear Naming Conventions: Use consistent names for buckets, objects, and directories.
  • Regularly Review User Permissions: Ensure that users have only the minimum required access.
  • Pause Unused Buckets or Services: Stop any unnecessary services or operations to optimize costs.
  • Optimize with Storage Class Selection: Choose the appropriate storage class for data to optimize both cost and access speed.
  • Utilize Object Versioning: Enable object versioning for data protection and recovery.
  • Efficient Data Uploads with Multipart: Upload large files in multipart to optimize transfer and reliability.
  • Use Lifecycle Policies for Data Management: Automate data archival or deletion with lifecycle policies.
  • Enable Object-Level Encryption: Ensure data is encrypted at rest and in transit using GCS’s security features.
  • Leverage Access Control Policies: Implement precise IAM roles and policies to secure access to your GCS buckets and objects.
  • Monitor Storage Usage: Regularly monitor storage usage and set up alerts to avoid unexpected charges.

Setup Guide to Integrate GCS With Boltic​

This guide will give you a brief idea of what steps you need to follow to integrate GCS into Boltic.

  1. Search for GCS destination: Go to integrations > destinations > Add new destination Integration Name

    Integration Name

  2. Add new destination integration: Enter a unique name for this GCS integration. Integration Name

  3. Add new destination integration: Add description and further gcs service account Integration Name

  4. Test and save: Validate your configuration by clicking Test & Save. This ensures that the connection is successfully established. Integration Name

Congratulations! You’ve successfully configured GCS roles, users, warehouses, databases, and schemas for Boltic integration. For further assistance, refer to GCS’s official documentation.