Skip to main content

MongoDB Configuration

Welcome to your ultimate guide on configuring MongoDB as a source database for Boltic Pipes! This document will guide you through every necessary step to achieve a flawless integration of MongoDB with Boltic’s dynamic data pipeline framework.

Boltic Pipes is crafted to handle data synchronization and transfer with precision. To prepare MongoDB as a source, you’ll need to adjust several configurations to ensure smooth operation and reliable data flow. This guide covers the essential settings and best practices to optimize your MongoDB setup for a seamless integration with Boltic Pipes.

Prerequisites​

  • MongoDB version 4.4 or higher is required.

  • Check your MongoDB version on Ubuntu with:

    mongod --version

  • Ensure a MongoDB user has read access to the target database and the local database.

  • The OpLog retention period must be at least 72 hours.

MongoDB Message Size Limitation​

Change Streams response documents in MongoDB must not exceed the 16 MB BSON limit. If a document exceeds this limit, the pipe pauses, and an error appears:

Failed to read documents from the Change Stream. Documents larger than 16 MB are not supported.

This issue can occur when:

  • The Change Streams update operation is set to return the full updated document.
  • An insert or replace operation involves a document close to or exceeding the 16 MB limit.

Failure Handling Mode​

The Failure Handling Mode lets users decide how to handle errors like oversized documents or events during data processing:

Pipe Failure Handling Mode

Skip: Logs the issue and continues processing the remaining data to avoid interruptions.

Fail: Stops processing and shows an error message to address the issue immediately.

Users can configure this option easily in the UI while creating a pipe. This feature provides clear control and ensures smooth data processing.

Performing CDC on Multiple Databases​

To set up Change Data Capture (CDC) for multiple databases in your MongoDB cluster, follow these steps. Ensure that you use admin as the authentication database when creating the integration.

1. Switch to the admin Database​

use admin;

2. Create User for CDC​

db.createUser({
    user: "boltic_cdc_user",
    pwd: "password",
    roles: []
});

3. Create a role with required privileges for CDC​

db.createRole({
    role: "cdcRole",
    privileges: [
        { resource: { db: "", collection: "" }, actions: ["find", "changeStream", "listCollections"] }
    ],
    roles: []
});

4. Grant the created role and other necessary roles to the user​

db.grantRolesToUser(
    "boltic_cdc_user",
    [
        "read",
        { role: "cdcRole", db: "admin" },
        { role: "clusterManager", db: "admin" },
        { role: "read", db: "local" }
    ]
);

Performing CDC on a Single Database​

To set up CDC for a single database, you can either use admin or the specific database as the authentication database. Follow the appropriate set of commands based on your choice.

Option 1: Using admin as Authentication DB​

If you choose admin as the authentication database and want to perform CDC on the employees database, use the following commands:

1. Switch to the admin Database​

use admin;

2. Create User for CDC​

db.createUser({
    user: "boltic_cdc_user",
    pwd: "password",
    roles: []
});

3. Create a role with required privileges for CDC​

db.createRole({
    role: "cdcRole",
    privileges: [
        { resource: { db: "employees", collection: "" }, actions: ["find", "changeStream", "listCollections"] }
    ],
    roles: []
});

4. Grant the created role and other necessary roles to the user​

db.grantRolesToUser(
    "boltic_cdc_user",
    [
        "read",
        { role: "cdcRole", db: "admin" },
        { role: "clusterManager", db: "admin" },
        { role: "read", db: "local" }
    ]
);

Option 2: Using employees as Authentication DB​

If you choose employees as the authentication database and want to perform CDC on the employees database, use the following commands:

1. Switch to the employees Database​

use employees;

2. Create User for CDC​

db.createUser({
    user: "boltic_cdc_user",
    pwd: "password",
    roles: []
});

3. Create a role with required privileges for CDC​

db.createRole({
    role: "cdcRole",
    privileges: [
        { resource: { db: "employees", collection: "" }, actions: ["find", "changeStream", "listCollections"] }
    ],
    roles: []
});

4. Grant the created role and other necessary roles to the user​

db.grantRolesToUser(
    "boltic_cdc_user",
    [
        "read",
        { role: "cdcRole", db: "admin" },
        { role: "clusterManager", db: "admin" },
        { role: "read", db: "local" }
    ]
);

🤓 Have Questions?​

We're here to help! Feel free to reach out via email at 📧 [email protected].