> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/mongodb.md).

# MongoDB

MongoDB batch source. Reads documents with optional incremental ingestion; supports aggregation-pipeline queries by passing a pipeline array as the source\_table value.

[MongoDB](https://www.mongodb.com/docs/) is a distributed NoSQL document database. Nilus reads from MongoDB as a **batch source**, with optional incremental ingestion driven by a monotonic field, and an aggregation-pipeline type for source-side filtering and projection.

For row-level change capture from MongoDB, use the [MongoDB (CDC)](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/mongodb.md) instead.

## Requirements

Connectivity and credentials must both be in place before the pipeline can run.

### Connectivity

* The Nilus runtime must reach the MongoDB endpoint. Self-hosted clusters listen on TCP `27017` by default; managed Atlas clusters are typically reached via SRV-discovery URIs.
* The connector accepts both `mongodb://` and `mongodb+srv://` URI schemes.
* For Atlas, allowlist the runtime egress IP range under **Network Access** before the first run, otherwise authentication will succeed but every read will time out.

### Required parameters

| Parameter        | Required    | Default | Description                                                                                                                               |
| ---------------- | ----------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `host`           | Yes         | -       | MongoDB server hostname (or comma-separated replica-set member list).                                                                     |
| `port`           | No          | `27017` | MongoDB port. Omit when using `mongodb+srv://`.                                                                                           |
| `username`       | Conditional | -       | Required when the cluster enforces authentication.                                                                                        |
| `password`       | Conditional | -       | Required when the cluster enforces authentication.                                                                                        |
| Query parameters | No          | -       | Standard connection-string options forwarded to the driver: `tls=true`, `retryWrites=true`, `authSource=admin`, `replicaSet=<name>`, etc. |

### Database-side permissions

The connection user needs `read` on the source database and collection:

```javascript
db.grantRolesToUser("<username>", [
  { role: "read", db: "<database_name>" }
])
```

### URI format

```
mongodb://<username>:<password>@<host>:<port>?<optional-query-params>
```

SRV-discovery (Atlas-style):

```
mongodb+srv://<username>:<password>@<cluster>.mongodb.net?retryWrites=true&w=majority
```

Depot-backed (recommended for production):

```
dataos://<mongodb-depot>
```

> **Note** Do **not** put the database name in the URI path. The MongoDB source derives the database from the **first segment** of `source_table` (`<database>.<collection>`).

## Source options

| Option              | Required | Description                                                                                                                                                                                                                                                                                                     |
| ------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `source_table`      | Yes      | Either a plain `<database>.<collection>` reference, or `<database>.<collection>:[<aggregation-pipeline>]` to drive extraction with a source-side aggregation pipeline (see "Custom aggregation pipelines" below).                                                                                               |
| `primary_key`       | No       | Field (or comma-separated fields) used by `incremental_strategy: merge` to match documents. Defaults to `_id`; setting it replaces `_id` as the merge key. The field name must match the document field exactly.                                                                                                |
| `incremental_key`   | No       | Document field used to identify newly visible documents per run. Should be indexed in MongoDB and consistently present across documents.                                                                                                                                                                        |
| `interval_start`    | No       | Optional ISO-8601 lower bound for the extraction window.                                                                                                                                                                                                                                                        |
| `interval_end`      | No       | Optional ISO-8601 upper bound for the extraction window.                                                                                                                                                                                                                                                        |
| `max_table_nesting` | No       | String. How many levels of nested objects are flattened into separate columns. Default `"0"`: each nested object lands as one `json` column. `"1"` flattens one level, `"2"` two levels. Applies to plain collections and aggregation pipelines alike. See [Output schema and types](#output-schema-and-types). |

> **Important** Earlier versions of this page documented `filter_`, `projection`, `chunk_size`, `parallel`, and `data_item_format` as YAML source options. **They are not currently forwarded** by the Nilus MongoDB source wrapper, so setting them in `source.options` has no effect. Use the aggregation-pipeline form of `source_table` for filtering and projection at the source side.

## Custom aggregation pipelines

To drive extraction with a MongoDB aggregation pipeline, suffix the collection name with `:[<pipeline>]`:

```yaml
source:
  address: dataos://mongodbdepot
  options:
    source_table: 'analytics.orders:[{"$match": {"status": "active"}}, {"$project": {"_id": 1, "customer_id": 1, "total": 1, "updated_at": 1}}]'
    incremental_key: updated_at
    interval_start: "2024-01-01T00:00:00Z"
```

#### Pipeline requirements:

* The pipeline must be a **JSON array** (not a `db.collection.aggregate(...)` shell expression). MongoDB Extended JSON v2 is supported via the BSON `json_util` parser.
* When `incremental_key` is set, the field **must** be projected out by your pipeline; otherwise the source raises a validation error before extraction begins.
* When `incremental_key` is set, the connector substitutes `interval_start` / `interval_end` into the pipeline if that bound is present. The placeholder must be the JSON string `":interval_start"` or `":interval_end"` as the whole value, for example `{"$match": {"updated_at": {"$gte": ":interval_start"}}}`. `$interval_start` is not replaced.

## Output schema and types

There is no fixed Nilus schema for a MongoDB collection. At `max_table_nesting: "0"` (the default), a document like this:

```json
{
  "_id": {"$oid": "507f1f77bcf86cd799439011"},
  "name": "A",
  "updated_at": {"$date": "2024-01-01T00:00:00Z"},
  "addr": {"city": "Pune", "geo": {"lat": 18.5}}
}
```

lands as:

| Column       | Type                |
| ------------ | ------------------- |
| `_id`        | text (ObjectId hex) |
| `name`       | text                |
| `updated_at` | timestamp           |
| `addr`       | json                |

{% hint style="info" %}
Some destinations also write load-metadata columns (`_dlt_load_id` / `_dlt_id`, or `_nilus_load_id` / `_nilus_id`). They are not MongoDB fields and are not guaranteed on every destination.
{% endhint %}

{% hint style="info" %}
Schema is inferred from the documents Nilus actually reads. Later pages can add nullable columns in the same run. A field that is missing from every early page, or all-null until a later page, can still be dropped unless you pin it with `type_hints`. See [Schema evolution](https://v2.dataos.info/references/resources/nilus/concepts/schema-evolution.md).
{% endhint %}

### `_id`

`_id` is loaded on a plain collection read and is the default `primary_key` for `incremental_strategy: merge`. Other strategies ignore the key. An aggregation `$project` that omits `_id` drops it from the output. How it lands depends on its type in MongoDB:

| `_id` in MongoDB                                       | Landed type           |
| ------------------------------------------------------ | --------------------- |
| ObjectId                                               | text (the hex string) |
| String                                                 | text                  |
| Number                                                 | bigint                |
| Document (for example, the output of a `$group` stage) | json                  |

### Nested objects and `max_table_nesting`

Field names map to columns directly. For a document like `{"name": "A", "addr": {"city": "Pune", "geo": {"lat": 18.5}}}`:

| `max_table_nesting` | Columns                                         |
| ------------------- | ----------------------------------------------- |
| `"0"` (default)     | `name`, `addr` (json)                           |
| `"1"`               | `name`, `addr__city`, `addr__geo` (json)        |
| `"2"`               | `name`, `addr__city`, `addr__geo__lat` (double) |

The setting applies to every collection, including aggregation-pipeline sources.

### Arrays

Top-level array fields land as a single `json` column at every `max_table_nesting` value, so they don't become child tables.

> **Note** Nilus decides which fields are arrays by looking at the **first document** it reads. If an array field is missing from that document and `max_table_nesting` is `"1"` or higher, the field can land as a child table (`<collection>__<field>`) instead of a json column. Keep `max_table_nesting` at `"0"`, or make sure the field is present in the first document returned, for example with a `$match` or `$sort` stage.

### Other BSON types

| BSON type          | Landed type        |
| ------------------ | ------------------ |
| Date               | timestamp          |
| Timestamp          | timestamp          |
| Decimal128         | text               |
| Regular expression | text (the pattern) |

### What `$project` changes

With an aggregation pipeline, the output of the pipeline decides which fields exist, so `$project` controls the columns that land. Include `"_id": 1` if you need `_id` downstream; an inclusion `$project` that omits it drops `_id`. If `incremental_key` is set, `$project` must keep that field, or the run fails validation before extraction starts.

### How `incremental_key` tracks progress

`incremental_key` uses the pipeline's saved state, not a fixed window:

* **First run:** starts from `interval_start`.
* **Later runs:** continue from the last value saved in pipeline state.
* **With `interval_end` set:** Nilus loads exactly the range from `interval_start` to `interval_end` and **doesn't save state**. Use this for backfills; it doesn't move the regular incremental position.

## Sample Nilus configs

Each example below is self-contained and uses the current Nilus pipeline shape.

### Batch, full-collection snapshot to Lakehouse

```yaml
name: nilus-mongodb-batch
version: v1alpha
type: nilus
description: MongoDB collection → DataOS Lakehouse snapshot
spec:
  type: batch
  compute: runnable-default
  source:
    address: dataos://mongodbdepot
    options:
      source_table: retail.customers
      max_table_nesting: "0"
  sink:
    address: dataos://analyticslakehouse
    options:
      dest_table: retail.customers_raw
      incremental_strategy: replace
      aws_region: us-west-2
```

### Batch, incremental ingestion driven by `updated_at`

```yaml
spec:
  type: batch
  source:
    address: dataos://mongodbdepot
    options:
      source_table: retail.orders
      incremental_key: updated_at
      max_table_nesting: "0"
  sink:
    address: dataos://analyticslakehouse
    options:
      dest_table: retail.orders_raw
      incremental_strategy: merge
```

### Batch, aggregation pipeline with projection

```yaml
spec:
  type: batch
  source:
    address: dataos://mongodbdepot
    options:
      source_table: 'analytics.events:[{"$match": {"event_type": "purchase"}}, {"$project": {"_id": 1, "user_id": 1, "amount": 1, "occurred_at": 1}}]'
      incremental_key: occurred_at
  sink:
    address: dataos://analyticslakehouse
    options:
      dest_table: analytics.purchase_events
      incremental_strategy: append
```

## Behavior and capabilities

* **Compute model**: the Nilus runtime drives the cluster through the official `pymongo` driver; the cluster does the read.
* **Object model**: MongoDB databases and collections; one Nilus pipeline reads exactly one collection (or one aggregation pipeline output).
* **Pipeline mode**: `batch` only on this page; for row-level change capture see the [MongoDB (CDC)](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/mongodb.md).
* **Aggregation-pipeline form**: the pipeline's output documents are what land. `max_table_nesting` applies to them the same way as to a plain collection (default `0`).
* **Document IDs**: `_id` is loaded on a plain collection read and is the default `primary_key`. An aggregation `$project` can drop it. See [Output schema and types](#output-schema-and-types) for how each `_id` type lands.
* **Incremental key requirements**: the field should be indexed in MongoDB so the source can scan a bounded range efficiently. Without an index, the cluster falls back to a full collection scan with a server-side filter.
* **Later pages can add fields**: if later pages introduce fields that earlier pages did not have, Nilus unions those columns onto the destination as nullable. Documents without the field land as NULL. Pin a sparse field that is all-null on early pages with `type_hints`. See [Schema evolution](https://v2.dataos.info/references/resources/nilus/concepts/schema-evolution.md) and [Correctness knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-correctness-knobs.md).

## Troubleshooting

| Symptom                                                                                                 | Likely cause                                                                                                                                                               | Resolution                                                                                                                                                                                                                               |
| ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ServerSelectionTimeoutError`                                                                           | Atlas IP allow-list rejects the runtime egress IP, or the SRV record could not be resolved.                                                                                | Add the runtime egress range to **Atlas Network Access**, and confirm DNS resolution for the SRV record from inside the runtime.                                                                                                         |
| `OperationFailure: Authentication failed`                                                               | Wrong username/password, wrong `authSource`, or the user is not provisioned on the database.                                                                               | Verify credentials, add `authSource=admin` (or the correct auth DB) to the URI query, and confirm the user has `read` on the source database.                                                                                            |
| `Invalid MongoDB query format: …`                                                                       | `source_table` was supplied with a `:[…]` suffix that is not valid JSON / Extended JSON.                                                                                   | Convert MongoDB shell syntax (`new Date(...)`, `ISODate(...)`, etc.) into Extended JSON v2 form, or use the helper utilities documented in the MongoDB driver docs.                                                                      |
| `Query must be a JSON array representing a MongoDB aggregation pipeline`                                | The aggregation suffix decoded to a single document instead of a pipeline array.                                                                                           | Wrap the stages in `[ ... ]`.                                                                                                                                                                                                            |
| `Incremental key '<field>' is not projected by the pipeline`                                            | The pipeline's final `$project` stage drops the `incremental_key` field.                                                                                                   | Include the incremental key in the `$project` stage so the runtime can apply the watermark filter.                                                                                                                                       |
| Slow runs / high cluster CPU on `incremental_key` queries                                               | The incremental field is not indexed; the cluster does a full COLLSCAN per run.                                                                                            | Build an index on the incremental field upstream: `db.<collection>.createIndex({ <field>: 1 })`.                                                                                                                                         |
| TLS handshake failures                                                                                  | Cluster requires TLS; URI did not opt in.                                                                                                                                  | Append `?tls=true` (and, for self-signed dev clusters, `tlsAllowInvalidCertificates=true`).                                                                                                                                              |
| `ValueError: PyArrow table contains more columns: … Update the schema first (hint, use union_by_name).` | Later pages introduced fields that earlier pages did not have, and the destination schema was not unioned. Current Nilus unions those columns as nullable in the same run. | Retry on the current runtime. You do not need to drop the destination table. See [Schema evolution](https://v2.dataos.info/references/resources/nilus/concepts/schema-evolution.md).                                                     |
| `ValueError: Could not find field with name: <field>`                                                   | A sparse field was all-null on early pages, so it was dropped from the inferred schema, then a later page had a value.                                                     | Pin the field with `type_hints` (for example `deleted_utc: timestamp`) and rerun. See [Correctness knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-correctness-knobs.md). |

## Related docs

* [Batch data movement](https://v2.dataos.info/references/resources/nilus/batch.md)
* [Batch pipeline config](https://v2.dataos.info/references/resources/nilus/batch/pipeline-config.md)
* [Batch sample configs](https://v2.dataos.info/references/resources/nilus/batch/sample-configs.md)
* [MongoDB](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/mongodb.md): companion CDC connector for row-level change capture.
* [MongoDB](https://v2.dataos.info/references/resources/nilus/destinations/databases/mongodb.md): companion destination connector.
* [Optimize sink datasets](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets.md): guidance on `incremental_strategy`, `max_table_nesting`, `partition_by`, and other dataset-shape settings.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/batch/batch-sources/mongodb.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
