> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/v1/resources/nilus/metadata-pipelines/metadata-sources/mongodb.md).

# MongoDB

[MongoDB](https://www.mongodb.com/docs/) is supported as a metadata source. A `spec.type: metadata` pipeline introspects a MongoDB deployment and publishes source context — databases, collections, inferred fields, and profiles — into the DataOS metadata catalog **without copying any documents**. Because MongoDB is schema-less, Nilus infers each collection's field structure by sampling documents.

For batch row movement out of MongoDB, see the [MongoDB (Batch)](/references/v1/resources/nilus/batch/batch-sources/mongodb.md); for change data capture, see the [MongoDB (CDC)](/references/v1/resources/nilus/cdc/cdc-sources/mongodb.md). For the field-by-field authoring contract, see [Understanding Metadata Pipeline Config](/references/v1/resources/nilus/metadata-pipelines/pipeline-config.md).

## Metadata stages

`service_type: mongodb` supports `metadata`, `profiler`, and `classification` only.

{% hint style="warning" %}
**Lineage and usage are not supported** — MongoDB has no query-history or view-definition surface for Nilus to parse, so those stages produce no output.
{% endhint %}

| Stage            | Runs in           | What it lands in the catalog                                                                                                                |
| ---------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `metadata`       | `shallow`, `deep` | Databases, collections, and the fields inferred from sampled documents (including nested documents and arrays).                             |
| `profiler`       | `deep`            | Per-field statistics computed from a document sample (value counts, null ratios, basic distributions) and collection-level document counts. |
| `classification` | `deep`            | Auto-classification tags applied to fields from sampled values (PII heuristics).                                                            |
| `lineage`        | —                 | Not supported for MongoDB.                                                                                                                  |
| `usage`          | —                 | Not supported for MongoDB.                                                                                                                  |

Because lineage and usage are not applicable, a `shallow` MongoDB pipeline effectively runs the `metadata` stage only, and a `deep` pipeline runs `metadata` + `profiler` + `classification`. `mode` is still required by the schema. `query_log_duration` and `result_limit` have no effect on MongoDB.

## Asset hierarchy

MongoDB assets map into the Datasets App as `Database → Collection → Field`. Nested documents are represented as nested field structures, and arrays are represented as repeated fields, so the shape sampled from the documents is preserved in the catalog.

## Source options

Metadata pipelines accept only the customer-facing `source.options` keys below.

{% hint style="warning" %}
Do **not** set `source_table` — Nilus assigns a stage-specific value to each DAG node internally.
{% endhint %}

| Option            | Required | Used by stages                           | Description                                                                              |
| ----------------- | -------- | ---------------------------------------- | ---------------------------------------------------------------------------------------- |
| `service_type`    | Yes      | all                                      | Must be `mongodb`.                                                                       |
| `database_filter` | No       | `metadata`, `profiler`, `classification` | Restrict by database name. Object with `includes` / `excludes` arrays of regex patterns. |
| `table_filter`    | No       | `metadata`, `profiler`, `classification` | Restrict by collection name. Same shape as `database_filter`.                            |
| `threads`         | No       | `profiler`                               | Parallel worker count for the profiler. Raise it to cut runtime on large scopes.         |

{% hint style="info" %}
`schema_filter`, `query_log_duration`, and `result_limit` do not apply to MongoDB — it has no schema layer between database and collection, and no query-history surface. `mode` (`shallow` or `deep`) is a required `spec` field, not a `source.options` key.
{% endhint %}

## Required permissions

Metadata extraction needs read access to the databases and collections in scope, plus the ability to enumerate the deployment:

* `read` on each database in scope (or `readAnyDatabase`) — required for inventory, profiling, and classification, all of which sample documents;
* `clusterMonitor` on `admin` — so Nilus can list databases and read deployment topology.

Example grant for a dedicated metadata user:

```javascript
use admin
db.createUser({
  user: "nilus_metadata",
  pwd: "<password>",
  roles: [
    { role: "readAnyDatabase", db: "admin" },
    { role: "clusterMonitor",  db: "admin" }
  ]
})
```

To limit scope to specific databases, replace `readAnyDatabase` with a per-database `read` grant for each database in scope. `clusterMonitor` is still required so Nilus can enumerate databases.

## Sample Nilus config

### Depot-backed (recommended for production)

```yaml
name: mongodb-metadata
version: v1alpha
type: nilus
tags: [nilus, metadata]
description: Catalog MongoDB databases, collections, and profiles
spec:
  type: metadata
  mode: deep
  compute: comet-compute
  schedule:
    crons:
      - "0 */6 * * *"
    concurrencyPolicy: Forbid
  source:
    address: dataos://mongometadatadepot?purpose=rw
    options:
      service_type: mongodb
      database_filter:
        includes: ["appdb", "analytics"]
        excludes: ["^admin$", "^local$", "^config$"]
      table_filter:
        excludes: ["^system\\.", "_tmp$"]
      threads: 4
```

### Direct URI (no depot)

Project credentials through `spec.use.projection`:

```yaml
name: mongodb-metadata-direct
version: v1alpha
type: nilus
tags: [nilus, metadata]
spec:
  type: metadata
  mode: shallow
  compute: comet-compute
  schedule:
    crons:
      - "0 * * * *"
    concurrencyPolicy: Forbid
  use:
    projection:
      secrets:
        - id: engineering:mongo-secret
          contextAlias: mongosecret
      projections:
        envVars:
          - key: MONGO_USER
            template: "{{ secrets['mongosecret'].username | base64_decode }}"
          - key: MONGO_PASSWORD
            template: "{{ secrets['mongosecret'].password | base64_decode }}"
  source:
    address: mongodb://{MONGO_USER}:{MONGO_PASSWORD}@mongo1.example.com:27017,mongo2.example.com:27017?databaseName=appdb&sslmode=require
    options:
      service_type: mongodb
      database_filter:
        includes: ["appdb"]
```

`shallow` mode runs the `metadata` stage only; `deep` mode additionally runs `profiler` and `classification`. Lineage and usage stages are never generated for MongoDB. For more ready-to-edit examples, see [Metadata Sample Configs](/references/v1/resources/nilus/metadata-pipelines/sample-configs.md).

## Behavior and capabilities

* **Connection** — use a MongoDB depot (`dataos://<depot>?purpose=rw`) or a direct `mongodb://...` URI with projected credentials. Because `spec.type` is `metadata`, Nilus adds the internal `metadata+` routing prefix automatically. The URI accepts a comma-separated list of replica-set members in the host position; pass `databaseName` and `sslmode` as query parameters. `sslmode` defaults to `disable` when omitted.
* **Schema inference** — MongoDB is schema-less, so field structure is inferred from a sample of documents per collection. Fields that appear only in unsampled documents may not be captured; heterogeneous collections are represented by the shape observed in the sample.
* **Nested data** — nested documents and arrays are preserved as nested / repeated field structures in the catalog.
* **No lineage or usage** — MongoDB exposes no query history or view definitions, so lineage and usage are not produced. Model relationships downstream if you need them; do not expect column-level lineage for document collections.
* **Scope discipline** — set `database_filter` / `table_filter` in production, and exclude the `admin`, `local`, `config`, and `system.*` namespaces.

## Troubleshooting

| Symptom                                                        | Likely cause                                                         | Resolution                                                                                                 |
| -------------------------------------------------------------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `Authentication failed`                                        | Wrong credentials or missing `read` on the target database.          | Verify the projected secret and grant `read` (or `readAnyDatabase`).                                       |
| `not authorized on admin to execute command { listDatabases }` | The user lacks `clusterMonitor`.                                     | Grant `clusterMonitor` on `admin` so Nilus can enumerate databases.                                        |
| Collections appear with missing fields                         | The document sample did not include documents carrying those fields. | Expected for heterogeneous collections; increase the sample where supported, or accept the inferred shape. |
| Lineage / usage empty                                          | Not supported for MongoDB.                                           | Expected — MongoDB has no query-history or view surface. Use `metadata` + `profiler` + `classification`.   |
| TLS handshake errors                                           | Server enforces TLS but the URI omits `sslmode`.                     | Add `?sslmode=require` (or a stricter mode) to the URI.                                                    |

## Related Docs

* [MongoDB (Batch)](/references/v1/resources/nilus/batch/batch-sources/mongodb.md) — batch row movement out of MongoDB.
* [MongoDB (CDC)](/references/v1/resources/nilus/cdc/cdc-sources/mongodb.md) — log-based change capture.
* [Metadata Sources](/references/v1/resources/nilus/metadata-pipelines/metadata-sources.md) — all metadata-capable sources and how to scope extraction.
* [Understanding Metadata Pipelines](/references/v1/resources/nilus/metadata-pipelines.md) — the conceptual model.
* [Understanding Metadata Pipeline Config](/references/v1/resources/nilus/metadata-pipelines/pipeline-config.md) — the `spec.type: metadata` contract and DAG anatomy.
* [Metadata Sample Configs](/references/v1/resources/nilus/metadata-pipelines/sample-configs.md) — ready-to-edit YAML.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/v1/resources/nilus/metadata-pipelines/metadata-sources/mongodb.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.