> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/mongodb.md).

# MongoDB

Configuration reference for the MongoDB metadata source: supported stages, source options, required roles, and troubleshooting.

[MongoDB](https://www.mongodb.com/docs/) is supported as a metadata source. A `spec.type: metadata` pipeline introspects a MongoDB deployment and publishes source context, including databases, collections, inferred fields, and profiles, into the DataOS metadata catalog **without copying any documents**. Because MongoDB is schema-less, Nilus infers each collection's field structure by sampling documents.

For batch row movement out of MongoDB, see [MongoDB (batch)](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/mongodb.md). For change data capture, see [MongoDB (CDC)](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/mongodb.md). For the field-by-field authoring contract, see [Understanding metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md).

## Metadata stages

`service_type: mongodb` supports `metadata`, `profiler`, and `classification` only.

{% hint style="warning" %}
**Lineage and usage are not supported.** MongoDB has no query-history or view-definition surface for Nilus to parse, so those stages produce no output.
{% endhint %}

| Stage            | Runs in           | What it lands in the catalog                                                                                                                |
| ---------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `metadata`       | `shallow`, `deep` | Databases, collections, and the fields inferred from sampled documents (including nested documents and arrays).                             |
| `profiler`       | `deep`            | Per-field statistics computed from a document sample (value counts, null ratios, basic distributions) and collection-level document counts. |
| `classification` | `deep`            | Auto-classification tags applied to fields from sampled values (PII heuristics).                                                            |
| `lineage`        | Not applicable    | Not supported for MongoDB.                                                                                                                  |
| `usage`          | Not applicable    | Not supported for MongoDB.                                                                                                                  |

Because lineage and usage are not applicable, a `shallow` MongoDB pipeline effectively runs the `metadata` stage only, and a `deep` pipeline runs `metadata` + `profiler` + `classification`. `mode` is still required by the schema. `query_log_duration` and `result_limit` have no effect on MongoDB.

## Asset hierarchy

The catalog models a MongoDB **server** as one database named `default`. Real Mongo databases appear one level down as **schemas**, and collections as **tables**. Nested documents are nested fields, and arrays are repeated fields.

Use `schema_filter` with the Mongo database name. A real Mongo database name does not match the catalog database, so `database_filter` is ineffective for this purpose.

## Source options

Metadata pipelines accept only the customer-facing `source.options` keys below.

{% hint style="warning" %}
Do **not** set `source_table`. Nilus assigns a stage-specific value to each DAG node internally.
{% endhint %}

| Option            | Required | Used by stages                           | Description                                                                                                                   |
| ----------------- | -------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `service_type`    | Yes      | all                                      | Must be `mongodb`.                                                                                                            |
| `schema_filter`   | No       | `metadata`, `profiler`, `classification` | Restrict by **Mongo database name** (the catalog schema level). Object with `includes` / `excludes` arrays of regex patterns. |
| `table_filter`    | No       | `metadata`, `profiler`, `classification` | Restrict by collection name. Same shape as `schema_filter`.                                                                   |
| `database_filter` | No       | Not applicable                           | Not useful on MongoDB. The catalog database is always `default`; names like `appdb` match nothing.                            |
| `threads`         | No       | `profiler`                               | Parallel worker count for the profiler. Raise it to cut runtime on large scopes.                                              |

{% hint style="info" %}
`query_log_duration` and `result_limit` do not apply to MongoDB because it has no query-history surface. `mode` (`shallow` or `deep`) is a required `spec` field, not a `source.options` key.
{% endhint %}

## Required permissions

Metadata extraction needs read access to the databases and collections in scope, plus the ability to enumerate the deployment:

* `read` on each database in scope (or `readAnyDatabase`) is required for inventory, profiling, and classification, all of which sample documents.
* `clusterMonitor` on `admin` lets Nilus list databases and read deployment topology.

Example grant for a dedicated metadata user:

```javascript
use admin
db.createUser({
  user: "nilus_metadata",
  pwd: "<password>",
  roles: [
    { role: "readAnyDatabase", db: "admin" },
    { role: "clusterMonitor",  db: "admin" }
  ]
})
```

To limit scope to specific databases, replace `readAnyDatabase` with a per-database `read` grant for each database in scope. `clusterMonitor` is still required so Nilus can enumerate databases.

## Sample Nilus config

### Depot-backed (recommended for production)

```yaml
name: mongodb-metadata
version: v1alpha
type: nilus
tags: [nilus, metadata]
description: Catalog MongoDB databases, collections, and profiles
spec:
  type: metadata
  mode: deep
  compute: comet-compute
  schedule:
    crons:
      - "0 */6 * * *"
    concurrencyPolicy: Forbid
  source:
    address: dataos://mongometadatadepot?purpose=rw
    options:
      service_type: mongodb
      schema_filter:
        includes: ["appdb", "analytics"]
        excludes: ["^admin$", "^local$", "^config$"]
      table_filter:
        excludes: ["^system\\.", "_tmp$"]
      threads: 4
```

### Direct URI (no depot)

Project credentials through `spec.use.projection`:

```yaml
name: mongodb-metadata-direct
version: v1alpha
type: nilus
tags: [nilus, metadata]
spec:
  type: metadata
  mode: shallow
  compute: comet-compute
  schedule:
    crons:
      - "0 * * * *"
    concurrencyPolicy: Forbid
  use:
    projection:
      secrets:
        - id: engineering:mongo-secret
          contextAlias: mongosecret
      projections:
        envVars:
          - key: MONGO_USER
            template: "{{ secrets['mongosecret'].username | base64_decode }}"
          - key: MONGO_PASSWORD
            template: "{{ secrets['mongosecret'].password | base64_decode }}"
  source:
    address: mongodb://{MONGO_USER}:{MONGO_PASSWORD}@mongo1.example.com:27017,mongo2.example.com:27017?databaseName=appdb&sslmode=require
    options:
      service_type: mongodb
      schema_filter:
        includes: ["appdb"]
```

`shallow` mode runs the `metadata` stage only; `deep` mode additionally runs `profiler` and `classification`. Lineage and usage stages are never generated for MongoDB. For more ready-to-edit examples, see [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md).

## Behavior and capabilities

* **Connection:** Use a MongoDB depot (`dataos://<depot>?purpose=rw`) or a direct `mongodb://...` URI with projected credentials. Because `spec.type` is `metadata`, Nilus adds the internal `metadata+` routing prefix automatically. The URI accepts a comma-separated list of replica-set members in the host position. Pass `databaseName` and `sslmode` as query parameters. `sslmode` defaults to `disable` when omitted. Depot connections, and URIs without `databaseName`, still land under the `default` catalog database.
* **Filter on schema, not database:** A `database_filter` matching a real Mongo database name leaves inventory unchanged. It can also make `classification` and `profiler` complete with no work. Use `schema_filter` with the Mongo database name.
* **Schema inference:** MongoDB is schema-less, so field structure is inferred from a sample of documents per collection. Fields that appear only in unsampled documents may not be captured. Heterogeneous collections are represented by the shape observed in the sample.
* **Nested data:** Nested documents and arrays are preserved as nested and repeated field structures in the catalog.
* **No lineage or usage:** MongoDB exposes no query history or view definitions, so lineage and usage are not produced. Model relationships downstream if needed. Do not expect column-level lineage for document collections.
* **Scope discipline:** Set `schema_filter` and `table_filter` in production. Exclude the `admin`, `local`, `config`, and `system.*` namespaces.

## Troubleshooting

| Symptom                                                        | Likely cause                                                         | Resolution                                                                                                 |
| -------------------------------------------------------------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `Authentication failed`                                        | Wrong credentials or missing `read` on the target database.          | Verify the projected secret and grant `read` (or `readAnyDatabase`).                                       |
| `not authorized on admin to execute command { listDatabases }` | The user lacks `clusterMonitor`.                                     | Grant `clusterMonitor` on `admin` so Nilus can enumerate databases.                                        |
| Collections appear with missing fields                         | The document sample did not include documents carrying those fields. | Expected for heterogeneous collections; increase the sample where supported, or accept the inferred shape. |
| `classification` / `profiler` completed with no tags or stats  | `database_filter` named a Mongo database.                            | Use `schema_filter` with that name instead.                                                                |
| Lineage / usage empty                                          | Not supported for MongoDB.                                           | Expected. MongoDB has no query-history or view surface. Use `metadata` + `profiler` + `classification`.    |
| TLS handshake errors                                           | Server enforces TLS but the URI omits `sslmode`.                     | Add `?sslmode=require` (or a stricter mode) to the URI.                                                    |

## Related Docs

* [MongoDB](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/mongodb.md): batch row movement out of MongoDB.
* [MongoDB](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/mongodb.md): log-based change capture.
* [Metadata sources](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources.md): all metadata-capable sources and how to scope extraction.
* [Metadata pipelines](https://v2.dataos.info/references/resources/nilus/metadata-pipelines.md): the conceptual model.
* [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md): the `spec.type: metadata` contract and DAG anatomy.
* [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md): ready-to-edit YAML.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/mongodb.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
