> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/concepts/resources/nilus/cdc/service-config.md).

# CDC Service Config

> Nilus CDC pipelines are authored as `type: nilus` resources with `spec.type: cdc`.

This page explains the current Nilus CDC pipeline config shape. Author the `nilus` resource and keep connector-specific CDC settings under `spec.source.cdc`.

<details>

<summary>Example Config YAML</summary>

```yaml
name: nilus-mongo-cdc
version: v1alpha
type: nilus
spec:
  type: cdc
  compute: universe-compute
  logLevel: INFO
  source:
    address: dataos://testingmongocdc
    options:
      strategy: flatten
      max_table_nesting: 0
    cdc:
      collection.include.list: "sample.unnest"
      topic.prefix: "cdc_changelog"
      transforms.unwrap.array.encoding: array
  sink:
    address: dataos://testinglh
    options:
      dest_table: mdb_test
      incremental_strategy: append
      aws_region: us-west-2
```

</details>

## Configuration elements

Fields are grouped below by function.

### 1. Pipeline metadata

| Field         | Description                                  |
| ------------- | -------------------------------------------- |
| `name`        | Unique pipeline identifier.                  |
| `version`     | Pipeline config version.                     |
| `type`        | Must be `nilus` for Nilus-managed pipelines. |
| `description` | Optional human-readable summary.             |

### 2. Nilus spec

| Field       | Description                                                             |
| ----------- | ----------------------------------------------------------------------- |
| `spec.type` | Must be `cdc` for change-data-capture pipelines.                        |
| `compute`   | Compute profile used to run the pipeline.                               |
| `logLevel`  | Optional log level such as `DEBUG`, `INFO`, `WARNING`, or `ERROR`.      |
| `runAsUser` | Optional runtime identity. When omitted, Nilus uses the resource owner. |
| `resources` | Optional CPU and memory requests or limits for the generated service.   |
| `use`       | Optional secret projection rules for direct-URI CDC connectors.         |
| `source`    | Defines how Nilus reads the source system.                              |
| `sink`      | Defines where Nilus writes the captured records.                        |

### 3. Source

The `source` section defines how Nilus connects to the upstream system.

```yaml
source:
  address: dataos://testingmongocdc
  options:
    strategy: flatten
    max_table_nesting: 0
  cdc:
    collection.include.list: "sample.unnest"
    topic.prefix: "cdc_changelog"
    transforms.unwrap.array.encoding: array
```

* `address` can be a `dataos://` depot reference or a direct connection URI, depending on the connector.
* `options.strategy` controls the event shape written downstream. `flatten` is the common default for row-shaped records.
* `options.max_table_nesting` is a universal source option (not CDC-specific). It caps how deeply nested fields are flattened, useful for MongoDB documents and JSON-shaped CDC payloads.
* `cdc` holds source-specific CDC settings such as include lists, snapshot controls, heartbeats, and queue sizing.

Use `spec.use.projection` when the CDC connector is configured through a direct URI with placeholders:

```yaml
spec:
  use:
    projection:
      secrets:
        - id: engineering:pg-secret
          contextAlias: pgsecret
      projections:
        envVars:
          - key: PG_USERNAME
            template: "{{ secrets['pgsecret'].username | base64_decode }}"
          - key: PG_PASSWORD
            template: "{{ secrets['pgsecret'].password | base64_decode }}"
  source:
    address: postgresql://{PG_USERNAME}:{PG_PASSWORD}@postgres.example.com:5432/commerce
```

#### `source.options`

A small set of universal source options apply to CDC pipelines on top of the connector-specific `cdc:` block.

| Option              | Required | What it does                                                                                                                                                                                                                                                                      | Typical shape                                    |
| ------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| `strategy`          | No       | Event shape written downstream. `flatten` emits the row payload (the common default); `changelog` preserves the full CDC event structure.                                                                                                                                         | `flatten`                                        |
| `max_table_nesting` | No       | Schema hint that caps how deeply nested fields are flattened. Nodes deeper than this level are loaded as a struct or JSON column instead of being expanded into their own table. Most relevant for MongoDB documents and JSON-shaped CDC payloads. `0` keeps the runtime default. | `0` (default): increase to flatten nested fields |

#### `source.cdc`

Connector-specific CDC settings live here. The MongoDB example uses:

| Option                             | Description                                                                     | Required |
| ---------------------------------- | ------------------------------------------------------------------------------- | -------- |
| `collection.include.list`          | List of MongoDB collections to monitor, using fully qualified namespace values. | Yes      |
| `topic.prefix`                     | Logical stream prefix used for connector identity and downstream naming.        | Yes      |
| `transforms.unwrap.array.encoding` | Controls how arrays are represented in the emitted records.                     | Optional |

Connector-specific CDC keys vary by source. Common families include:

* include-list settings such as `table.include.list` or `collection.include.list`
* identity settings such as `topic.prefix` and connector-specific slot or server names
* snapshot, heartbeat, queue, polling, and event-shape settings

Document the source-specific CDC keys in the connector page when they are not universal.

### 4. Sink

The `sink` section defines where the captured change records are written.

```yaml
sink:
  address: dataos://testinglh
  options:
    dest_table: mdb_test
    incremental_strategy: append
    aws_region: us-west-2
```

| Field                   | Description                                                                                  |
| ----------------------- | -------------------------------------------------------------------------------------------- |
| `address`               | Target destination or depot reference.                                                       |
| `dest_table`            | Schema and table to write captured records into.                                             |
| `incremental_strategy`  | Write mode for CDC pipelines. `append` is the common choice.                                 |
| Additional sink options | Destination-specific settings such as region, catalog, warehouse, or authentication details. |

### 5. Execution model

The important authoring consequence is:

* author the `nilus` resource in `v1alpha`
* keep connector-specific CDC keys under `spec.source.cdc`
* keep source requirements and sink behavior aligned with the connector pages

## Validation notes

* Keep `topic.prefix` stable after the first production deployment so offsets, schema history, and downstream naming remain consistent.
* Use narrow include lists during onboarding rather than capturing an entire database by default.
* Put connector-specific tuning keys such as snapshot, heartbeat, queue, or polling settings under `spec.source.cdc`. Universal source options like `strategy` and `max_table_nesting` belong under `spec.source.options`.
* Use `spec.use.projection` only when the source or sink URI is assembled directly. Use `dataos://...purpose=` when the connection should come from a depot.
* Keep the sink page as the source of truth for destination-specific write options.

## Related docs

* [CDC Sample Configs](/concepts/resources/nilus/cdc/sample-configs.md)
* [Secrets and Projections](/concepts/resources/nilus/concepts/secrets-and-projections.md)
* [Understanding Batch Pipeline Config](/concepts/resources/nilus/batch/pipeline-config.md)
* [Understanding Stream Pipeline Config](/concepts/resources/nilus/stream.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/concepts/resources/nilus/cdc/service-config.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.