> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-cdc.md).

# Optimizing CDC

Tune pod memory, Debezium queue depth, and JVM heap for CDC pipelines.

Use this guide when a CDC pipeline is correct but needs more (or less) memory, queue depth, or heap for the embedded Debezium JVM. Batch and stream pipelines do not run that JVM and ignore `NILUS_JVM_HEAP_PERCENT`.

CDC resource pressure is a **split** between two runtimes in the same pod:

* The **Debezium JVM** holds the change-event queue (`max.batch.size`, `max.queue.size`, `max.queue.size.in.bytes`).
* **Nilus / dlt** uses the remaining pod memory for extract, normalize, load, native libraries, and destination writes.

Tune those together. Raising the CDC queue without giving the JVM more heap, or giving the JVM most of the pod, will OOM or stall.

{% hint style="info" %}
The numbers below are a **tested customer-scale reference** from MongoDB CDC load tests, not a universal formula. Event size, burst length, destination throughput, and snapshot volume can change the envelope you need.
{% endhint %}

## Tested customer-scale reference

Start here for high-volume Mongo CDC. Product default heap remains **25%**; this profile raises it to **30%** together with large Debezium queues.

| Knob                             | Tested value                                                      |
| -------------------------------- | ----------------------------------------------------------------- |
| `spec.resources.limits.cpu`      | `2000m`                                                           |
| `spec.resources.limits.memory`   | `12Gi`                                                            |
| `NILUS_JVM_HEAP_PERCENT`         | `30`                                                              |
| `spec.source.cdc.max.batch.size` | `500000`                                                          |
| `spec.source.cdc.max.queue.size` | `700000`                                                          |
| Nilus / dlt worker env vars      | **defaults** (`LOAD__WORKERS`, `NORMALIZE__WORKERS`, and similar) |

On a 12 GiB pod, 30% heap is about **3.6 GiB** for the JVM and about **8.4 GiB** left for Nilus / dlt, native libraries, and the rest of the container. Kubernetes still enforces the **total** memory limit.

This reference completed:

* About **1 KiB** events at **500 events/second** for two hours
* About **16 KiB** events at **100 events/second** for 12 hours
* Idle recovery with bounded memory, exact source-to-destination count match, and no OOM, unexplained restart, or connector disconnect

Raising CPU from `2000m` to `4000m` did not improve the rate-capped 1 KiB workload. Keep `2000m` unless you see CPU saturation.

`500000` / `700000` is the **validated customer-scale** batch/queue pair. A smaller pair (`250000` / `350000`) also completed the 1 KiB / 500 events/second run, but peak queue occupancy stayed far below those caps, so the smaller pair is **not** established as enough for bursts or destination backpressure. Do not treat `500000` / `700000` as the proven minimum.

```yaml
name: mongo-cdc-events
version: v1alpha
type: nilus
spec:
  type: cdc
  compute: universe-compute
  resources:
    limits:
      cpu: "2000m"
      memory: "12Gi"
  use:
    projection:
      projections:
        envVars:
          - key: NILUS_JVM_HEAP_PERCENT
            template: "30"
  source:
    address: dataos://mongocdc?purpose=rw
    options:
      strategy: flatten
    cdc:
      collection.include.list: "app.events"
      topic.prefix: "cdc_events"
      heartbeat.interval.ms: 10000
      max.batch.size: 500000
      max.queue.size: 700000
  sink:
    address: dataos://lakehouse?purpose=rw
    options:
      dest_table: raw.events
      incremental_strategy: append
```

Confirm the heap in the pipeline log:

```
Configured JVM MaxRAMPercentage=30%
```

## Do not use these on the same workload

| Configuration                                                       | What happened                                                                                                                                                                                                             |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `max.batch.size: 2048` / `max.queue.size: 3000`                     | Stable, but only about **105 events/second**, with a large leftover backlog. Too small for a 500 events/second stream.                                                                                                    |
| About **16 KiB** events at **500 events/second**                    | Repeated Java heap OOM and restart cycles. Raising heap percent did **not** make this rate stable. The tested stable point for \~16 KiB events is **100 events/second**. The boundary between 100 and 500 is unmeasured.  |
| 8 GiB memory limit                                                  | Passed 1 KiB at 500 events/second, but used about **95.6%** of the JVM heap ceiling. Not the general baseline.                                                                                                            |
| `NORMALIZE__WORKERS=20` (or other tested dlt concurrency overrides) | Exact delivery, then **clean-exit service restarts**. Leave dlt worker env vars at defaults on CDC.                                                                                                                       |
| `max.queue.size.in.bytes` of **256 MiB**                            | Hit the cap and raised CDC lag on the 1 KiB workload. A **1 GiB** cap completed that run with lower lag. There is **no** universal byte-cap value yet. Size it from serialized event size and burst duration, or omit it. |

{% hint style="warning" %}
**`Please note:`**

* `2048` / `3000` cannot sustain \~500 events/s
* \~16 KiB events at 500 events/s OOMed; tested stable point for that size is 100 events/s
* 8 GiB is not the general baseline
* `NORMALIZE__WORKERS=20` caused clean-exit restarts
* `500000` / `700000` is a validated reference, not the proven minimum
  {% endhint %}

## Symptoms

| Symptom                                                        | Likely cause                                                                                                                                | First response                                                                                   |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| Pod OOMKilled during snapshot or a burst                       | JVM heap + dlt buffers exceed the memory limit.                                                                                             | Raise pod memory toward the 12 GiB reference, or lower `NILUS_JVM_HEAP_PERCENT` / queue sizes.   |
| Java heap OOM while event rate is high and documents are large | Event size × rate exceeds the heap that can hold the queue.                                                                                 | Drop the ingest rate, or move toward the 12 GiB / 30% reference. Do not only raise heap percent. |
| CDC lag grows while CPU is low                                 | Queue too small for the event rate (for example 2048 / 3000 on a 500 events/second stream), or heap too small to hold the configured queue. | Move toward `max.queue.size: 700000` **and** heap 30%, with 12 GiB for the pod.                  |
| Heap committed sits at the max while dlt OOMs                  | Heap percent is too high for the pod.                                                                                                       | Lower `NILUS_JVM_HEAP_PERCENT` so dlt keeps a larger share.                                      |
| Service restarts with a clean exit after raising dlt workers   | CDC does not tolerate the tested normalize/load concurrency overrides.                                                                      | Remove `NORMALIZE__WORKERS` / `LOAD__WORKERS` overrides; keep dlt defaults.                      |
| Snapshot is slow, then streaming is fine                       | Snapshot batch/queue is oversized for the pod.                                                                                              | Land the snapshot on the 12 GiB / 30% envelope before raising queues further.                    |

## `NILUS_JVM_HEAP_PERCENT`

| Goes under                                            | Default                  | Valid values                                                                                  |
| ----------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------- |
| Runtime env (`spec.use.projection` or equivalent env) | `25` (25% of pod memory) | Integer percentage of the **pod memory limit** assigned to the embedded Debezium JVM max heap |

The JVM max heap is `pod_memory_limit × NILUS_JVM_HEAP_PERCENT / 100`. The rest of the pod stays available for Nilus/dlt. Invalid values fail fast at startup rather than running with a silent default.

Approximate heap ceilings on a **12 GiB** limit:

| `NILUS_JVM_HEAP_PERCENT` | Max Java heap | Left outside the Java heap |
| ------------------------ | ------------- | -------------------------- |
| `25` (product default)   | \~3 GiB       | \~9 GiB                    |
| `30` (tested reference)  | \~3.6 GiB     | \~8.4 GiB                  |
| `40`                     | \~4.8 GiB     | \~7.2 GiB                  |
| `50`                     | \~6 GiB       | \~6 GiB                    |

Memory outside the Java heap is shared by the Python engine, Java non-heap, thread stacks, direct buffers, native libraries, and allocator overhead. The JVM may use less than its maximum; the container limit is still the hard cap.

## Tuning order

1. **Size the pod first.** For high-volume CDC, start from the tested envelope: `limits.memory: 12Gi` and `limits.cpu: 2000m`. Heap percent is a *share* of the memory limit.
2. **Leave the default heap (25%)** on small or unknown workloads. Raise to **30%** when you also set the large Debezium queues below.
3. **Use the tested queues for high event rates.** `max.batch.size: 500000` and `max.queue.size: 700000` live under `spec.source.cdc`. Do not keep connector-default 2048 / 3000 (or 2048 / 8192) if you need hundreds of events per second.
4. **Keep a dlt share.** After the heap slice, remaining pod memory must still cover normalize/load. If destination writes OOM while the JVM log shows a healthy heap, *lower* the heap percent rather than raising the pod forever.
5. **Keep dlt worker env vars at defaults on CDC.** Do not copy batch/lakehouse `NORMALIZE__WORKERS` or `LOAD__WORKERS` profiles onto a CDC pipeline.
6. **Change one variable per run.** Heap, queue, event size, and destination throughput interact. Compare lag, throughput, RSS, and source vs dest counts on the same workload before stacking changes.

Connector-specific CDC keys (`heartbeat.interval.ms`, include lists, snapshot mode) stay under `spec.source.cdc`. See [CDC pipeline config](https://v2.dataos.info/references/resources/nilus/cdc/pipeline-config.md).

## Avoid these shortcuts

* Do not set `NILUS_JVM_HEAP_PERCENT` near `100`. The JVM would starve Nilus/dlt in the same pod.
* Do not raise `max.queue.size` without raising heap or pod memory. The queue is stored on the JVM heap.
* Do not treat heap percent as a substitute for `spec.resources`; a 30% slice of a 512Mi pod is still a small heap.
* Do not assume a larger heap percent will absorb **16 KiB** events at **500 events/second**. That combination OOMed in testing.
* Nested **arrays** in CDC payloads become **child tables** when `max_table_nesting` is raised. That extra table is a schema-shape concern, not a heap fix. See [Shape knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-shape-knobs.md).

## Related Docs

* [Pipeline optimization](https://v2.dataos.info/references/resources/nilus/pipeline-optimization.md)
* [Optimizing for resource](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-resource.md)
* [CDC sample configs](https://v2.dataos.info/references/resources/nilus/cdc/sample-configs.md)
* [CDC sources](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-cdc.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
