> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/pipeline-optimization.md).

# Pipeline optimization

Symptom-to-page routing table for Nilus pipeline optimization, plus the measurement loop and common tuning knobs shared across guides.

Pipeline optimization is a measured change to improve one of three outcomes: shorter runtime, lower resource use, or a better destination layout.

## Optimize only with a symptom

| Symptom                                                              | First pages                                                                                                                   |
| -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Pipeline takes too long                                              | [Optimizing for time](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-time.md)         |
| Pipeline uses too much CPU or memory                                 | [Optimizing for resource](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-resource.md) |
| CDC lag, snapshot OOM, or Debezium heap pressure                     | [Optimizing CDC](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-cdc.md)                   |
| Validating, testing, or doing a partial backfill                     | [Optimizing for time](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-time.md)         |
| Destination has too many files, poor partitioning, or costly queries | [Optimize sink datasets](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets.md)   |
| Pipeline fails or produces unexpected data                           | [Troubleshooting](https://v2.dataos.info/references/resources/nilus/troubleshooting.md)                                       |

## Measurement loop

1. Capture the current run: status, duration, records processed, CPU, memory, source object, destination, and config.
2. Pick one hypothesis: extraction bottleneck, load bottleneck, destination layout, source throttling, or resource pressure.
3. Change one knob.
4. Run the same workload again.
5. Compare duration, throughput, cost, and destination correctness.
6. Keep the change only if it improves the symptom without hiding a correctness issue.

## Common knobs

| Knob                     | Placement           | Use for                                                                                                                                                                                                                              |
| ------------------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `page_size`              | `source.options`    | Reducing memory pressure or API/page overhead.                                                                                                                                                                                       |
| `extract_parallelism`    | `source.options`    | Increasing supported extraction concurrency.                                                                                                                                                                                         |
| `loader_file_size`       | `sink.options`      | Reducing small files or loader batch overhead.                                                                                                                                                                                       |
| `partition_by`           | `sink.options`      | Improving Lakehouse query pruning.                                                                                                                                                                                                   |
| `cluster_by`             | `sink.options`      | Improving locality for common filters where supported.                                                                                                                                                                               |
| `LOAD__WORKERS`          | Runtime environment | Increasing destination load worker concurrency for large writes.                                                                                                                                                                     |
| `NORMALIZE__WORKERS`     | Runtime environment | Increasing normalization worker concurrency for wide or nested data. Keep the default on **CDC** pipelines.                                                                                                                          |
| `NILUS_JVM_HEAP_PERCENT` | Runtime environment | CDC only: share of pod memory given to the embedded Debezium JVM. See [Optimizing CDC](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-cdc.md) for the tested 12 GiB / 30% / 500k–700k reference. |

Set runtime environment variables under `runtime.env`.

For the full knob reference, including placement, defaults, valid values, and symptoms, see [Optimizing for Time](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-time.md) (throughput and validation-sampling knobs) and [Optimizing for Resource](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-resource.md) (memory/CPU and width-sampling knobs). For CDC heap and queue tuning, see [Optimizing CDC](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-cdc.md). For how the destination dataset is physically laid out, see [Optimize Sink Datasets](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets.md).

## Recommended profiles

Use these as starting points. Tune each profile for available CPU, memory, and source limits.

{% hint style="info" %}
For Lakehouse (Iceberg) destinations, table shape and query patterns drive file sizing. See [Tuning large Lakehouse (Iceberg) loads](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-lakehouse-iceberg-loads.md) for benchmark-backed configurations.
{% endhint %}

### Faster: throughput-oriented

For large lakehouse or warehouse loads where wall-clock time matters more than resource cost.

```yaml
spec:
  source:
    address: dataos://postgresdepot
    options:
      source_table: public.events
      page_size: 100000
      extract_parallelism: 8
  sink:
    address: dataos://lakehouse
    options:
      dest_table: analytics.events_raw
      incremental_strategy: append
      loader_file_format: parquet
      loader_file_size: 300000
  runtime:
    env:
      LOAD__WORKERS: "8"
      NORMALIZE__WORKERS: "2"
      LOAD__PARALLELISM_STRATEGY: "parallel"
```

### Lower-resources: memory/CPU-conservative

For shared runtimes, tight pod budgets, or pipelines that previously OOMKilled.

```yaml
spec:
  source:
    address: dataos://postgresdepot
    options:
      source_table: public.events
      page_size: 25000
      extract_parallelism: 1
  sink:
    address: dataos://lakehouse
    options:
      dest_table: analytics.events_raw
      incremental_strategy: append
      loader_file_format: parquet
      loader_file_size: 500000
  runtime:
    env:
      LOAD__WORKERS: "1"
      NORMALIZE__WORKERS: "1"
      LOAD__PARALLELISM_STRATEGY: "sequential"
```

### Balanced: default starting point

The shape of most well-behaved pipelines.

```yaml
spec:
  source:
    address: dataos://postgresdepot
    options:
      source_table: public.events
      page_size: 50000
      extract_parallelism: 5
  sink:
    address: dataos://lakehouse
    options:
      dest_table: analytics.events_raw
      incremental_strategy: append
      loader_file_format: parquet
      loader_file_size: 150000
  runtime:
    env:
      LOAD__WORKERS: "5"
      NORMALIZE__WORKERS: "1"
      LOAD__PARALLELISM_STRATEGY: "parallel"
```

## Sizing playbook

Use this process for a new pipeline with unfamiliar data.

1. **Baseline.** Run once with defaults and conservative compute. Record runtime, peak CPU/memory, network throughput, and destination file count. If the run gets OOMKilled, prioritize memory mitigations: lower `extract_parallelism`, lower `LOAD__WORKERS`, lower `page_size` to 25k, and keep `loader_file_size` large.
2. **Size files first.** Estimate `row_bytes ≈ raw_table_size_bytes / total_rows`. Target Parquet files of 128–512 MB. Use `loader_file_size ≈ target_file_bytes / row_bytes`, then check `expected_files = ceil(total_rows / loader_file_size)`. Aim for 20–200 files for a very large table.
3. **Bound memory via `page_size`.** Estimate page memory as `page_size × row_bytes`. Choose `page_size` so peak RSS stays well under pod memory. Start at 50k–100k; drop to 25k–50k for OOM-prone or wide-row pipelines.
4. **Set parallelism from available CPU.** `extract_parallelism` and `LOAD__WORKERS` near 60–80% of allocatable vCPU is a good starting point. Keep `NORMALIZE__WORKERS` low (1–2). On **CDC** pipelines, leave `NORMALIZE__WORKERS` and `LOAD__WORKERS` at defaults. See [Optimizing CDC](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-cdc.md). Use `parallel` for lakehouse and most warehouses; use `sequential` for fragile or strictly locked destinations.
5. **Validate on a slice.** Use `sql_limit` or a tight `interval_start`/`interval_end` window to confirm sizing, memory headroom, and file count on a partial run before scaling to the full dataset. See [Optimizing for time](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-time.md).
6. **Iterate by bottleneck:**
   * Long commit tail → raise `loader_file_size`.
   * Low CPU, long runtime → raise `extract_parallelism` and/or `LOAD__WORKERS`.
   * Source throttling → lower `extract_parallelism` or `page_size`.
   * Memory spikes → lower parallelism first; only then lower `page_size`.
7. **Add partitioning last.** Align `partition_by` with consumer queries. Confirm each partition still hits a healthy file size. High-cardinality partitions that produce tiny files cost more than they save. See [Shape knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-shape-knobs.md).

## Related docs

* [Observability](https://v2.dataos.info/references/resources/nilus/observability.md)
* [Optimizing for time](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-time.md)
* [Optimizing for resource](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-resource.md)
* [Optimizing CDC](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-cdc.md)
* [Optimize sink datasets](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/pipeline-optimization.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
