> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/batch/pipeline-config.md).

# Batch pipeline config

Complete field reference for Nilus batch pipelines: spec structure, all source.options and sink.options, supported values, and annotated examples.

This page explains the current batch pipeline authoring shape. Author the `nilus` resource and keep the operational intent under `spec`.

Nilus batch pipelines are created as `type: nilus` resources with `spec.type: batch`.

<details>

<summary>Example Config YAML</summary>

```yaml
name: nilus-batch-example
version: v1alpha
type: nilus
tags:
  - workflow
  - nilus-batch
description: Batch sync from PostgreSQL into a DataOS depot
spec:
  type: batch
  compute: universe-compute
  schedule:
    crons:
      - "30 11 * * *"
    concurrencyPolicy: Forbid
  resources:
    requests:
      cpu: "200m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "2Gi"
  use:
    projection:
      secrets:
        - id: engineering:niluspgsecret
          contextAlias: pgsecret
      projections:
        envVars:
          - key: PG_USERNAME
            template: "{{ secrets['pgsecret'].username | base64_decode }}"
          - key: PG_PASSWORD
            template: "{{ secrets['pgsecret'].password | base64_decode }}"
  source:
    address: postgresql://{PG_USERNAME}:{PG_PASSWORD}@postgres.example.com:5432/postgres
    options:
      source_table: public.customer
  sink:
    address: dataos://niluspgdepot?purpose=rw
    options:
      dest_table: testing_nilus.customer
      incremental_strategy: replace
```

</details>

## Configuration Elements

Fields are grouped below by function.

#### 1. Metadata

| Field         | Description                                           |
| ------------- | ----------------------------------------------------- |
| `name`        | Unique pipeline name.                                 |
| `version`     | Use `v1alpha` for the current Nilus resource shape.   |
| `type`        | Must be `nilus`.                                      |
| `tags`        | Optional labels for search, grouping, and operations. |
| `description` | Optional human-readable summary.                      |

#### 2. Nilus Spec

The `spec` block defines the batch pipeline contract that Nilus validates and then renders into an executable workflow.

| Field       | Required              | Description                                                           |
| ----------- | --------------------- | --------------------------------------------------------------------- |
| `compute`   | Yes                   | Compute profile used to run the batch workload.                       |
| `resources` | No                    | Optional CPU and memory requests or limits.                           |
| `logLevel`  | No                    | Optional log level such as `DEBUG`, `INFO`, `WARNING`, or `ERROR`.    |
| `repo`      | No                    | Optional repository block for repo-backed connector extensions.       |
| `sink`      | Yes for data movement | Destination definition for batch pipelines that write extracted data. |

#### 3. Schedule and Runtime Controls

Use `schedule` only when the batch pipeline should run on a recurring cadence.

```yaml
spec:
  schedule:
    crons:
      - "30 11 * * *"
    timezone: UTC
    endOn: "2026-12-31T23:59:59Z"
    concurrencyPolicy: Forbid
```

* `crons` is an array, not a single `cron` string.
* If `schedule` is omitted, the batch pipeline behaves like an instance that must be triggered manually.
* `concurrencyPolicy` controls overlap handling for scheduled runs.

Runtime settings such as compute, logs, and resource sizing stay under `spec`:

```yaml
spec:
  compute: universe-compute
  logLevel: INFO
  resources:
    requests:
      cpu: "200m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "2Gi"
```

**Persistent volumes and restricted runtimes**

For large Lakehouse/Iceberg loads, or any load that needs durable writable scratch space, attach a DataOS Volume and point Nilus at it with the `DATAOS_PERSISTENT_DIR` environment variable. Nilus then writes its intermediate staging files to that directory instead of the pod's ephemeral filesystem. This prevents disk-exhaustion evictions and out-of-memory failures on big loads. The volume `directory` and `DATAOS_PERSISTENT_DIR` must point to the **same path**:

```yaml
spec:
  use:
    volumes:
      - id: <volume-id>
        directory: /var/dataos/public/nilus_scratch
        readOnly: false
    projection:
      projections:
        envVars:
          - key: DATAOS_PERSISTENT_DIR
            template: "/var/dataos/public/nilus_scratch"
```

* `id` references an existing DataOS **Volume** resource.
* `directory` is the in-container mount path for the volume.
* `readOnly` defaults to `false`; set `true` only for read-only reference mounts.
* `DATAOS_PERSISTENT_DIR` must match `directory` so staging files land on the volume.

See [tuning large Lakehouse (Iceberg) loads](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-lakehouse-iceberg-loads.md) for why this matters and how to size the volume for memory-bound loads.

#### 4. Secrets and Repo Settings

For direct connector URIs, project secrets under `spec.use.projection` and reference them through `{ENV_VAR}` placeholders in `source.address` or `sink.address`.

```yaml
spec:
  use:
    projection:
      secrets:
        - id: engineering:niluspgsecret
          contextAlias: pgsecret
      projections:
        envVars:
          - key: PG_USERNAME
            template: "{{ secrets['pgsecret'].username | base64_decode }}"
```

Use `spec.repo` only when the pipeline depends on a repo-backed extension:

```yaml
spec:
  repo:
    url: https://bitbucket.org/org/custom-connectors
    baseDir: connectors
    secretId: engineering:bitbucketsecret
```

#### 5. Source

The `source` block defines where Nilus reads from. Use a direct connector URI or a `dataos://` depot address.

```yaml
spec:
  source:
    address: postgresql://{PG_USERNAME}:{PG_PASSWORD}@postgres.example.com:5432/postgres
    options:
      source_table: public.customer
      incremental_key: updated_at
```

**Source Fields**

| Field     | Required | Description                                   |
| --------- | -------- | --------------------------------------------- |
| `address` | Yes      | Connector URI or `dataos://` depot reference. |

**`source.options`**

These are the batch-oriented `source.options` keys currently consumed by the Nilus runtime. Some are broadly useful across SQL-style batch connectors, while others are connector-specific tuning knobs.

| Option                | Required | What it does                                                                                                                                                                                                                                                                                                                                      | Typical shape                                |
| --------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- |
| `source_table`        | Yes      | Logical source object. Depending on the connector, this may be a SQL table, a file selector, a sheet name, an API object, or a report definition.                                                                                                                                                                                                 | `public.customer`                            |
| `primary_key`         | No       | Merge key. Write the **source** column name, not snake\_case. Table reads can pick up a table primary key. `query:` never does, so set this for every `query:` merge. See [custom query (SQL sources)](https://v2.dataos.info/references/resources/nilus/batch/custom-sources/custom-query-sql-sources.md).                                       | `id` / `OrderId`                             |
| `incremental_key`     | No       | Cursor field for connectors that support bounded reads. Write the source column name. Some SaaS sources reject it and manage incrementality themselves. Stripe uses `:sync:incremental` on `created` only. See [Stripe](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/stripe.md).                                         | `updated_at` / `LastModifiedDate`            |
| `interval_start`      | No       | Optional lower bound for bounded incremental extraction.                                                                                                                                                                                                                                                                                          | `"2026-01-01T00:00:00Z"`                     |
| `interval_end`        | No       | Optional upper bound for bounded incremental extraction.                                                                                                                                                                                                                                                                                          | `"2026-01-31T23:59:59Z"`                     |
| `type_hints`          | No       | Column type overrides that Nilus maps into the underlying column-hints contract.                                                                                                                                                                                                                                                                  | `{ amount: decimal, created_at: timestamp }` |
| `mask`                | No       | Column masking rules. Nilus expects an object whose values are algorithm strings such as `hash`, `redact`, or `partial:3`.                                                                                                                                                                                                                        | `{ email: hash, phone: partial:3 }`          |
| `extract_parallelism` | No       | Number of extraction jobs to run in parallel for supported connectors.                                                                                                                                                                                                                                                                            | `5`                                          |
| `page_size`           | No       | Rows or items fetched per extraction page for paged connectors.                                                                                                                                                                                                                                                                                   | `50000`                                      |
| `sql_limit`           | No       | Extraction row limit for table reads. **Rejected** on `source_table: "query:…"`; put `LIMIT` in the SQL instead.                                                                                                                                                                                                                                  | `5000`                                       |
| `sql_exclude_columns` | No       | Columns to skip during extraction. The current runtime expects a comma-separated string.                                                                                                                                                                                                                                                          | `debug_payload,raw_html`                     |
| `yield_limit`         | No       | Caps how many extraction pages or batches the source yields before the run stops.                                                                                                                                                                                                                                                                 | `10`                                         |
| `max_table_nesting`   | No       | How deeply nested objects are flattened into parent columns. Default `"0"` keeps nested objects as `json` on the parent. Raise to `"1"` or `"2"` to flatten. Quote the string; an integer is rejected. See [shape knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-shape-knobs.md). | `"0"` (default)                              |

**Source Option Notes**

* Start with `source_table` only, then add `incremental_key` and `primary_key` when the connector and load strategy need them. Write both as **source** names, such as `OrderId` and `LastModifiedDate`.
* `query:` sources never inherit a table primary key. Set `primary_key` for merge, and do not set `sql_limit`. See [custom query (SQL sources)](https://v2.dataos.info/references/resources/nilus/batch/custom-sources/custom-query-sql-sources.md).
* On a DataOS Lakehouse destination, table reads and `query:` write different Iceberg decimal precision and scale for every SQL batch source. Do not switch styles on the same `dest_table`. See [schema evolution](https://v2.dataos.info/references/resources/nilus/concepts/schema-evolution.md).
* Some SaaS sources reject `incremental_key`. Stripe incrementality is `:sync:incremental` on `created` only. Later updates to older charges are not picked up. See [Stripe](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/stripe.md).
* `type_hints` and `mask` are object-valued options. Nilus turns them into the lower-level runtime contract for you. On schema-flexible sources, later extraction pages can add nullable columns in the same run. Pin a sparse, all-null early column with `type_hints`. See [schema evolution](https://v2.dataos.info/references/resources/nilus/concepts/schema-evolution.md).
* Set `SCHEMA_NAMING=direct` through `spec.use.projection.projections.envVars`. Do not set `schema_naming` under `source.options`. `direct` preserves source names such as `pkey`. See [Correctness Knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-correctness-knobs.md) and [Secrets and Projections](https://v2.dataos.info/references/resources/nilus/concepts/secrets-and-projections.md).
* `extract_parallelism`, `page_size`, `sql_limit`, `sql_exclude_columns`, `yield_limit`, and `max_table_nesting` are best treated as tuning knobs, not mandatory pipeline fields.
* `max_table_nesting` only applies to sources that produce nested records. Default `"0"` keeps nested objects as `json`. Raising it flattens objects and can break **arrays** into child tables. See [shape knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-shape-knobs.md).
* Connector pages remain the source of truth for connector-specific `source_table` syntax and any additional connector-only options.

**Direct URI vs Depot Address**

* Use a direct connector URI when you need to assemble the connection explicitly and provide credentials through `spec.use.projection`.
* Use `dataos://<depot>?purpose=<ro|rw>` when the connection should be inferred from a depot.

#### 6. Sink

The `sink` block defines where Nilus writes the batch output.

```yaml
spec:
  sink:
    address: dataos://niluspgdepot?purpose=rw
    options:
      dest_table: testing_nilus.customer
      incremental_strategy: replace
```

**Sink Fields**

| Field     | Required | Description                                     |
| --------- | -------- | ----------------------------------------------- |
| `address` | Yes      | Destination URI or `dataos://` depot reference. |

**`sink.options`**

These are the commonly used batch `sink.options` keys currently wired through the Nilus runtime.

| Option                 | Required | What it does                                                                                                                                                | Typical shape              |
| ---------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- |
| `dest_table`           | Yes      | Target object name in the destination.                                                                                                                      | `analytics.customer`       |
| `incremental_strategy` | Yes      | Write behavior across repeated runs. Accepts only `replace`, `append`, or `merge`. Other values, such as `delete+insert`, `scd2`, or `none`, fail at apply. | `replace`                  |
| `partition_by`         | No       | Destination layout hint for partitioned sinks. This is most relevant for lakehouse-style destinations and should follow the destination page examples.      | structured partition rules |
| `cluster_by`           | No       | Optional clustering hint for destinations that implement clustering or sort-like organization.                                                              | `customer_id`              |
| `loader_file_size`     | No       | Controls how many rows Nilus writes per output file or loader batch.                                                                                        | `100000`                   |

**Sink Option Notes**

* `incremental_strategy` belongs under `sink.options`. Customer YAML accepts `replace`, `append`, or `merge` only.
* `incremental_strategy: merge` with no usable `primary_key` does **not** fail. Nilus logs a warning and falls back to **append**, which duplicates rows. Table reads can pick up a table primary key. `query:` cannot. See [correctness knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-correctness-knobs.md).
* `partition_by` and `cluster_by` are destination-sensitive. Use them only when the destination page explicitly supports them and the access pattern justifies them.
* `loader_file_size` is a performance and file-layout knob. Start with the runtime default and tune gradually.

Destination-specific options such as region overrides and staging locations are documented in the corresponding destination page.

#### 7. Execution Model

The important authoring consequence is:

* Author the `nilus` resource in `v1alpha`
* Keep the operational intent in `spec`
* Treat this page as the primary reference for batch pipeline configuration

## Validation Notes

* Use `type: nilus` with `spec.type: batch` for batch pipeline authoring.
* Put recurring schedule settings under `spec.schedule.crons`, not a legacy top-level workflow schedule block.
* Use `spec.use.projection` for direct connector credentials, use `dataos://...purpose=` when a depot should supply credentials automatically.
* Referenced resources are validated at apply time. `dataos-ctl resource apply` fails fast if the `compute`, source depot, or sink depot does not exist in the tenant, with an error like `invalid resource dependency, not found: depot:<name>`. Create the dependency before applying. See [common troubleshooting guides](https://v2.dataos.info/references/resources/nilus/troubleshooting.md).
* Keep `source_table` semantics connector-aware: `schema.table` for SQL sources, named objects for SaaS sources, and resource-specific identifiers for connectors such as sheets or streams.
* Treat `extract_parallelism`, `page_size`, `sql_limit`, `sql_exclude_columns`, `yield_limit`, `max_table_nesting`, `partition_by`, `cluster_by`, and `loader_file_size` as optional tuning knobs, not baseline required fields.
* The Nilus runtime accepts some additional internal values and destination-specific options, but this page documents the stable batch pipeline authoring surface users should rely on by default.

## Related Docs

* [Batch sample configs](https://v2.dataos.info/references/resources/nilus/batch/sample-configs.md)
* [Secrets and projections](https://v2.dataos.info/references/resources/nilus/concepts/secrets-and-projections.md)
* [CDC pipeline config](https://v2.dataos.info/references/resources/nilus/cdc/pipeline-config.md)
* [Stream pipelines](https://v2.dataos.info/references/resources/nilus/stream.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/batch/pipeline-config.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
