> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/move/mode/batch.md).

# Batch

Move data on a schedule or on demand with a Nilus batch pipeline.

Batch movement extracts data from a source, normalizes it, and loads it into your engine in chunks, either on a schedule or on demand (manually triggered). It is the default type and the widest in source support.

## When to use batch

* Periodic updates (hourly, daily, weekly) are acceptable.
* You need a full reload, or an incremental extraction driven by a timestamp or cursor.
* The source is a SaaS API, an analytics warehouse, object storage, or a file system.

## Before you begin

Ensure you have:

* A Depot for the source, or an approved connection with secrets projected via `spec.use.projection`.
* A Depot or connection for the destination.
* A compute profile, read access to the source, and write access to the destination.

## 1. Choose your source

Batch supports the widest range of sources of any move type. See [supported sources and destinations](https://v2.dataos.info/build/move/supported-sources-and-destinations.md) for the summary, and [supported batch sources](https://v2.dataos.info/references/resources/nilus/batch/batch-sources) in References for the full, current list plus per-connector prerequisites and options.

## 2. Write the manifest

Create a `nilus` resource with `spec.type: batch`:

```yaml
name: ${{pipeline-name}}
version: v1alpha
type: nilus
tags:
  - nilus-batch
description: ${{description}}
spec:
  type: batch
  compute: ${{compute-profile}}
  logLevel: INFO
  schedule:                       # optional; omit to run on demand (manually triggered)
    crons:
      - "0 2 * * *"              # daily at 02:00 UTC
    timezone: UTC
    concurrencyPolicy: Forbid
  source:
    address: dataos://${{source-depot}}?purpose=ro
    options:
      source_table: "${{schema.table}}"
      incremental_key: updated_at  # optional; for delta loads
      primary_key: id              # required for merge
  sink:
    address: dataos://${{sink-depot}}?purpose=rw
    options:
      dest_table: ${{schema.table}}
      incremental_strategy: merge   # with incremental_key, use merge or append; replace keeps only the latest delta
```

Pick the sink strategy that matches your refresh:

| Strategy  | Use it when                                                        |
| --------- | ------------------------------------------------------------------ |
| `append`  | Add new rows without touching existing ones                        |
| `replace` | Drop and reload the destination on every run                       |
| `merge`   | Upsert by `primary_key` (requires `primary_key` in source options) |

For the full attribute reference, see [batch configuration](https://v2.dataos.info/references/resources/nilus/batch/pipeline-config) in References.

## 3. Apply and confirm

```bash
dataos-ctl resource apply -f ${{path-to-manifest.yaml}}
dataos-ctl resource get -t nilus -n ${{pipeline-name}}
```

A STATUS of `active` means the resource was accepted; check RUNTIME to confirm the pipeline is running. When the data has landed, [explore the moved data](https://v2.dataos.info/build/move/explore-moved-data.md) to confirm it.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/build/move/mode/batch.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
