> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/concepts/resources/nilus/batch.md).

# Batch Data Movement

Not every source can expose a continuous stream of changes. Many systems are better handled through periodic extraction: relational tables, SaaS APIs, file stores, spreadsheets, and operational exports. That is where batch data movement fits. Instead of chasing every individual event, Nilus runs a bounded extract and lands the resulting dataset into the chosen destination.

## Overview

Nilus models batch ingestion as `type: nilus` with `spec.type: batch`. The manifest declares a source, a sink, and any optional runtime or scheduling controls, while Nilus handles the execution shell, state, and destination loading path. Batch is the right fit when the source is snapshot-oriented, when freshness requirements are measured in minutes or hours rather than seconds, or when the simplest operational model is the best one. For the full list of `source.options` and `sink.options` keys (tables, defaults, and when to use each), see [Understanding Batch Pipeline Config](/concepts/resources/nilus/batch/pipeline-config.md). That page is the single reference for batch manifest authoring; this page explains the conceptual model only.

## Core Capabilities

Nilus batch mode is built around three main patterns:

### Wide Connector Coverage

* Batch ingestion is the primary mode for many SaaS, API, spreadsheet, object-store, and relational connectors.
* `source_table` stays connector-specific: it might mean `schema.table`, an API object, a sheet identifier, or another logical resource name.
* Nilus keeps the same manifest shape across those connector types even when the extraction behavior differs.

### Snapshot and Incremental Patterns

* Some batch sources are naturally snapshot-oriented and are best paired with `replace`.
* Some batch sources support incremental extraction through an `incremental_key`.
* Others manage refresh behavior inside the connector itself, so the manifest should avoid forcing a cursor that the connector does not actually honor.

### Flexible Destination Writes

* Batch sinks still use destination-specific behavior such as `append`, `replace`, or `merge`.
* Nilus keeps the source contract and destination contract separate so that source extraction and target write semantics can be tuned independently.
* This makes it easier to document one connector at a time while preserving a common batch authoring model.

## Flow

1. Nilus resolves the source URI or depot reference and loads any required secrets.
2. The batch source extracts rows, objects, or files according to the connector contract.
3. Nilus applies optional shaping such as masking, type overrides, or bounded extraction windows.
4. The sink writes the batch output using the configured destination strategy.

## Constraints

* Batch pipelines trade freshness for simplicity, so they are not a substitute for CDC when near-real-time state matters.
* Large snapshot runs can create bursty source or destination load if schedules and resource sizing are not planned carefully.
* Incremental behavior is connector-dependent; the source page should always be treated as the final reference for whether `incremental_key` is valid.
* Destination behavior still depends on the chosen sink, so the sink page remains the source of truth for target-specific options.

## Related Docs

* [Understanding Batch Pipeline Config](/concepts/resources/nilus/batch/pipeline-config.md)
* [Batch Sample Configs](/concepts/resources/nilus/batch/sample-configs.md)
* [Sources](/concepts/resources/nilus/sources.md)
* [Destinations](/concepts/resources/nilus/destinations.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/concepts/resources/nilus/batch.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.