> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus.md).

# Nilus

Nilus is DataOS's unified data movement framework. It moves data from databases, SaaS APIs, event streams, and data warehouses into analytical destinations.

Nilus connects sources to analytical destinations through a single, consistent pipeline model. Whether you are copying a relational table on a schedule, streaming low-latency database changes, consuming a Kafka topic, or cataloging a data warehouse's schema, every Nilus pipeline is authored as a `type: nilus` resource with the same basic shape.

## How it works

A Nilus pipeline has two required pieces: a **source address** (where data comes from) and a **sink address** (where it goes). The runtime handles credentials, schema inference, state, retries, and loading.

For what the runtime actually does with that source and sink address, see [Architecture](/references/resources/nilus/concepts/architecture-and-mechanism.md).

**Example Config YAML**

```yaml
version: v1alpha
name: customers-batch
type: nilus
spec:
  type: batch
  compute: universe-compute
  source:
    address: dataos://postgressource?purpose=ro
    options:
      source_table: public.customers
      incremental_key: updated_at
      primary_key: id
  sink:
    address: dataos://warehouse?purpose=rw
    options:
      dest_table: raw.customers
      incremental_strategy: merge
```

Source and sink addresses are either a `dataos://` Depot reference (credentials resolved automatically) or a direct connector URI with secrets projected via `spec.use.projection`, depending on the connector. Some connectors are direct URI only (for example, Databricks sources and IBM DB2 CDC sources). See [Secrets and projections](/references/resources/nilus/concepts/secrets-and-projections.md) for the full pattern.

## Pipeline types

Choose a pipeline type based on how your source delivers data. Every type follows the same Extract, Normalize, and Load (ENL) lifecycle.

| Type         | When to use                                                                                                                                          | spec.type  |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| **Batch**    | Periodic table, view, file, SaaS, or warehouse loads, for sources where scheduled or incremental cursor extraction is sufficient                     | `batch`    |
| **CDC**      | Near-real-time row-level change capture from a database log, for sources that expose a durable change log                                            | `cdc`      |
| **Stream**   | Continuous consumption from an event-streaming system (Kafka or NATS+JetStream) as a long-running consumer service                                   | `stream`   |
| **Metadata** | Catalog extraction from a data source (schemas, lineage, column profiles, and usage) published into the DataOS metadata catalog without copying rows | `metadata` |

## Connectors

Nilus connectors support ingesting data from sources, writing data to destinations, and extracting metadata from connected systems.

### **Sources**

Includes relational and document databases, cloud data warehouses, data lakes, SaaS applications, analytics platforms, spreadsheets, and event-streaming systems.

→ [Browse all sources](/references/resources/nilus/sources.md)

#### Sources by pipeline type

Pipeline support varies by source.

* [Batch sources](/references/resources/nilus/batch/batch-sources.md)
* [CDC sources](/references/resources/nilus/cdc/cdc-sources.md)
* [Stream sources](/references/resources/nilus/stream/stream-sources.md)
* [Metadata sources](/references/resources/nilus/metadata-pipelines/metadata-sources.md)

#### **Nilus is extensible**

Build a custom source for any system without a first-party connector.

* [Custom sources](/references/resources/nilus/batch/custom-sources.md)

### **Destinations**

Includes data lakehouses, cloud data warehouses, and relational or document databases. All destinations support batch, CDC, and stream writes.

→ [Browse all destinations](/references/resources/nilus/destinations.md)

## Write strategies

Every sink uses one of three `incremental_strategy` values, which control how repeated runs interact with the destination table.

| Strategy  | Behavior                                                       | Best for                                       |
| --------- | -------------------------------------------------------------- | ---------------------------------------------- |
| `replace` | Drop and re-create the destination table on every run          | Small dimension tables, snapshots, SaaS data   |
| `append`  | Add new rows; never update or delete                           | Immutable events, logs, telemetry              |
| `merge`   | Upsert on `primary_key`: update existing rows, insert new ones | Mutable entity tables, CDC current-state views |

{% hint style="info" %}

### Incremental extraction

For sources that grow over time, set `incremental_key` to a monotonically increasing column (typically `updated_at` or a sequence). Nilus records the last seen value after each successful run and uses it as the lower bound for the next extraction, so only new or changed rows are read.

Set `primary_key` alongside `incremental_key` when using `merge` strategy to tell Nilus which column identifies a row.
{% endhint %}

## Access requirements

Before creating or managing a Nilus resource, confirm you have what you need.

Every Nilus pipeline needs two separate sets of permissions in place before it can run — one governs the pipeline object inside DataOS, the other governs the credential it connects with on the source and destination systems. Confirm both before you start:

* **DataOS access:** creating a `type: nilus` resource, and using the Compute, Depot, and Secret it depends on, requires specific Tenant roles and `Can Use` grants. For more details, see [Nilus: Access control](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/nilus-access-control).
* **Source and destination privileges:** the credential the pipeline actually connects with needs its own SQL `GRANT`s or IAM roles on the source and destination systems, scoped to what that pipeline's mode (batch, CDC, stream, or metadata) does. For more details, see [Roles and permissions](/references/resources/nilus/roles-and-permissions.md).

> These are independent grants. Having one does not imply the other. DataOS will let you create the pipeline resource without the second, and it will simply fail to connect when it runs.

## Build a data movement pipeline

Find your source in Sources, then choose its pipeline type. Each guide explains when to use it, its core capabilities, runtime flow, and operating constraints. Use the configuration reference for supported fields, then adapt a sample manifest.

* [Batch data movement](/references/resources/nilus/batch.md)
* [Change data capture](/references/resources/nilus/cdc.md)
* [Stream data movement](/references/resources/nilus/stream.md)

## Build a metadata pipeline

Metadata pipelines publish schemas, lineage, profiles, and usage into the DataOS catalog without copying a single row. Useful on day one, before you have moved anything.

* [Metadata pipelines overview](/references/resources/nilus/metadata-pipelines.md)
* [Metadata pipeline config](/references/resources/nilus/metadata-pipelines/pipeline-config.md)
* [Metadata sample configs](/references/resources/nilus/metadata-pipelines/sample-configs.md)

## Operate & observe

Once a pipeline is running, these are the places to check whether it's healthy, tune it, and diagnose for any issues.

* [Observability](/references/resources/nilus/observability.md) - run history, metrics, and dashboards that tell you whether the last run did what you expected
* [Pipeline optimization](/references/resources/nilus/pipeline-optimization.md)- maps performance and cost symptoms to the settings that address them.
* [Troubleshooting](/references/resources/nilus/troubleshooting.md) - triage path for failures, plus runbooks for the errors that come up most


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.