> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/nilus/stream.md).

# Stream pipelines

Conceptual overview of Nilus stream pipelines: connecting to Kafka or NATS+JetStream, consuming records in micro-batches, and writing them to a destination.

Not every source exposes changes through a database log. Event systems such as Kafka and NATS+JetStream publish a continuous sequence of records that must be consumed as they arrive. That is where stream pipelines fit. Nilus connects to the broker, consumes records in micro-batches, and writes each batch to the selected destination.

## Stream **pipeline model**

Nilus models stream ingestion as `type: nilus` with `spec.type: stream`. The manifest declares a source topic or stream, a sink, and consumer settings. Nilus handles the runtime service, offset tracking, and destination loading. Stream suits continuously published events where the destination needs low lag.

For manifest fields and connector options, see [Stream pipeline config](https://v2.dataos.info/references/resources/nilus/stream/pipeline-config.md). For ready-to-use Kafka and NATS+JetStream manifests, see [Stream sample configs](https://v2.dataos.info/references/resources/nilus/stream/sample-configs.md).

## Core capabilities

Stream type is built around the following capabilities:

### Continuous consumption

* Stream pipelines stay connected after startup and pull new records indefinitely when `run_in_loop: true` is set.
* Nilus tracks consumer positions (Kafka consumer-group offsets, NATS JetStream sequence numbers) so a restart resumes from the last committed position without reprocessing old records.
* Consumer identity (`group_id` for Kafka, `durable=` for NATS) must stay stable across deployments for offset continuity to hold.

### Micro-batch model

* Records are pulled from the source in configurable batches before being written to the sink.
* `batch_size` and batch timeout control how many records are accumulated per write and how long Nilus waits to fill a batch on a quiet topic.
* This model balances throughput against write latency and makes it straightforward to tune for different traffic profiles.

### Flexible destination writes

* Stream sinks follow the same destination model as batch and CDC pipelines.
* `incremental_strategy: append` is the default for event-shaped data.
* `merge` is supported when the destination has a stable primary key and deduplication is required.

## Flow

1. Nilus connects to the source broker or cluster using the configured URI and projected credentials.
2. A consumer pulls a micro-batch of records from the topic or stream.
3. Nilus applies any configured masking or shaping to the batch.
4. The sink writes the batch to the destination and Nilus commits the consumer offset.
5. Steps 2–4 repeat continuously while `run_in_loop: true` is set.

## Constraints

* Stream pipelines are long-running services, not cron jobs. Do not add a `spec.schedule` block.
* `run_in_loop: true` must be set explicitly under `source.options`; without it, Nilus exits after the first pull.
* Source offset retention governs how far back a restarted pipeline can resume. If the consumer falls behind beyond the retention window, a re-snapshot may be required.
* Destination behavior still depends on the chosen sink; the destination page is the source of truth for target-specific options.

## Related docs

* [Stream pipeline config](https://v2.dataos.info/references/resources/nilus/stream/pipeline-config.md)
* [Stream sample configs](https://v2.dataos.info/references/resources/nilus/stream/sample-configs.md)
* [Sources overview](https://v2.dataos.info/references/resources/nilus/sources.md)
* [Destinations](https://v2.dataos.info/references/resources/nilus/destinations.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/nilus/stream.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
