> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/stage-1-discover/bring-data-in/streaming-data-ingestion.md).

# Streaming data movement

Use this page when your source is a message broker, and you need to consume and land events continuously without a fixed schedule.

Stream data movement processes records as they arrive. It does not require a snapshot and does not use a cursor or extraction window, the way batch does.

***

## When to use stream data movement

Use it when:

* The sources are like Apache Kafka or NATS JetStream.
* Events must be landed with minimal delay.
* The use case does not fit a scheduled batch window.
* You do not need transaction-log-level change capture; use [Change Data Capture](/build/stage-1-discover/bring-data-in/cdc-ingestion.md) for that.

***

## Before you start

Make sure you have:

* A source subject available and accessible from DataOS.
* A DataOS Depot for the source broker or an approved connection method with secrets projected via `use.projection`.
* A DataOS Depot for the destination.
* A compute profile available for the pipeline.

***

{% stepper %}
{% step %}

## Choose your source

Nilus stream data movement is supported for two sources.

| Source             | Description                               |
| ------------------ | ----------------------------------------- |
| **Apache Kafka**   | Topic-based event streams.                |
| **NATS JetStream** | Subject-based persistent message streams. |

For per-source connection options and configuration details, see the Nilus Stream Pipelines reference.
{% endstep %}

{% step %}

## Write the manifest

Create a `nilus` resource manifest with `spec.type: stream`.

```yaml
name: ${{pipeline-name}}
version: v1alpha
type: nilus
tags:
  - nilus-stream
description: ${{description}}

spec:
  type: stream
  compute: ${{compute-profile}}
  logLevel: INFO

  resources:
    requests:
      cpu: "200m"
      memory: "256Mi"

  source:
    address: dataos://${{source-depot}}?purpose=ro
    options:
      source_table: "${{topic-or-subject}}"   # Kafka topic name or NATS subject

  sink:
    address: dataos://${{sink-depot}}?purpose=rw
    options:
      dest_table: ${{schema.table}}
      incremental_strategy: append
```

For source-specific options and the full attribute reference, see the Nilus Stream Pipelines.
{% endstep %}

{% step %}

## Apply the pipeline

```bash
dataos-ctl resource apply -f ${{path-to-manifest.yaml}}
```

Confirm the resource is active:

```bash
dataos-ctl resource get -t nilus -a
# or 
dataos-ctl resource get -t nilus -n ${{pipeline-name}}
```

{% endstep %}
{% endstepper %}

***

**Related reference**

* [Stream Pipelines reference](/concepts/resources/nilus/stream.md): architecture, supported sources, and full configuration reference.
* [Supported destinations](/concepts/resources/nilus/destinations.md): configure the sink for your target system.
* [Configurations reference](https://app.gitbook.com/s/TVR3yNyAr5eErLRWsr2P/resources/nilus/stream/pipeline-config): full batch attribute table.
* [Schema evolution](/concepts/resources/nilus/concepts/schema-evolution.md): how Nilus handles column additions and type changes.
* [Data masking](/concepts/resources/nilus/concepts/understanding-data-masking.md): mask or redact columns during movement.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/build/stage-1-discover/bring-data-in/streaming-data-ingestion.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.