Streaming
Consume and land events continuously from Kafka or NATS with Nilus streaming.
Streaming movement runs a long-lived consumer that reads records from an event-streaming system in micro-batches and writes them to the sink. It processes records as they arrive, with no snapshot, cursor, or extraction window. Use it when the source is a message broker and events must land with minimal delay.
When to use streaming
The source is Apache Kafka or NATS JetStream.
Events must be landed continuously, with minimal delay.
The use case does not fit a scheduled batch window.
You do not need transaction-log change capture; use CDC for that.
Before you begin
Ensure you have:
A source topic or subject reachable from DataOS.
A Depot for the source broker, or an approved connection with secrets projected via
spec.use.projection. NATS has no Depot, so it always uses a directnats+jetstream://URI.A Depot for the destination, and a compute profile.
1. Choose your source
Streaming supports 2 brokers:
Apache Kafka
Topic-based event streams
NATS JetStream
Subject-based persistent message streams
For per-source options, see the stream pipelines reference in References.
2. Write the manifest
Create a nilus resource with spec.type: stream:
For NATS, set source.address to a direct nats+jetstream://<host>:<port>?subject=<subject> URI and project the credentials through spec.use.projection. The subject goes in the URI; source_table is the JetStream stream name. See NATS in References for the authentication forms and examples.
For the full attribute reference, see stream configuration in References.
3. Apply and confirm
The pipeline runs continuously. Once events are landing, explore the moved data to confirm records are arriving as expected.
Last updated
Was this helpful?