For the complete documentation index, see llms.txt. This page is also available as Markdown.

Streaming

Consume and land events continuously from Kafka or NATS with Nilus streaming.

Streaming movement runs a long-lived consumer that reads records from an event-streaming system in micro-batches and writes them to the sink. It processes records as they arrive, with no snapshot, cursor, or extraction window. Use it when the source is a message broker and events must land with minimal delay.

When to use streaming

  • The source is Apache Kafka or NATS JetStream.

  • Events must be landed continuously, with minimal delay.

  • The use case does not fit a scheduled batch window.

  • You do not need transaction-log change capture; use CDC for that.

Before you begin

Ensure you have:

  • A source topic or subject reachable from DataOS.

  • A Depot for the source broker, or an approved connection with secrets projected via spec.use.projection. NATS has no Depot, so it always uses a direct nats+jetstream:// URI.

  • A Depot for the destination, and a compute profile.

1. Choose your source

Streaming supports 2 brokers:

Source
Description

Apache Kafka

Topic-based event streams

NATS JetStream

Subject-based persistent message streams

For per-source options, see the stream pipelines reference in References.

2. Write the manifest

Create a nilus resource with spec.type: stream:

For NATS, set source.address to a direct nats+jetstream://<host>:<port>?subject=<subject> URI and project the credentials through spec.use.projection. The subject goes in the URI; source_table is the JetStream stream name. See NATS in References for the authentication forms and examples.

For the full attribute reference, see stream configuration in References.

3. Apply and confirm

The pipeline runs continuously. Once events are landing, explore the moved data to confirm records are arriving as expected.

Last updated

Was this helpful?