For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs.
Skip to content

Kafka

Kafka stream source. Consumes topics in micro-batches as a long-running consumer; supports plaintext, SASL/SSL, Confluent Cloud, and mTLS configurations.

Apache Kafka is a distributed event-streaming platform used to publish, subscribe to, store, and process streams of records. Nilus consumes Kafka as a stream source (spec.type: stream): a long-running consumer pulls messages in configurable batches, deserializes them, and writes the result to the configured sink.

Nilus connects to Kafka in one of two ways:

  • Direct URI: kafka://?bootstrap_servers=... with all connection options expressed as query parameters.

  • DataOS Depot: dataos://<depot>?purpose=rw for clusters whose connection string, credentials, and TLS material are managed centrally in DataOS.

Even though the underlying source is event-streaming, Nilus pulls in micro-batches. Tune batch_size and batch_timeout to match downstream sink throughput.

Requirements

Connectivity and credentials must both be in place before the pipeline can run.

Connectivity

  • The Nilus runtime must reach every broker listed in bootstrap_servers over the configured port (commonly 9092 for plaintext or 9093 for TLS).

  • For SASL/SSL clusters, the runtime needs read access to any certificate, key, or truststore files referenced from the URI.

  • The consumer must have permission to read from the target topic and to commit offsets for the chosen group_id.

Required parameters

Parameter
Required
Default
Description

bootstrap_servers

Yes

-

Comma-separated list of Kafka brokers in host:port format.

group_id

Yes

-

Consumer group identifier used for partition assignment and offset tracking.

source_table

Yes

-

Kafka topic to consume from. Passed under source.options.

Optional parameters

Parameter
Default
Description

batch_size

3000

Maximum number of messages pulled per micro-batch.

batch_timeout

3

Maximum seconds Nilus waits to fill a batch before flushing.

security_protocol

PLAINTEXT

One of PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.

sasl_mechanisms

-

Required when security_protocol is SASL-based. Examples: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512.

sasl_username

-

SASL username. Project from secrets.

sasl_password

-

SASL password. Project from secrets.

ssl_ca_location

-

Path to the CA certificate that signs the broker certificates.

ssl_certificate_location

-

Path to the client certificate (mutual TLS).

ssl_key_location

-

Path to the client key (mutual TLS).

ssl_key_password

-

Password for the client key.

ssl_truststore_location

-

Java-style truststore (.jks / .p12) location.

ssl_truststore_password

-

Truststore password.

ssl_truststore_type

JKS

Truststore type: JKS or PKCS12.

run_in_loop

false

When true, the stream service stays running and continues pulling new batches indefinitely.

Treat sasl_password, ssl_key_password, and ssl_truststore_password as secrets. Project them through spec.use.projection rather than inlining literal values.

URI formats

Depot-backed SASL_SSL setup

For a SASL_SSL cluster, define a Kafka depot plus a secret that carries both the SASL credentials and the TLS material. Nilus projects the certificate files into the runtime and wires them to ssl_ca_location / ssl_truststore_location automatically, so you do not need to pass any ssl_*_location paths in the address.

Secret: SASL credentials in data, certificate/truststore files in files:

Depot: references the secret with a purpose:

The pipeline then references only the depot; connection options like group_id, batch_size, and batch_timeout are appended to the address:

Core concepts

  • Consumer groups. Nilus relies on Kafka's native consumer-group protocol for partition assignment and offset commits. Each pipeline must use a stable group_id so restarts resume from the last committed offset.

  • Micro-batching. Messages are pulled in batches sized by batch_size and bounded by batch_timeout. The whole batch is processed and then committed before the next pull.

  • Message shape. Records are deserialized as JSON by default. The sink receives the message payload alongside Kafka metadata (topic, partition, offset, timestamp, key).

  • Security. SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512) and SSL/TLS (server-only or mutual) are both supported. FIPS-compliant truststores (.jks / .p12) are honored when configured.

Source options

Option
Required
Description
Passed in

source_table

Yes

Kafka topic name.

source.options

batch_size

No

Number of messages per micro-batch.

source.address query string

batch_timeout

No

Maximum seconds to wait for a batch.

source.address query string

run_in_loop

No

Keep the consumer running after the first batch.

source.options

Security & SSL params

No

SASL/SSL options listed above.

source.address query string

Examples

Plaintext development cluster

SASL_SSL with mTLS

Throughput tuning

Sink options

Option
Required
Description

dest_table

Yes

Destination object (schema.table for warehouses, fully-qualified path for object stores).

incremental_strategy

Yes

Write strategy: append, replace, or merge. Streaming pipelines normally use append.

primary_key

When incremental_strategy: merge

Column(s) used to deduplicate.

create_table

No

Auto-create the destination if missing. Default: true.

Example sink

Sample Nilus config

Best practices

  • Use a dedicated consumer group per pipeline so offsets do not collide with other consumers.

  • Keep batch_size aligned with the destination's preferred write size; oversized batches cause sink-side back-pressure, undersized batches under-utilize the pipeline.

  • For low-traffic topics, raise batch_timeout so Nilus does not flush near-empty batches.

  • Always set run_in_loop: true for long-running stream services. Without it, Nilus exits after the first pull.

  • Project secrets through spec.use.projection; never commit literal sasl_password or key passwords to YAML.

Troubleshooting

Symptom
Likely cause
Resolution

Topic does not exist

source_table mismatch or topic not yet created on the cluster.

Verify the topic name and that auto-create is enabled or the topic was provisioned.

SASL authentication failed

Wrong mechanism or credentials.

Confirm sasl_mechanisms matches the broker config and that the projected username/password are correct.

SSL handshake failed

Missing or wrong certificate paths.

Confirm ssl_ca_location (and client cert/key when using mTLS) exist inside the runtime container.

Pipeline exits after the first batch

run_in_loop not set.

Set run_in_loop: true under source.options.

Offsets stuck / replays old data

Stale group_id.

Use a new group_id or reset offsets with the standard Kafka tooling.

Last updated

Was this helpful?