Kafka
Kafka stream source. Consumes topics in micro-batches as a long-running consumer; supports plaintext, SASL/SSL, Confluent Cloud, and mTLS configurations.
Apache Kafka is a distributed event-streaming platform used to publish, subscribe to, store, and process streams of records. Nilus consumes Kafka as a stream source (spec.type: stream): a long-running consumer pulls messages in configurable batches, deserializes them, and writes the result to the configured sink.
Nilus connects to Kafka in one of two ways:
Direct URI:
kafka://?bootstrap_servers=...with all connection options expressed as query parameters.DataOS Depot:
dataos://<depot>?purpose=rwfor clusters whose connection string, credentials, and TLS material are managed centrally in DataOS.
Requirements
Connectivity and credentials must both be in place before the pipeline can run.
Connectivity
The Nilus runtime must reach every broker listed in
bootstrap_serversover the configured port (commonly9092for plaintext or9093for TLS).For SASL/SSL clusters, the runtime needs read access to any certificate, key, or truststore files referenced from the URI.
The consumer must have permission to read from the target topic and to commit offsets for the chosen
group_id.
Required parameters
bootstrap_servers
Yes
-
Comma-separated list of Kafka brokers in host:port format.
group_id
Yes
-
Consumer group identifier used for partition assignment and offset tracking.
source_table
Yes
-
Kafka topic to consume from. Passed under source.options.
Optional parameters
batch_size
3000
Maximum number of messages pulled per micro-batch.
batch_timeout
3
Maximum seconds Nilus waits to fill a batch before flushing.
security_protocol
PLAINTEXT
One of PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
sasl_mechanisms
-
Required when security_protocol is SASL-based. Examples: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512.
sasl_username
-
SASL username. Project from secrets.
sasl_password
-
SASL password. Project from secrets.
ssl_ca_location
-
Path to the CA certificate that signs the broker certificates.
ssl_certificate_location
-
Path to the client certificate (mutual TLS).
ssl_key_location
-
Path to the client key (mutual TLS).
ssl_key_password
-
Password for the client key.
ssl_truststore_location
-
Java-style truststore (.jks / .p12) location.
ssl_truststore_password
-
Truststore password.
ssl_truststore_type
JKS
Truststore type: JKS or PKCS12.
run_in_loop
false
When true, the stream service stays running and continues pulling new batches indefinitely.
URI formats
Depot-backed SASL_SSL setup
For a SASL_SSL cluster, define a Kafka depot plus a secret that carries both the SASL credentials and the TLS material. Nilus projects the certificate files into the runtime and wires them to ssl_ca_location / ssl_truststore_location automatically, so you do not need to pass any ssl_*_location paths in the address.
Secret: SASL credentials in data, certificate/truststore files in files:
Depot: references the secret with a purpose:
The pipeline then references only the depot; connection options like group_id, batch_size, and batch_timeout are appended to the address:
Core concepts
Consumer groups. Nilus relies on Kafka's native consumer-group protocol for partition assignment and offset commits. Each pipeline must use a stable
group_idso restarts resume from the last committed offset.Micro-batching. Messages are pulled in batches sized by
batch_sizeand bounded bybatch_timeout. The whole batch is processed and then committed before the next pull.Message shape. Records are deserialized as JSON by default. The sink receives the message payload alongside Kafka metadata (
topic,partition,offset,timestamp,key).Security. SASL (
PLAIN,SCRAM-SHA-256,SCRAM-SHA-512) and SSL/TLS (server-only or mutual) are both supported. FIPS-compliant truststores (.jks/.p12) are honored when configured.
Source options
source_table
Yes
Kafka topic name.
source.options
batch_size
No
Number of messages per micro-batch.
source.address query string
batch_timeout
No
Maximum seconds to wait for a batch.
source.address query string
run_in_loop
No
Keep the consumer running after the first batch.
source.options
Security & SSL params
No
SASL/SSL options listed above.
source.address query string
Examples
Plaintext development cluster
SASL_SSL with mTLS
Throughput tuning
Sink options
dest_table
Yes
Destination object (schema.table for warehouses, fully-qualified path for object stores).
incremental_strategy
Yes
Write strategy: append, replace, or merge. Streaming pipelines normally use append.
primary_key
When incremental_strategy: merge
Column(s) used to deduplicate.
create_table
No
Auto-create the destination if missing. Default: true.
Example sink
Sample Nilus config
Best practices
Use a dedicated consumer group per pipeline so offsets do not collide with other consumers.
Keep
batch_sizealigned with the destination's preferred write size; oversized batches cause sink-side back-pressure, undersized batches under-utilize the pipeline.For low-traffic topics, raise
batch_timeoutso Nilus does not flush near-empty batches.Always set
run_in_loop: truefor long-running stream services. Without it, Nilus exits after the first pull.Project secrets through
spec.use.projection; never commit literalsasl_passwordor key passwords to YAML.
Troubleshooting
Topic does not exist
source_table mismatch or topic not yet created on the cluster.
Verify the topic name and that auto-create is enabled or the topic was provisioned.
SASL authentication failed
Wrong mechanism or credentials.
Confirm sasl_mechanisms matches the broker config and that the projected username/password are correct.
SSL handshake failed
Missing or wrong certificate paths.
Confirm ssl_ca_location (and client cert/key when using mTLS) exist inside the runtime container.
Pipeline exits after the first batch
run_in_loop not set.
Set run_in_loop: true under source.options.
Offsets stuck / replays old data
Stale group_id.
Use a new group_id or reset offsets with the standard Kafka tooling.
Related docs
Last updated
Was this helpful?