For the complete documentation index, see llms.txt. This page is also available as Markdown.

Kafka

Minimum Kafka ACLs for Nilus stream pipelines, across all four supported security protocols. Kafka is supported as a source only.

This page documents the minimum Kafka ACLs a platform or Kafka administrator must provision for Nilus to run a stream pipeline against a Kafka topic. Kafka is supported as a source only (type: stream); no write, CDC, or metadata pipeline mode exists for Kafka.

Supported security protocols

Protocol
Supported
Auth method

PLAINTEXT

Yes

No authentication

SASL_PLAINTEXT

Yes

SASL credentials over a plain (unencrypted) connection

SSL

Yes

TLS encryption with CA certificate verification (server cert only)

SASL_SSL

Yes

SASL credentials over a TLS-encrypted connection

Read data: Kafka as source (type: stream)

Nilus uses this path when the depot is referenced in source.address of a pipeline config. Nilus consumes messages from the target topic using the Kafka consumer protocol.

Kafka permissions are managed through ACLs, scoped to the principal User:<username> for SASL-authenticated connections. For PLAINTEXT, no ACLs are enforced.

Minimum grants

# Allow reading messages from the topic
kafka-acls.sh --bootstrap-server <broker> \
  --add --allow-principal User:<username> \
  --operation Read --topic <topic_name>

# Allow committing and reading consumer offsets
kafka-acls.sh --bootstrap-server <broker> \
  --add --allow-principal User:<username> \
  --operation Read --group '*'

# Allow topic metadata lookup (required for partition assignment)
kafka-acls.sh --bootstrap-server <broker> \
  --add --allow-principal User:<username> \
  --operation Describe --topic <topic_name>
ACL
Resource type
Why required

READ

Topic:<topic_name>

Consume messages from the topic

READ

Group:<pipeline_resource_id>

Commit and read consumer offsets

DESCRIBE

Topic:<topic_name>

Fetch partition metadata for consumer assignment

Permission matrix

Use case

PLAINTEXT

SASL_PLAINTEXT

SSL

SASL_SSL

Kafka ACL required

Stream read (topic consumer)

Yes

Yes

Yes

Yes

READ topic + READ group + DESCRIBE topic

Notes

  • Kafka is a source-only connector in Nilus. There is no write, CDC, or metadata pipeline mode for Kafka depots.

  • ACL enforcement depends on the security protocol: PLAINTEXT connections are not subject to ACL checks, while SASL_PLAINTEXT, SSL, and SASL_SSL all require the ACLs above.

  • The consumer group ACL targets Group:<pipeline_resource_id>. Grant READ on the specific group your pipeline resource uses, or scope it to '*' if the group name is not known ahead of time.

Last updated

Was this helpful?