> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/concepts/resources/secret/data-sources/kafka.md). # Kafka ## Prerequisites To create a Secret for securing Kafka credentials, you need the following information. ### Source system requirements Collect the values that match your Kafka security mode. #### Common for all Kafka modes * **`brokers`**: Kafka bootstrap server addresses, such as `broker1:9092,broker2:9092`. Get these from your Kafka admin, managed Kafka console, or cluster connection settings. * **`security_protocol`**: Connection security mode used by the cluster. Supported values in this guide are `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, and `SASL_SSL`. Get this from your Kafka client configuration, platform team, or cluster admin. * **`sasl_mechanism`**: Required only for `SASL`-based modes. Supported values in this guide are `PLAIN` and `SCRAM-SHA-256`. Get this from the same client configuration or from the Kafka admin. #### Mode-specific attributes and credentials {% tabs %} {% tab title="SASL + PLAINTEXT" %} Use this mode when the cluster uses username-password authentication without TLS. * **`username`**: Kafka username or service account name. Get this from the Kafka admin or from the user provisioning workflow of your managed Kafka service. * **`password`**: Password or API secret for the Kafka user. Get this when the user is created, or request a reset from the Kafka admin if needed. {% endtab %} {% tab title="Passwordless" %} Use this mode when the cluster uses `PLAINTEXT`. * No additional Secret fields are required beyond `security_protocol`. {% endtab %} {% tab title="SSL" %} Use this mode when the cluster requires TLS client authentication. * **`trust_store_type`**: Format of the trust store file. The examples use `JKS`. Get this from the security team that issued the trust store. * **`trust_store_password`**: Password that protects the trust store. Get this from the security team or Kafka admin. * **`trust_store_file`**: Local path to the trust store file that contains the CA certificates used to validate the broker certificate. This is usually exported and shared by the security team or platform admin. * **`key_store_type`**: Format of the client key store file. The examples use `JKS`. Get this from the team that issued the client certificate. * **`key_store_password`**: Password that protects the key store. Get this from the security team or Kafka admin. * **`key_password`**: Password for the private key inside the key store. Get this from the security team or whoever generated the client key pair. * **`key_store_file`**: Local path to the key store file that contains the client certificate and private key. This is usually generated by your PKI or security team. {% endtab %} {% tab title="SASL + SSL (`PLAIN`)" %} Use this mode when the cluster uses TLS plus SASL `PLAIN`. * **`username`**: Kafka username or service account name. Get this from the Kafka admin or managed Kafka access control setup. * **`password`**: Password or API secret for the Kafka user. Get this during user creation or from the Kafka admin. * **`trust_store_type`**: Format of the trust store file. The examples use `JKS`. Get this from the security team. * **`trust_store_password`**: Password that protects the trust store. Get this from the security team or Kafka admin. * **`trust_store_file`**: Local path to the trust store file used to validate broker certificates. Get this from the security team or platform admin. {% endtab %} {% tab title="SASL + SSL (`SCRAM-SHA-256`)" %} Use this mode when the cluster uses TLS plus SASL `SCRAM-SHA-256`. * **`username`**: Kafka username or SCRAM user name. Get this from the Kafka admin or managed Kafka identity setup. * **`password`**: SCRAM password for the Kafka user. Get this when the SCRAM user is created, or rotate it through the Kafka admin. * **`trust_store_type`**: Format of the trust store file. The examples use `JKS`. Get this from the security team. * **`trust_store_password`**: Password that protects the trust store. Get this from the security team or Kafka admin. * **`trust_store_file`**: Local path to the trust store file used to validate broker certificates. Get this from the security team or platform admin. * **`ca_file`**: Optional CA certificate file path. Use this only when your team shares the CA certificate directly instead of a prepared trust store. {% endtab %} {% endtabs %} ## Create a Secret for securing Kafka credentials DataOS supports multiple Kafka security modes. Choose the matching Secret template from the tabs below. ### Supported Kafka Secret modes
Modesecurity_protocolsasl_mechanismTypical Secret fields
SASL + PLAINTEXTSASL_PLAINTEXTPLAINusername, password
PasswordlessPLAINTEXTNAsecurity_protocol
SSLSSLNAtrust_store_*, key_store_*, key_password, files
SASL + SSL (PLAIN)SASL_SSLPLAINtrust_store_*, username, password, files
SASL + SSL (SCRAM-256)SASL_SSLSCRAM-SHA-256trust_store_*, username, password, files
### Step 1: Create a manifest file Create a manifest file with the configuration details for your Kafka Secret. {% tabs %} {% tab title="SASL" %} ```yaml name: ${{kafka-sasl-plaintext-secret-name}} version: v2alpha type: secret tags: - ${{tag-1}} - ${{tag-2}} description: "Credentials for Kafka SASL_PLAINTEXT depot." layer: user secret: type: key-value data: security_protocol: SASL_PLAINTEXT sasl_mechanism: PLAIN username: ${{kafka-username}} password: ${{kafka-password}} ``` {% endtab %} {% tab title="Passwordless" %} ```yaml name: "${{passwordless-secret-name}}" version: v2alpha type: secret tags: - ${{tag-1}} - ${{tag-2}} description: "creds for kafka passwordless depot" layer: user secret: type: key-value data: security_protocol: ${{passwordless-security-protocol}} ``` {% endtab %} {% tab title="SSL" %} ```yaml name: ${{kafka-ssl-secret-name}} version: v2alpha type: secret tags: - ${{tag-1}} - ${{tag-2}} description: "Credentials for Kafka SSL depot." layer: user secret: type: key-value data: security_protocol: SSL trust_store_type: JKS trust_store_password: ${{trust-store-password}} key_store_type: JKS key_store_password: ${{key-store-password}} key_password: ${{key-password}} files: trust_store_file: ${{trust-store-file}} key_store_file: ${{key-store-file}} ``` {% endtab %} {% tab title="SASL\_SSL" %} ```yaml name: ${{kafka-sasl-ssl-plain-secret-name}} version: v2alpha type: secret tags: - ${{tag-1}} - ${{tag-2}} description: "Credentials for Kafka SASL_SSL (PLAIN) depot." layer: user secret: type: key-value data: security_protocol: SASL_SSL sasl_mechanism: PLAIN trust_store_type: JKS trust_store_password: ${{trust-store-password}} username: ${{kafka-username}} password: ${{kafka-password}} files: trust_store_file: ${{trust-store-file}} ``` {% endtab %} {% tab title="SASL\_SSL + SCRAM-SHA-256" %} ```yaml name: ${{kafka-sasl-scram256-secret-name}} version: v2alpha type: secret tags: - ${{tag-1}} - ${{tag-2}} description: "Credentials for Kafka SASL_SSL (SCRAM-SHA-256) depot." layer: user secret: type: key-value data: security_protocol: SASL_SSL sasl_mechanism: SCRAM-SHA-256 trust_store_type: JKS trust_store_password: ${{trust-store-password}} username: ${{kafka-username}} password: ${{kafka-password}} files: trust_store_file: ${{trust-store-file}} ca_file: ${{ca-file}} # optional ``` {% endtab %} {% endtabs %} For more information about each attribute, refer to the [configurations section](/concepts/resources/secret/manifest-configuration.md). ### Step 2: Apply the manifest Use the `apply` command to create the Kafka Secret in DataOS. ```bash dataos-ctl resource apply -f ${{manifest-file-path}} ``` **Example Usage:** ```bash dataos-ctl resource apply -f secret.yaml #output INFO[0000] 🛠 apply... INFO[0000] 🔧 applying kafka-secret:v1:secret... INFO[0004] 🔧 applying kafka-secret:v1:secret...created INFO[0004] 🛠 apply...complete ``` ### Step 3: Validate the Secret Use the `get` command to verify the Kafka Secret in DataOS. ```bash dataos-ctl resource get -t secret ``` **Expected Output:** ```bash INFO[0000] 🔍 get... INFO[0000] 🔍 get...complete NAME | VERSION | TYPE | WORKSPACE | STATUS | RUNTIME | OWNER -----------------|---------|-----------------|-----------|--------|-----------|------------------------------ kafka-secret | v2alpha | secret | | active | | iamgroottmdcio ``` To list all Secrets in the DataOS environment, run the following command. ```bash dataos-ctl resource get -t secret -a ``` **Expected Output:** ```bash time="2026-03-25T15:34:17+05:30" level=info msg="🔍 resource get..." time="2026-03-25T15:34:17+05:30" level=info msg="🔍 resource get...complete" NAME | VERSION | TYPE | STATUS | RUNTIME | OWNER --------------------------------+---------+--------+--------+---------+------------------------- kafka-secret | v2alpha | secret | active | | iamgroottmdcio azureconnection-testing | v2alpha | secret | active | | iamgroottmdcio azuresecretnilus | v2alpha | secret | active | | iamgroottmdcio bitbucket-secrets | v2alpha | secret | active | | iamgroottmdcio ``` ## Delete the Secret {% hint style="warning" %} Before deleting a Secret, remove any Resources that depend on it. For example, if a Depot depends on a Secret, deleting the Secret fails until you remove the Depot. The same rule applies to all dependent Resources, such as Workflow, Service, and Worker. The example below shows the error returned when a Resource still depends on the Secret. **Example usage:** ```bash dataos-ctl resource delete -t secret -n kafka-cred time="2026-03-25T15:46:12+05:30" level=info msg="🗑 delete..." time="2026-03-25T15:46:12+05:30" level=info msg="🗑 deleting kafka-cred:v2alpha:secret..." time="2026-03-25T15:46:13+05:30" level=info msg="🗑 deleting kafka-cred:v2alpha:secret...error" time="2026-03-25T15:46:13+05:30" level=warning msg="🗑 delete...error for resource kafka-cred" time="2026-03-25T15:46:13+05:30" level=error msg="Invalid Parameter - failure deleting tenant resource : cannot delete resource, it is a dependency of 'depot:v2alpha:kafkaconnection'" ``` {% endhint %} To delete the Kafka Secret, use one of the following commands: {% tabs %} {% tab title="Command 1" %} ```bash dataos-ctl resource delete -t secret -n ${{secret-name}} ``` {% endtab %} {% tab title="Command 2 " %} ```bash dataos-ctl resource delete -i "${{secret-name}}|v2alpha|secret" ``` {% endtab %} {% tab title="Command 3" %} ```bash dataos-ctl resource delete -f ${{manifest-file-path}} ``` {% endtab %} {% endtabs %} Specify the Resource type and Secret name in the `delete` command. **Example Usage:** {% tabs %} {% tab title="Command 1" %} ```bash dataos-ctl resource delete -t secret -n testsecret #output time="2026-03-25T15:53:55+05:30" level=info msg="🗑 delete..." time="2026-03-25T15:53:55+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret..." time="2026-03-25T15:53:56+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret...deleted" time="2026-03-25T15:53:56+05:30" level=info msg="🗑 delete...complete" ``` {% endtab %} {% tab title="Command 2" %} ```bash dataos-ctl resource delete -i "testsecret|valpha|secret" #output time="2026-03-25T15:55:37+05:30" level=info msg="🗑 delete..." time="2026-03-25T15:55:37+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret..." time="2026-03-25T15:55:37+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret...deleted" time="2026-03-25T15:55:37+05:30" level=info msg="🗑 delete...complete" ``` {% endtab %} {% tab title="Command 3" %} ```bash dataos-ctl resource delete -f docs\platform-entities\governance-resources\secret\test.yaml #output time="2026-03-25T15:53:55+05:30" level=info msg="🗑 delete..." time="2026-03-25T15:53:55+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret..." time="2026-03-25T15:53:56+05:30" level=info msg="🗑 deleting testsecret:v2alpha:secret...deleted" time="2026-03-25T15:53:56+05:30" level=info msg="🗑 delete...complete" ``` {% endtab %} {% endtabs %} [^1]: Username-password authentication without encryption. [^2]: No authentication and no encryption. [^3]: Encrypted connection using certificates. [^4]: Encrypted Kafka connection with username-password authentication [^5]: Stronger password-based authentication over an encrypted connection. [^6]: A secure password-based authentication method that verifies identity without sending the actual password directly. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/concepts/resources/secret/data-sources/kafka.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.