> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/operate/v1/dataos-operator/provision-data-plane/azure.md). # Azure Deploy the DataOS Data Plane on an existing Azure Kubernetes Service (AKS) cluster within your Azure subscription. Your SRE team is responsible for provisioning and operating the underlying Azure infrastructure in accordance with Modern's published requirements. DataOS platform components are then installed on the cluster using a Data Plane manifest applied through `dataos-ctl`. The Data Plane establishes an outbound connection to the DataOS Instance hosted in Modern Cloud. {% hint style="info" %} The DataOS Instance must already be provisioned before you deploy the Data Plane. The Instance FQDN you deploy against, and the deployment application user you create below, both live on that Instance. {% endhint %} ## Prerequisites
1. Instance Use a working DataOS **Instance FQDN**. For example: `pacific-staging.instance.dataos.cloud`
2. DataOS CLI Install `dataos-ctl` on the bastion host, from the **Setup CLI** section on the Instance home page. Use the DataOS CLI version named in the release inputs below. If the binary is not on the bastion host's `PATH`, call it by its full install path, for example `/home/jumpvmadmin/.dataos/v2/bin/dataos-ctl`. This guide uses the short form.
3. DataOS permissions Your own DataOS user needs all three of these roles on the Instance, so that you can create the application user and apply the Data Plane manifest: * `roles:id:user` * `roles:id:operator` * `roles:id:super-user` Check them in **Bifrost** at `https:///platform/bifrost/users`.
4. Database endpoints and credentials The Data Plane connects to an external PostgreSQL database provisioned during infrastructure setup. Have these values ready; you supply them directly in the Data Plane manifest. * `PG_HOST`, `PG_PORT`: PostgreSQL host and port. * `SYSTEM_HEALTHD_DB_NAME`, `SYSTEM_HEALTHD_DB_USERNAME`, `SYSTEM_HEALTHD_DB_PASSWORD`: Database name and credentials for System HealthD. * `VESTA_DB_NAME`, `VESTA_DB_USERNAME`, `VESTA_DB_PASSWORD`: Database name and credentials for Vesta.
5. Kubeconfig access You need a valid kubeconfig for the provisioned AKS cluster, and `kubectl` must reach the cluster with it. You base64-encode this kubeconfig during deployment.
6. Release inputs (from Modern) These values are supplied by Modern for the release you are installing; they are not customer-derivable. Confirm all of them before you apply. | Input | Manifest field | Example | | -------------------- | -------------------- | ---------------------------- | | Install artifact | `artifactIdentifier` | `` | | DataOS Manager image | `imageTag` | `` | | DataOS CLI version | : | `` | {% hint style="info" %} The values in the table are placeholders. Take the exact values from the release notes for the release you are installing, and install the CLI version those notes name: earlier versions may not recognize the manifest fields a newer release expects. {% endhint %} {% hint style="info" %} `artifactIdentifier` accepts two forms, and the choice matters. A snapshot, such as `artifact:aft_abc123abc123`, installs one fixed set of manifests every time it is applied. A release, such as `release:dataos-v2-install-manifests:prd_abc123abc123`, installs whatever is currently latest behind that release, so applying the same manifest again later can install a newer build. Use a snapshot in production. {% endhint %}
## Deploy Data Plane ### 1. Authenticate from the bastion host All provisioning commands run from the **bastion host** (for example, `jumpvmadmin`), because production AKS API servers accept traffic only from authorized IP ranges. Confirm the access path with your infrastructure team. The bastion host holds no persistent CLI session, so each command authenticates on its own. Supply these four values with every command: | Variable | Purpose | | ------------------ | --------------------------------- | | `USER_ID` | User the command authenticates as | | `DATAOS_FQDN` | Target DataOS Instance domain | | `APIKEY` | API key belonging to `USER_ID` | | `DATAOS_TENANT_ID` | Always `system` | Two identities are involved. Your **personal user** creates the application user's API key; the **application user** carries out every `domain` operation that follows. Start with your own credentials. {% hint style="warning" %} Never provision a Data Plane with your personal API key. It ties the Data Plane to an individual account. {% endhint %} **1.1 Get your user ID.** ```bash dataos-ctl user get ``` ``` INFO[0000] 😃 user get... INFO[0001] 😃 user get...complete NAME │ ID │ TYPE │ EMAIL │ TAGS ────────────┼──────────────────┼────────┼───────────────────┼────────────────────────────── iamgroot │ iamgroottmdcio │ person │ iamgroot@tmdc.io │ roles:id:user, │ │ │ │ roles:id:operator, │ │ │ │ users:id:iamgroottmdcio ``` The value under **ID** is your `USER_ID`. **1.2 Get your personal API key.** List the keys already issued to you: ```bash dataos-ctl user apikey get ``` Create one if you need a new key: ```bash dataos-ctl user apikey create --id --name -d 24h ``` This key authorizes a single operation: creating the application user's API key in Step 3. ### 2. Create the application user In **Bifrost** on your DataOS Instance (`https:///platform/bifrost`), create an application user dedicated to this Data Plane. Name it after the Data Plane, for example `azuredataplane0123-user`, and assign these roles: * `roles:id:user` * `roles:id:operator` * `roles:id:super-user` The application user owns the deployment, which keeps the Data Plane independent of any individual account. ### 3. Create an API key for the application user Run this command as your **personal user**: `USER_ID` and `APIKEY` are your own. ```bash USER_ID= \ DATAOS_FQDN= \ APIKEY= \ DATAOS_TENANT_ID=system \ dataos-ctl user apikey create -i -d 24h ``` The command prints the new API key. Copy it: you supply it as `APIKEY` in the lint, apply, and monitor commands that follow. ### 4. Base64-encode kubeconfig Confirm `kubectl` reaches the cluster, then base64-encode the kubeconfig into `KUBECONFIG_BASE64`: ```bash kubectl get nodes # confirm access first ``` The encoding command differs by operating system, because `base64` takes different flags on each and Windows has no `base64` at all. Run the one that matches the machine you are deploying from. {% tabs %} {% tab title="Linux" %} ```bash export KUBECONFIG_BASE64=$(kubectl config view --minify --flatten | base64 -w 0) ``` {% endtab %} {% tab title="macOS" %} ```bash export KUBECONFIG_BASE64=$(kubectl config view --minify --flatten | base64 | tr -d '\n') ``` {% endtab %} {% tab title="Windows" %} ```powershell $env:KUBECONFIG_BASE64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes((kubectl config view --minify --flatten | Out-String))) ``` {% endtab %} {% endtabs %} This value gives the DataOS Instance access to the AKS API. Treat it as a secret. ### 5. Set the variables the manifest resolves `dataos-ctl domain apply` resolves `${...}` references in the manifest from the shell you run it in. Set these values there before you apply: ```bash export KUBECONFIG_BASE64= # from the previous step export IMAGE_TAG= # provided by Modern export ARTIFACT_IDENTIFIER= # provided by Modern ``` | Variable | Purpose | Example | | --------------------- | ------------------------------------------------ | ------------------ | | `DATAPLANE_ID` | Stable Data Plane identifier | `acme-azure-dp-01` | | `INSTANCE_TENANT_ID` | DataOS Instance this Data Plane attaches to | `pacific-ocean` | | `KUBECONFIG_BASE64` | Base64-encoded kubeconfig from the previous step | : | | `IMAGE_TAG` | DataOS Manager image tag to pin (from Modern) | : | | `ARTIFACT_IDENTIFIER` | Install artifact to apply (from Modern) | : | `DATAPLANE_ID` and `INSTANCE_TENANT_ID` are supplied inline on the lint and apply commands in Step 7. ### 6. Prepare the Data Plane manifest Copy the sample manifest below. You only need to fill in a handful of values; everything else is fixed and wires the Data Plane to the DataOS Instance over the tunnel. **What you fill in** Identity and naming: * `name` / `DATAPLANE_ID`: the dataplane identifier. Use a name that encodes the Instance and cloud, for example `-azure-dp-01`. * `description`: a short, Instance- and Tenant-specific description for traceability. * `workpiece.name`: must end with the `.dataplane` suffix. This becomes the Data Plane domain. Cluster access: * `kubeConfigBase64`: the base64 kubeconfig from Step 4. Version and install: * `dataOsManagerConfigs.imageTag`: resolved from `${IMAGE_TAG}` (the `dataos-manager` version from Modern). * `installFromPrime.artifactIdentifier`: resolved from `${ARTIFACT_IDENTIFIER}` (the install artifact from Modern). Platform databases: * `pgHost`, `pgPort`: the PostgreSQL endpoint. `pgPort` is a quoted string, for example `"5432"`. * `systemHealthDb*` and `vestaDb*` values: database names and credentials for System HealthD and Vesta. Leave every other field exactly as shown. ```yaml name: ${DATAPLANE_ID} version: v1alpha entity: domain type: dataplane description: the ${DATAPLANE_ID} dataplane for ${INSTANCE_TENANT_ID} instance v1alpha: dataplane: name: ${DATAPLANE_ID} tenantDomainNames: - system-entities networkType: tunnel workpiece: name: ${DATAPLANE_ID}-${INSTANCE_TENANT_ID}.dataplane zone: dataos.cloud kernelRequest: template: dataplane-kernel-system-existing-compute-tunnel-v1 cloud: azure inputs: dataOsManagerConfigs: imageTag: ${IMAGE_TAG} logLevel: info replicas: 1 nodeSelector: dataos.io/purpose: core-kernel kubeConfigBase64: ${KUBECONFIG_BASE64} configs: - cloudflare:tmdc-prime dataPlaneKernelSystemInstall: installFromPrime: artifactIdentifier: ${ARTIFACT_IDENTIFIER} installFileRootDir: /install/dataplane-kernel-system installFile: release-prd-install.yaml applicationsFile: install.applications.yaml valuesFile: install.values.yaml onDemandCloudKernel: true envPurpose: prd azureEndpointSuffix: core.windows.net awsEndpointSuffix: amazonaws.com pgHost: pgPort: "5432" systemHealthDbName: systemHealthDbUsername: systemHealthDbPassword: vestaDbName: vestaDbUsername: vestaDbPassword: coreKernelNodeSelector: dataos.io/purpose: core-kernel features: cloudnativePg: enable: false ``` {% hint style="warning" %} The kubeconfig and database credentials inside the manifest are sensitive. Store the manifest encrypted and limit who can decrypt it. Decrypt it only at apply time. {% endhint %} ### 7. Lint and apply the Data Plane manifest Authenticate as the **application user** with the API key from Step 3, then **lint** the manifest before applying. `KUBECONFIG_BASE64` and `IMAGE_TAG` are resolved from the variables you set in Step 5. **Lint** (validate the manifest with `-l`): ```bash USER_ID= \ DATAOS_FQDN= \ APIKEY= \ DATAOS_TENANT_ID=system \ INSTANCE_TENANT_ID= \ DATAPLANE_ID= \ dataos-ctl domain apply -f azure-dataplane.yaml -l ``` Lint validates the manifest and resolves the variables without provisioning anything. If it succeeds but nothing is created, you left the `-l` flag on. **Apply** (once lint passes, rerun the same command without `-l`): ```bash USER_ID= \ DATAOS_FQDN= \ APIKEY= \ DATAOS_TENANT_ID=system \ INSTANCE_TENANT_ID= \ DATAPLANE_ID= \ dataos-ctl domain apply -f azure-dataplane.yaml ``` The DataOS Instance validates the request and starts installing the platform onto your cluster.
What happens after you apply When you run the apply, the DataOS Instance reads the `kubeConfigBase64` you supplied and uses it to reach the AKS API for the first time. It installs `dataos-manager` onto the cluster, which pulls the install manifests defined in `installFromPrime` and lays down the DataOS platform components. A tunnel client inside the cluster opens an outbound, encrypted connection from a worker node out through the cluster's outbound NAT to Modern Cloud. No inbound ports on your cluster are exposed. Once the tunnel is established, the DataOS Instance registers the Data Plane and manages it over that connection. The kubeconfig is used only for this DataOS Instance-to-cluster management path. It is not part of your data traffic. Every hop is TLS-encrypted end to end.
## Manage Data Plane ### Check status Provisioning continues in the background after the apply returns. Check where it has reached: ```bash USER_ID= \ DATAOS_FQDN= \ APIKEY= \ DATAOS_TENANT_ID=system \ dataos-ctl domain get # add -a to include Data Planes created by other users ``` ``` azuredataplane0123 | v1alpha | dataplane | azuredataplane0123 | active | workpiece-forge-kernel:success:✅ created '2026-07-07T15:53:46Z' elapsed '24.942115212s' | iamgroottmdcio | | | | | instance-manager-install:success:✅ created '2026-08-04T12:44:54Z' elapsed '1m19.210943467s' | ``` The Data Plane is ready when the status column reads `active` and `workpiece-forge-kernel` reports success. For a closer view of the installation, connect to the Data Plane cluster and follow the `dataos-manager` logs: ```bash kubectl logs -n -0-dsm dataos-manager-0 -f ``` {% hint style="info" %} Provisioning the infrastructure and installing the platform typically completes within 30 minutes. During that time the Instance reaches the cluster, installs `dataos-manager`, and waits for the tunnel to come up. If the status stays unchanged well beyond that window, the `dataos-manager` logs show where it stalled. {% endhint %} ### Describe applied configuration ```bash USER_ID= \ DATAOS_FQDN= \ APIKEY= \ DATAOS_TENANT_ID=system \ dataos-ctl domain get -t dataplane -n -azure-dp-01 -d ``` *** ## Dataplane configuration A field-by-field reference for the Data Plane manifest. #### Metadata Top-level fields that identify the Data Plane resource. | Attribute | Description | | ------------- | ----------------------------------------------------------------- | | `name` | Keep it stable and readable: it becomes the dataplane identifier. | | `version` | API version for the resource spec. | | `entity` | Always `domain`. | | `type` | Always `dataplane`: declares what you are provisioning. | | `description` | Make it environment- and Tenant-specific for traceability. | ```yaml name: ${DATAPLANE_ID} version: v1alpha entity: domain type: dataplane description: the ${DATAPLANE_ID} dataplane for ${INSTANCE_TENANT_ID} instance ``` {% hint style="info" %} Use a naming convention that clearly encodes the Instance and cloud. Example: `--dp-01`. {% endhint %} #### Dataplane Core Data Plane identity and DataOS Instance connectivity. | Attribute | Description | | ------------------- | --------------------------------------------------------------------------------------------------------- | | `dataplane.name` | The dataplane identifier. | | `tenantDomainNames` | Default domain mappings (platform-specific). | | `networkType` | Decides how the Data Plane reaches the DataOS Instance. `tunnel` uses outbound-only Cloudflare tunneling. | ```yaml v1alpha: dataplane: name: ${DATAPLANE_ID} tenantDomainNames: - system-entities networkType: tunnel ``` #### Workpiece Represents the execution unit for the provisioning pipeline. | Attribute | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `workpiece.name` | Must end with the `.dataplane` suffix. This produces the Data Plane domain, for example `azuredataplane0123.dataplane.dataos.cloud`. | | `workpiece.zone` | Always set to `dataos.cloud`. | ```yaml workpiece: name: ${DATAPLANE_ID}-${INSTANCE_TENANT_ID}.dataplane zone: dataos.cloud ``` #### Kernel Request Selects the kernel provisioning blueprint and target cloud. | Attribute | Description | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `template` | Defines which kernel provisioning blueprint is used. `dataplane-kernel-system-existing-compute-tunnel-v1` installs the platform onto an existing AKS cluster in tunnel mode. | | `cloud` | Selects cloud-specific behavior. Use `azure`. | ```yaml kernelRequest: template: dataplane-kernel-system-existing-compute-tunnel-v1 cloud: azure ``` #### Inputs (DataOS Manager + kubeconfig) | Attribute | Description | | ----------------------------------- | --------------------------------------------------------------------------------------- | | `dataOsManagerConfigs.imageTag` | `dataos-manager` image version, resolved from `${IMAGE_TAG}`. Provided by Modern. | | `dataOsManagerConfigs.logLevel` | Prefer `info` for production, `debug` temporarily during incidents. | | `dataOsManagerConfigs.replicas` | Keep as `1`; a single replica is sufficient. | | `dataOsManagerConfigs.nodeSelector` | Schedules `dataos-manager` onto the node pool labeled `dataos.io/purpose: core-kernel`. | | `kubeConfigBase64` | Base64-encoded kubeconfig that gives the DataOS Instance access to the AKS cluster. | ```yaml inputs: dataOsManagerConfigs: imageTag: ${IMAGE_TAG} logLevel: info replicas: 1 nodeSelector: dataos.io/purpose: core-kernel kubeConfigBase64: ${KUBECONFIG_BASE64} ``` #### Config Bundle | Attribute | Description | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `configs` | Config references applied to the kernel. `cloudflare:tmdc-prime` supplies the Cloudflare tunnel configuration for the Data Plane domain. This value is fixed: leave it as shown. | ```yaml configs: - cloudflare:tmdc-prime ``` #### Installation (Prime-based) Defines which installation manifests are applied for Data Plane components, pulled from Prime rather than a Git repository. | Attribute | Description | | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `artifactIdentifier` | The install artifact to apply, resolved from `${ARTIFACT_IDENTIFIER}`. Provided by Modern for the target release. Prefer a snapshot identifier (`artifact:aft_...`) over a release identifier, so a re-apply cannot pull a newer build. | | `installFileRootDir` | Root directory within the artifact for install files (`/install/dataplane-kernel-system`). | | `installFile`, `applicationsFile`, `valuesFile` | Install manifests and Helm-style values within the artifact. | | `onDemandCloudKernel` | When infrastructure provisioning generates a new output, `dataos-manager` creates a new Cloud Kernel JSON. **This value must remain `true`.** | | `envPurpose` | Environment purpose for the install (`prd`). | ```yaml installFromPrime: artifactIdentifier: ${ARTIFACT_IDENTIFIER} installFileRootDir: /install/dataplane-kernel-system installFile: release-prd-install.yaml applicationsFile: install.applications.yaml valuesFile: install.values.yaml onDemandCloudKernel: true envPurpose: prd ``` #### Platform databases Connection details for the external PostgreSQL databases the Data Plane uses. Supply the values provisioned during infrastructure setup. | Attribute | Description | | ------------------------------------------------------------------------ | ------------------------------------------------- | | `pgHost`, `pgPort` | PostgreSQL host and port. | | `systemHealthDbName`, `systemHealthDbUsername`, `systemHealthDbPassword` | Database name and credentials for System HealthD. | | `vestaDbName`, `vestaDbUsername`, `vestaDbPassword` | Database name and credentials for Vesta. | ```yaml pgHost: pgPort: "5432" systemHealthDbName: systemHealthDbUsername: systemHealthDbPassword: vestaDbName: vestaDbUsername: vestaDbPassword: ``` #### Cloud Endpoints Used for cloud integrations and endpoint resolution. ```yaml azureEndpointSuffix: core.windows.net awsEndpointSuffix: amazonaws.com ``` #### Node Selectors Ensures platform components schedule onto the intended node pool. | Attribute | Description | | ------------------------ | --------------------------------------------- | | `coreKernelNodeSelector` | Label selector for the core kernel node pool. | ```yaml coreKernelNodeSelector: dataos.io/purpose: core-kernel ``` {% hint style="info" %} **Do not change this:** the DataOS installer applies the same node label to node pools. {% endhint %} #### Features Optional platform feature toggles. | Attribute | Description | | ---------------------- | ------------------------------------------------------------------------------------ | | `cloudnativePg.enable` | Enables the CloudNativePG operator. Keep `false` unless your deployment requires it. | ```yaml features: cloudnativePg: enable: false ``` *** ## Troubleshooting
user apikey create fails The personal user's credentials or the target Instance are wrong. Verify `USER_ID`, your personal API key, and `DATAOS_FQDN`.
domain apply returns forbidden The application user is missing a role, or you are authenticating as yourself. Confirm the three roles on the application user, and that `USER_ID` and `APIKEY` belong to it.
Lint succeeds but nothing is provisioned The `-l` flag is still on the command. Rerun the apply without it.
Manifest decryption fails The identity decrypting the manifest lacks access to the key. Check its credentials and key-vault permissions.
Forge task fails The manifest does not match the deployed cluster. Validate the cluster's region and the values in the manifest.
Apply fails with an auth error Invalid or expired kubeconfig, or insufficient RBAC. Regenerate the kubeconfig with the required RBAC, re-encode it, and re-apply. If the failure is on the CLI authentication itself, confirm the application user's API key has not expired.
Pods stuck Pending The `dataos.io/purpose: core-kernel` label is missing, or the node pool lacks capacity. Label the nodes and scale the node pool.
Image pull failures Egress to the registry is blocked, or image pull secrets are missing for a private mirror. Allow TCP 443 to the registry and configure image pull secrets.
Tunnel does not establish Outbound egress to the Cloudflare tunnel edge is blocked, or DNS resolution is failing. The tunnel connects to `region1.v2.argotunnel.com` and `region2.v2.argotunnel.com` on port `7844`: UDP (QUIC) primary, TCP `7844` as fallback. If both are blocked, the tunnel cannot connect. If your firewall uses TLS inspection or SNI-based filtering, allow the SNI hostname pattern `*.cftunnel.com`. Confirm outbound NAT egress and DNS resolution from a worker node; the tunnel client retries automatically once egress is restored.
Reconciliation stalls (cluster unreachable) The AKS API is unreachable, or the kubeconfig identity lacks Kubernetes RBAC. Verify the kubeconfig's identity is bound to a Kubernetes RBAC role on the cluster.
Install step fails The Prime `artifactIdentifier` is wrong or unavailable for this release. Confirm the artifact identifier with Modern and that the Data Plane can reach Prime.
*** ## Best practices * Dedicate the `core-kernel` node pool to platform components. Do not co-schedule user Data Product workloads on it. * Monitor Azure quotas and AKS node-pool capacity ahead of production Data Product workload growth. * Capture provisioning logs from `dataos-ctl` and Azure Monitor during the apply for assertion and troubleshooting. * Store the manifest encrypted in version control; decrypt it only at apply time. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/operate/v1/dataos-operator/provision-data-plane/azure.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.