> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/operate/dataos-operator/provision-data-plane/aws.md).

# AWS

Provision AWS Data Plane

Deploy the DataOS Data Plane on an existing Amazon EKS cluster within your AWS account. Your SRE team is responsible for provisioning and operating the underlying AWS infrastructure in accordance with Modern’s published requirements. DataOS components are then installed on the cluster using a Data Plane manifest applied through `dataos-ctl`.

The Data Plane connects to the DataOS Instance hosted in Modern Cloud in one of two modes, which you choose before you start.

## Choose tunnel or ingress

| Mode        | Difference                                                                                                                                 |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Tunnel**  | No public load balancer. The Data Plane opens an outbound-only connection to Modern Cloud through Cloudflare.                              |
| **Ingress** | Public HTTP and TCP network load balancers (NLBs) backed by Elastic IPs in public subnets expose the Data Plane. Needs a gateway database. |

In tunnel mode, the Data Plane opens an outbound-only connection, so no inbound ports on your cluster are exposed. In ingress mode, the Data Plane is reachable through the public HTTP and TCP hostnames you provide.

## Prerequisites

<details>

<summary>1. Instance FQDN</summary>

Use a working DataOS Instance FQDN, for example `pacific-staging.instance.dataos.cloud`.

</details>

<details>

<summary>2. DataOS CLI</summary>

Install `dataos-ctl` on the bastion host, from the **Setup CLI** section on the Instance home page. Use the DataOS CLI version named in the release inputs below.

If the binary is not on the bastion host's `PATH`, call it by its full install path, for example `/home/jumpvmadmin/.dataos/v2/bin/dataos-ctl`. This guide uses the short form.

</details>

<details>

<summary>3. DataOS permissions</summary>

Your own DataOS user needs all three of these roles on the Instance, so that you can create the application user and apply the Data Plane manifest:

* `roles:id:user`
* `roles:id:operator`
* `roles:id:super-user`

Check them in **Bifrost** at `https://<instance-fqdn>/platform/bifrost/users`.

</details>

<details>

<summary>4. Database endpoints and credentials</summary>

The Data Plane connects to an external PostgreSQL database provisioned during infrastructure setup. Have these values ready; you supply them directly in the Data Plane manifest.

**PostgreSQL details**

* `PG_HOST`, `PG_PORT`: PostgreSQL host and port.

**System DB details**

* `SYSTEM_HEALTHD_DB_NAME`, `SYSTEM_HEALTHD_DB_USERNAME`, `SYSTEM_HEALTHD_DB_PASSWORD`: Database name and credentials for System HealthD.

**Vesta DB details**

* `VESTA_DB_NAME`, `VESTA_DB_USERNAME`, `VESTA_DB_PASSWORD`: Database name and credentials for Vesta.

**Gateway DB details (ingress only)**

* `GATEWAY_DB_NAME`, `GATEWAY_DB_USERNAME`, `GATEWAY_DB_PASSWORD`: Database name and credentials for the gateway.

**Vesta object storage details**

Vesta also keeps its objects in an S3 bucket. Have these values ready for the `vestaObjectStore` block:

* S3 bucket region, bucket name, folder (relative path), access key, secret key, and edge endpoint.

</details>

<details>

<summary>5. AWS credentials and kubeconfig access</summary>

**Credentials**

* **AWS\_ACCESS\_KEY\_ID:** Access key ID for the AWS identity used during Data Plane setup.
* **AWS\_SECRET\_ACCESS\_KEY:** Secret access key associated with the AWS access key ID.

**Kubeconfig access**

You need a valid kubeconfig for the provisioned EKS cluster, and `kubectl` must reach the cluster with it. You base64-encode this kubeconfig during deployment.

**Base64-encode the kubeconfig**

Once `kubectl get nodes` reaches the cluster, base64-encode the kubeconfig into `KUBECONFIG_BASE64`. The encoding command differs by operating system, because `base64` takes different flags on each and Windows has no `base64` at all. Run the one that matches the machine you are deploying from.

{% tabs %}
{% tab title="Linux" %}

```bash
export KUBECONFIG_BASE64=$(kubectl config view --minify --flatten | base64 -w 0)
```

{% endtab %}

{% tab title="macOS" %}

```bash
export KUBECONFIG_BASE64=$(kubectl config view --minify --flatten | base64 | tr -d '\n')
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:KUBECONFIG_BASE64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes((kubectl config view --minify --flatten | Out-String)))
```

{% endtab %}
{% endtabs %}

This value gives the DataOS Instance access to the EKS API. Treat it as a secret.

</details>

<details>

<summary>6. Release inputs (from Modern)</summary>

These values are supplied by Modern for the release you are installing; they are not customer-derivable. Confirm all of them before you apply.

| Input                | Manifest field       | Example                      |
| -------------------- | -------------------- | ---------------------------- |
| Install artifact     | `artifactIdentifier` | `<install-artifact>`         |
| DataOS Manager image | `imageTag`           | `<dataos-manager-image-tag>` |
| DataOS CLI version   | N/A                  | `<cli-version>`              |

The examples are placeholders. Take the exact values from the release notes for the release you are installing. Install the named CLI version on the bastion host, because an older CLI may not support the manifest fields the release expects.

In production, use the snapshot form of `artifactIdentifier` (`artifact:aft_abc123abc123`), which pins one exact snapshot. The release form (`release:dataos-v2-install-manifests:prd_abc123abc123`) always installs the latest artifact behind that release, so a re-apply can pick up a newer build.

</details>

<details>

<summary>7. Ingress networking (ingress mode only)</summary>

Skip this item for tunnel mode. For ingress mode, have these ready before you apply:

* `FQDN`: Hostname for HTTP ingress.
* `TCP_FQDN`: Hostname for TCP ingress.
* `PUBLIC_SUBNET_1`, `PUBLIC_SUBNET_2`, and optionally `PUBLIC_SUBNET_3`: Public subnet IDs in your VPC.
* `EIP_ID_1`, `EIP_ID_2`, and optionally `EIP_ID_3`: Elastic IP allocation IDs for the HTTP NLB.
* `TCP_EIP_ID_1`, `TCP_EIP_ID_2`, and optionally `TCP_EIP_ID_3`: Elastic IP allocation IDs for the TCP NLB.

Provide the same number of Elastic IPs as subnets for both load balancers. If your VPC has only two public subnets, use two subnets and two Elastic IPs for each.

</details>

## Deploy Data Plane

Work through the steps in order, and run every command from the bastion host. Each step prepares one of the inputs the final `dataos-ctl domain apply` needs. Those inputs are a deployment application user and API key, a base64 kubeconfig for the EKS cluster, the variables the manifest resolves, and the Data Plane manifest for your mode.

### 1. Authenticate from the bastion host

Provisioning is always initiated from the **bastion host** (for example, `jumpvmadmin`). In production the EKS API server is restricted to authorized CIDR ranges, so both `kubectl` and `dataos-ctl` must run from inside your network. Confirm the access path with your infrastructure team.

The bastion host keeps no logged-in session, so you never run `dataos-ctl init` or `dataos-ctl login`. Every command carries these four values instead; replace the placeholders with your own each time:

| Variable           | Purpose                           |
| ------------------ | --------------------------------- |
| `USER_ID`          | User the command authenticates as |
| `DATAOS_FQDN`      | Target DataOS Instance domain     |
| `APIKEY`           | API key belonging to `USER_ID`    |
| `DATAOS_TENANT_ID` | Always `system`                   |

Which user they name changes as you go. Your **personal user** creates the application user's API key in Step 3, and the **application user** runs every `domain` command after that. So start by collecting your own two values.

{% hint style="warning" %}
Never provision a Data Plane with your personal API key. It ties the Data Plane to an individual account.
{% endhint %}

**1.1 Get your user ID.** It is shown on your profile in the DataOS UI, and by `dataos-ctl user get` on a machine that has a logged-in CLI session:

```bash
dataos-ctl user get
```

```
INFO[0000] 😃 user get...
INFO[0001] 😃 user get...complete

    NAME    │        ID        │  TYPE  │       EMAIL       │             TAGS
────────────┼──────────────────┼────────┼───────────────────┼──────────────────────────────
 iamgroot   │ iamgroottmdcio   │ person │ iamgroot@tmdc.io  │ roles:id:user,
            │                  │        │                   │ roles:id:operator,
            │                  │        │                   │ users:id:iamgroottmdcio
```

The value under **ID** is your `USER_ID`.

**1.2 Create your personal API key.** Issue it from **Profile → Tokens** in the DataOS UI, or from a machine with a logged-in CLI session:

```bash
dataos-ctl user apikey create --id <your-user-id> --name <key-name> -d 24h
```

This key authenticates the one command in Step 3 that creates the application user's key. It is not used to provision anything.

### 2. Create the application user

Provision a dedicated **application user** for this Data Plane. In **Bifrost** on your DataOS Instance (`https://<instance-fqdn>/platform/bifrost`), create an application user named `<dataplane-name>-user` and assign these roles:

* `roles:id:user`
* `roles:id:operator`
* `roles:id:super-user`

The application user owns the deployment, which keeps the Data Plane independent of any individual account.

### 3. Create an API key for the application user

This is the one command you run as your **personal user**, so `USER_ID` and `APIKEY` are your own. Issue a short-lived key (for example, a 24-hour lifetime) for the application user.

```bash
USER_ID=<your-user-id> \
  DATAOS_FQDN=<instance-fqdn> \
  APIKEY=<your-personal-apikey-token> \
  DATAOS_TENANT_ID=system \
  dataos-ctl user apikey create -i <dataplane-name>-user -d 24h
```

The command prints the new API key in the terminal. Copy it. You supply it as `APIKEY` in the lint, apply, and monitor commands that follow.

### 4. Base64-encode kubeconfig

Confirm `kubectl` reaches the cluster, then base64-encode the kubeconfig into `KUBECONFIG_BASE64`. The kubeconfig must already use `aws-iam-authenticator`, as shown in the prerequisites.

```bash
kubectl get nodes   # confirm access first
```

The encoding command differs by operating system, because `base64` takes different flags on each and Windows has no `base64` at all. Run the one that matches the machine you are deploying from.

{% tabs %}
{% tab title="Linux" %}

```bash
export KUBECONFIG_BASE64=$(kubectl config view --minify --flatten | base64 -w 0)
```

{% endtab %}

{% tab title="macOS" %}

```bash
export KUBECONFIG_BASE64=$(kubectl config view --minify --flatten | base64 | tr -d '\n')
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:KUBECONFIG_BASE64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes((kubectl config view --minify --flatten | Out-String)))
```

{% endtab %}
{% endtabs %}

This value gives the DataOS Instance access to the EKS API. Treat it as a secret.

### 5. Set the variables the manifest resolves

`dataos-ctl domain apply` resolves `${...}` references in the manifest from the shell you run it in. Set these values there before you apply. These are shared by both modes:

```bash
export KUBECONFIG_BASE64=<base64-encoded-kubeconfig>   # from the previous step
export IMAGE_TAG=<dataos-manager-image-tag>            # provided by Modern
export ARTIFACT_IDENTIFIER=<install-artifact>          # provided by Modern
export REGION=<aws-region>                             # region of the EKS cluster
export AWS_ACCESS_KEY_ID=<access-key-id>
export AWS_SECRET_ACCESS_KEY=<secret-access-key>
```

For **ingress mode**, also set these. Drop the third subnet and Elastic IP lines if your VPC has only two public subnets.

```bash
export FQDN=<http-hostname>
export TCP_FQDN=<tcp-hostname>
export PUBLIC_SUBNET_1=<subnet-id>
export PUBLIC_SUBNET_2=<subnet-id>
export PUBLIC_SUBNET_3=<subnet-id>
export EIP_ID_1=<eipalloc-...>
export EIP_ID_2=<eipalloc-...>
export EIP_ID_3=<eipalloc-...>
export TCP_EIP_ID_1=<eipalloc-...>
export TCP_EIP_ID_2=<eipalloc-...>
export TCP_EIP_ID_3=<eipalloc-...>
```

| Variable                                     | Purpose                                                           | Example          |
| -------------------------------------------- | ----------------------------------------------------------------- | ---------------- |
| `DATAPLANE_ID`                               | Stable Data Plane identifier                                      | `acme-aws-dp-01` |
| `INSTANCE_TENANT_ID`                         | DataOS Instance this Data Plane attaches to                       | `pacific-ocean`  |
| `KUBECONFIG_BASE64`                          | Base64-encoded kubeconfig from the previous step                  | :                |
| `IMAGE_TAG`                                  | DataOS Manager image tag to pin (from Modern)                     | :                |
| `ARTIFACT_IDENTIFIER`                        | Install artifact to apply (from Modern)                           | :                |
| `REGION`                                     | AWS region where the EKS cluster runs                             | `ap-south-1`     |
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | AWS credentials for ongoing AWS operations                        | :                |
| `FQDN`, `TCP_FQDN`                           | Ingress only. HTTP and TCP hostnames                              | :                |
| `PUBLIC_SUBNET_*`                            | Ingress only. Public subnet IDs                                   | :                |
| `EIP_ID_*`, `TCP_EIP_ID_*`                   | Ingress only. Elastic IP allocation IDs for the HTTP and TCP NLBs | :                |

`DATAPLANE_ID` and `INSTANCE_TENANT_ID` are supplied inline on the lint and apply commands in Step 7, so you do not need to set them here.

### 6. Prepare manifest

Copy the sample manifest for your mode. You only need to fill in a handful of values; everything else is fixed.

**What you fill in**

Identity and naming:

* `name` / `DATAPLANE_ID`: the dataplane identifier. Use a name that encodes the Instance and cloud, for example `<instance-name>-aws-dp-01`. Do not append a date suffix.
* `description`: a short, Instance- and Tenant-specific description for traceability.
* `workpiece.name`: must end with the `.dataplane` suffix. This becomes the Data Plane domain.

Cluster and cloud access:

* `directInputs.region` and the `AWS_REGION` entry in `kubeAuthConfigs`: resolved from `${REGION}` (the AWS region where your EKS cluster runs). Do not set `kernelRequest.region`.
* `kubeConfigBase64`: the base64 kubeconfig from Step 4.
* `directInputs` and `kubeAuthConfigs`: your AWS credentials from Step 5.

Version and install:

* `dataOsManagerConfigs.imageTag`: resolved from `${IMAGE_TAG}` (the `dataos-manager` version from Modern).
* `installFromPrime.artifactIdentifier`: resolved from `${ARTIFACT_IDENTIFIER}` (the install artifact from Modern).

PostgreSQL databases and object store:

* `pgHost`, `pgPort`: the PostgreSQL endpoint. `pgPort` is a quoted string, for example `"5432"`.
* `systemHealthDb*` and `vestaDb*` values: database names and credentials for System HealthD and Vesta.
* `gatewayDb*` values (ingress only): database name and credentials for the gateway.
* `vestaObjectStore.s3` values: the S3 bucket Vesta uses.

Ingress only:

* `inputs.ingress`: the FQDNs, public subnets, and Elastic IP IDs, resolved from the variables you set in Step 5.

Leave every other field exactly as shown.

{% tabs %}
{% tab title="Tunnel" %}

```yaml
name: ${DATAPLANE_ID}
version: v1alpha
entity: domain
type: dataplane
description: the ${DATAPLANE_ID} dataplane for ${INSTANCE_TENANT_ID} instance
v1alpha:
  dataplane:
    name: ${DATAPLANE_ID}
    networkType: tunnel
    workpiece:
      name: ${DATAPLANE_ID}-${INSTANCE_TENANT_ID}.dataplane
      zone: dataos.cloud
      kernelRequest:
        template: dataplane-kernel-system-existing-compute-tunnel-v1
        cloud: aws
        inputs:
          dataOsManagerConfigs:
            imageTag: ${IMAGE_TAG}
            logLevel: info
            replicas: 1
            nodeSelector:
              dataos.io/purpose: core-kernel
          kubeConfigBase64: ${KUBECONFIG_BASE64}
        configs:
          - cloudflare:tmdc-prime
        directInputs:
          aws_access_key_id: ${AWS_ACCESS_KEY_ID}
          aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
          region: ${REGION}
      kubeAuthConfigs:
        - key: AWS_ACCESS_KEY_ID
          value: ${AWS_ACCESS_KEY_ID}
          base64: false
          envVar: true
        - key: AWS_SECRET_ACCESS_KEY
          value: ${AWS_SECRET_ACCESS_KEY}
          base64: false
          envVar: true
        - key: AWS_REGION
          value: ${REGION}
          base64: false
          envVar: true
    dataPlaneKernelSystemInstall:
      installFromPrime:
        artifactIdentifier: release:dataos-v2-install-manifests:prd_fqj0qvg8t3pc
        installFileRootDir: /install/dataplane-kernel-system
        installFile: release-prd-install.yaml
        applicationsFile: install.applications.yaml
        valuesFile: install.values.yaml
        onDemandCloudKernel: true
      envPurpose: prd
      pgHost: ""
      pgPort: "5432"
      systemHealthDbName: ""
      systemHealthDbUsername: ""
      systemHealthDbPassword: ""
      vestaDbName: ""
      vestaDbUsername: ""
      vestaDbPassword: ""
      vestaObjectStore:
        s3:
          region: ""
          bucket: ""
          relativePath: ""
          accessKey: ""
          secretKey: ""
          edgeEndpoint: ""
      azureEndpointSuffix: core.windows.net
      awsEndpointSuffix: amazonaws.com
      gcpEndpointSuffix: googleapis.com
      coreKernelNodeSelector:
        dataos.io/purpose: core-kernel
      features:
        cloudnativePg:
          enable: false
        clusterAutoScaler:
          enable: false
```

{% endtab %}

{% tab title="Ingress" %}
Drop the third subnet and Elastic IP entries if your VPC has only two public subnets.

```yaml
name: ${DATAPLANE_ID}
version: v1alpha
entity: domain
type: dataplane
description: the ${DATAPLANE_ID} dataplane for ${INSTANCE_TENANT_ID} instance
v1alpha:
  dataplane:
    name: ${DATAPLANE_ID}
    tenantDomainNames:
      - system-entities
    networkType: ingress
    workpiece:
      name: ${DATAPLANE_ID}-${INSTANCE_TENANT_ID}.dataplane
      zone: dataos.cloud
      kernelRequest:
        template: dataplane-kernel-system-existing-compute-ingress-v1
        cloud: aws
        inputs:
          dataOsManagerConfigs:
            imageTag: ${IMAGE_TAG}
            logLevel: info
            replicas: 1
            nodeSelector:
              dataos.io/purpose: core-kernel
          kubeConfigBase64: ${KUBECONFIG_BASE64}
          ingress:
            fqdn: ${FQDN}
            tcpFqdn: ${TCP_FQDN}
            subnets:
              - ${PUBLIC_SUBNET_1}
              - ${PUBLIC_SUBNET_2}
              - ${PUBLIC_SUBNET_3}
            eipIds:
              - ${EIP_ID_1}
              - ${EIP_ID_2}
              - ${EIP_ID_3}
            tcpEipIds:
              - ${TCP_EIP_ID_1}
              - ${TCP_EIP_ID_2}
              - ${TCP_EIP_ID_3}
        directInputs:
          aws_access_key_id: ${AWS_ACCESS_KEY_ID}
          aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
          region: ${REGION}
      kubeAuthConfigs:
        - key: AWS_ACCESS_KEY_ID
          value: ${AWS_ACCESS_KEY_ID}
          base64: false
          envVar: true
        - key: AWS_SECRET_ACCESS_KEY
          value: ${AWS_SECRET_ACCESS_KEY}
          base64: false
          envVar: true
        - key: AWS_REGION
          value: ${REGION}
          base64: false
          envVar: true
    dataPlaneKernelSystemInstall:
      installFromPrime:
        artifactIdentifier: ${ARTIFACT_IDENTIFIER}
        installFileRootDir: /install/dataplane-kernel-system
        installFile: release-prd-install.yaml
        applicationsFile: install.applications.yaml
        valuesFile: install.values.yaml
        onDemandCloudKernel: true
      envPurpose: prd
      pgHost: <PG_HOST>
      pgPort: "5432"
      systemHealthDbName: <SYSTEM_HEALTHD_DB_NAME>
      systemHealthDbUsername: <SYSTEM_HEALTHD_DB_USERNAME>
      systemHealthDbPassword: <SYSTEM_HEALTHD_DB_PASSWORD>
      gatewayDbName: <GATEWAY_DB_NAME>
      gatewayDbUsername: <GATEWAY_DB_USERNAME>
      gatewayDbPassword: <GATEWAY_DB_PASSWORD>
      vestaDbName: <VESTA_DB_NAME>
      vestaDbUsername: <VESTA_DB_USERNAME>
      vestaDbPassword: <VESTA_DB_PASSWORD>
      vestaObjectStore:
        s3:
          region: <BUCKET_NAME_REGION>
          bucket: <BUCKET_NAME>
          relativePath: <FOLDER_NAME>
          accessKey: <ACCESS_KEY>
          secretKey: <SECRET_KEY>
          edgeEndpoint: <CLOUD_ENDPOINT>
      azureEndpointSuffix: core.windows.net
      awsEndpointSuffix: amazonaws.com
      gcpEndpointSuffix: googleapis.com
      coreKernelNodeSelector:
        dataos.io/purpose: core-kernel
      features:
        clusterAutoScaler:
          enable: false
        cloudnativePg:
          enable: false
        trinoEngineGateway:
          enable: true
```

{% endtab %}
{% endtabs %}

{% hint style="warning" %}
The kubeconfig, AWS credentials, and database credentials inside the manifest are sensitive. Store the manifest encrypted and limit who can decrypt it.
{% endhint %}

### 7. Lint and apply the Data Plane manifest

Authenticate as the **application user** with the API key from Step 3, then **lint** the manifest before applying. Save the manifest from Step 6 as `aws-dataplane-tunnel.yaml` or `aws-dataplane-ingress.yaml`, and use that file name below where you see `<mode>`. `KUBECONFIG_BASE64`, `IMAGE_TAG`, `REGION`, and, for ingress, the networking variables are resolved from the variables you set in Step 5.

**Lint** (validate the manifest with `-l`):

```bash
USER_ID=<dataplane-name>-user \
  DATAOS_FQDN=<instance-fqdn> \
  APIKEY=<application-user-apikey-token> \
  DATAOS_TENANT_ID=system \
  INSTANCE_TENANT_ID=<instance-name> \
  DATAPLANE_ID=<dataplane-name> \
  dataos-ctl domain apply -f aws-dataplane-<mode>.yaml -l
```

Lint validates the manifest and resolves the variables without provisioning anything. If it succeeds but nothing is created, you left the `-l` flag on.

**Apply** (once lint passes, rerun the same command without `-l`):

```bash
USER_ID=<dataplane-name>-user \
  DATAOS_FQDN=<instance-fqdn> \
  APIKEY=<application-user-apikey-token> \
  DATAOS_TENANT_ID=system \
  INSTANCE_TENANT_ID=<instance-name> \
  DATAPLANE_ID=<dataplane-name> \
  dataos-ctl domain apply -f aws-dataplane-<mode>.yaml
```

The DataOS Instance validates the request and starts installing the Data Plane onto your cluster.

<details>

<summary>What happens after you apply</summary>

When you run the apply, the DataOS Instance reads the `kubeConfigBase64` you supplied and uses it to reach the EKS API for the first time. It installs `dataos-manager` onto the cluster, which pulls the install manifests defined in `installFromPrime` and lays down the DataOS components.

**Tunnel mode.** A tunnel client inside the cluster opens an outbound, encrypted connection from a worker node out through the NAT Gateway to Modern Cloud. No inbound ports on your cluster are exposed. Once the tunnel is established, the DataOS Instance registers the Data Plane and manages it over that connection.

**Ingress mode.** The HTTP and TCP network load balancers come up on the public subnets and Elastic IPs you supplied, and the DataOS Instance registers the Data Plane through the hostnames in `inputs.ingress`.

The kubeconfig is used only for this DataOS Instance-to-cluster management path. It is not part of your data traffic. Every hop is TLS-encrypted end to end.

</details>

### 8. Monitor provisioning

Provisioning may take some time to complete. Check the Data Plane status until it reports active.

```bash
USER_ID=<dataplane-name>-user \
  DATAOS_FQDN=<instance-fqdn> \
  APIKEY=<application-user-apikey-token> \
  DATAOS_TENANT_ID=system \
  dataos-ctl domain get -t dataplane -n <dataplane-name> -v v1alpha -d
```

To watch the domain list refresh instead of rerunning the command, use `domain ls -r`:

```bash
USER_ID=<dataplane-name>-user \
  DATAOS_FQDN=<instance-fqdn> \
  APIKEY=<application-user-apikey-token> \
  DATAOS_TENANT_ID=system \
  dataos-ctl domain ls -r
```

{% hint style="info" %}
Spinning up the infrastructure and installing the platform typically completes within 30 minutes. The Instance reaches the cluster, installs `dataos-manager`, and waits for the tunnel (or the ingress endpoints) to come up. If the status has not changed well past that, follow the `dataos-manager` logs as shown under [Check status](#check-status).
{% endhint %}

## Manage Data Plane

### Check status

```bash
USER_ID=<dataplane-name>-user \
  DATAOS_FQDN=<instance-fqdn> \
  APIKEY=<application-user-apikey-token> \
  DATAOS_TENANT_ID=system \
  dataos-ctl domain get
#add -a to include Data Planes created by other users
```

{% hint style="info" %}
If the command above shows <mark style="color:$success;">`workpiece-forge-kernel: success`</mark>, the infrastructure setup is complete.
{% endhint %}

Then access the Data Plane cluster with `kubectl` and inspect the `dataos-manager` logs.

```bash
kubectl logs -n <instance_name>-0-dsm dataos-manager-0 -f
```

### Describe applied configuration

```bash
USER_ID=<dataplane-name>-user \
  DATAOS_FQDN=<instance-fqdn> \
  APIKEY=<application-user-apikey-token> \
  DATAOS_TENANT_ID=system \
  dataos-ctl domain get -t dataplane -n <dataplane-name> -v v1alpha -d
```

## Dataplane configuration

A field-by-field reference for the Data Plane manifest. Where a field differs by mode, the table says so.

#### Metadata

Top-level fields that identify the Data Plane resource.

| Attribute     | Description                                                       |
| ------------- | ----------------------------------------------------------------- |
| `name`        | Keep it stable and readable. It becomes the dataplane identifier. |
| `version`     | API version for the resource spec.                                |
| `entity`      | Always `domain`.                                                  |
| `type`        | Always `dataplane`. It declares what you are provisioning.        |
| `description` | Make it environment- and Tenant-specific for traceability.        |

```yaml
name: ${DATAPLANE_ID}
version: v1alpha
entity: domain
type: dataplane
description: the ${DATAPLANE_ID} dataplane for ${INSTANCE_TENANT_ID} instance
```

{% hint style="info" %}
Use a naming convention that clearly encodes the Instance and cloud. Example: `<instance-name>-<cloud>-dp-01`.
{% endhint %}

#### Dataplane

Core Data Plane identity and DataOS Instance connectivity.

| Attribute           | Description                                                                                                                                                                                     |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dataplane.name`    | The dataplane identifier. Do not append a date suffix.                                                                                                                                          |
| `tenantDomainNames` | Default domain mappings (platform-specific).                                                                                                                                                    |
| `networkType`       | Decides how the Data Plane reaches the DataOS Instance. `tunnel` uses outbound-only Cloudflare tunneling. `ingress` uses public HTTP and TCP load balancers. It must match the kernel template. |

```yaml
v1alpha:
  dataplane:
    name: ${DATAPLANE_ID}
    tenantDomainNames:
      - system-entities
    networkType: tunnel   # or ingress
```

#### Workpiece

Represents the execution unit for the provisioning pipeline.

| Attribute        | Description                                                                                                                        |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `workpiece.name` | Must end with the `.dataplane` suffix. This produces the Data Plane domain, for example `awsdataplane0123.dataplane.dataos.cloud`. |
| `workpiece.zone` | Always set to `dataos.cloud`.                                                                                                      |

```yaml
workpiece:
  name: ${DATAPLANE_ID}-${INSTANCE_TENANT_ID}.dataplane
  zone: dataos.cloud
```

#### Kernel Request

Selects the kernel provisioning blueprint and target cloud.

| Attribute  | Description                                                                                                                                                                                                                                                       |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template` | Defines which kernel provisioning blueprint is used. `dataplane-kernel-system-existing-compute-tunnel-v1` installs the platform onto an existing EKS cluster in tunnel mode. `dataplane-kernel-system-existing-compute-ingress-v1` does the same in ingress mode. |
| `cloud`    | Selects cloud-specific behavior. Use `aws` for both modes.                                                                                                                                                                                                        |

```yaml
kernelRequest:
  template: dataplane-kernel-system-existing-compute-tunnel-v1   # or ...-ingress-v1
  cloud: aws
```

{% hint style="warning" %}
Do not set `kernelRequest.region`. The region belongs in `directInputs.region` and in the `AWS_REGION` entry of `kubeAuthConfigs`.
{% endhint %}

`inputs`, `directInputs`, and (tunnel only) `configs` are siblings under `kernelRequest`. `kubeAuthConfigs` is a sibling of `kernelRequest`, under `workpiece`.

#### Inputs (DataOS Manager, kubeconfig)

| Attribute                           | Description                                                                                               |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------- |
| `dataOsManagerConfigs.imageTag`     | `dataos-manager` image version, resolved from `${IMAGE_TAG}`. Provided by Modern.                         |
| `dataOsManagerConfigs.logLevel`     | Prefer `info` for production, `debug` temporarily during incidents.                                       |
| `dataOsManagerConfigs.replicas`     | Keep as `1`; a single replica is sufficient.                                                              |
| `dataOsManagerConfigs.nodeSelector` | Schedules `dataos-manager` onto the node pool labeled `dataos.io/purpose: core-kernel`.                   |
| `kubeConfigBase64`                  | Base64-encoded kubeconfig that gives the DataOS Instance access to the EKS cluster.                       |
| `ingress`                           | Ingress mode only. Public hostnames, subnets, and Elastic IPs. See [Ingress](#ingress-ingress-mode-only). |

```yaml
inputs:
  dataOsManagerConfigs:
    imageTag: ${IMAGE_TAG}
    logLevel: info
    replicas: 1
    nodeSelector:
      dataos.io/purpose: core-kernel
  kubeConfigBase64: ${KUBECONFIG_BASE64}
```

#### Config Bundle (tunnel mode only)

| Attribute | Description                                                                                                                                                                         |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `configs` | Config references applied to the kernel. `cloudflare:tmdc-prime` supplies the Cloudflare tunnel configuration for the Data Plane domain. This value is fixed, so leave it as shown. |

```yaml
configs:
  - cloudflare:tmdc-prime
```

{% hint style="info" %}
Omit the entire `configs` block in an ingress manifest.
{% endhint %}

#### Ingress (ingress mode only)

Public HTTP and TCP network load balancers, backed by Elastic IPs in public subnets you already have. Add this block under `inputs`, at the same level as `kubeConfigBase64`.

| Attribute   | Description                                                 |
| ----------- | ----------------------------------------------------------- |
| `fqdn`      | Hostname for HTTP ingress.                                  |
| `tcpFqdn`   | Hostname for TCP ingress.                                   |
| `subnets`   | Public subnet IDs. Use two or three.                        |
| `eipIds`    | Elastic IP allocation IDs for the HTTP NLB. One per subnet. |
| `tcpEipIds` | Elastic IP allocation IDs for the TCP NLB. One per subnet.  |

```yaml
ingress:
  fqdn: ${FQDN}
  tcpFqdn: ${TCP_FQDN}
  subnets:
    - ${PUBLIC_SUBNET_1}
    - ${PUBLIC_SUBNET_2}
    - ${PUBLIC_SUBNET_3}
  eipIds:
    - ${EIP_ID_1}
    - ${EIP_ID_2}
    - ${EIP_ID_3}
  tcpEipIds:
    - ${TCP_EIP_ID_1}
    - ${TCP_EIP_ID_2}
    - ${TCP_EIP_ID_3}
```

{% hint style="info" %}
Drop the third subnet and Elastic IP entries if your VPC has only two public subnets. Unlike Azure, AWS ingress uses FQDNs, subnets, and Elastic IP IDs, not `ip`, `tcpIp`, or `resourceGroupName`.
{% endhint %}

#### Auth (AWS credentials and region)

AWS credentials and region used by the platform for ongoing AWS API operations. Supply them in two places. `directInputs` passes them to the kernel provisioning inputs, and `kubeAuthConfigs` injects them into the cluster as environment variables. Both reference the same values.

| Attribute | Description                                                                               |
| --------- | ----------------------------------------------------------------------------------------- |
| `key`     | Environment variable name: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, or `AWS_REGION`. |
| `value`   | The credential or region value.                                                           |
| `base64`  | Keep `false` for every entry in this guide.                                               |
| `envVar`  | Set `true` to inject the value into the cluster as an environment variable.               |

```yaml
kernelRequest:
  directInputs:
    aws_access_key_id: ${AWS_ACCESS_KEY_ID}
    aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    region: ${REGION}
kubeAuthConfigs:
  - key: AWS_ACCESS_KEY_ID
    value: ${AWS_ACCESS_KEY_ID}
    base64: false
    envVar: true
  - key: AWS_SECRET_ACCESS_KEY
    value: ${AWS_SECRET_ACCESS_KEY}
    base64: false
    envVar: true
  - key: AWS_REGION
    value: ${REGION}
    base64: false
    envVar: true
```

#### Installation (Prime-based)

Defines which installation manifests are applied for Data Plane components, pulled from Prime rather than a Git repository.

| Attribute                                       | Description                                                                                                                                                                                                                             |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `artifactIdentifier`                            | The install artifact to apply, resolved from `${ARTIFACT_IDENTIFIER}`. Provided by Modern for the target release. Prefer a snapshot identifier (`artifact:aft_...`) over a release identifier, so a re-apply cannot pull a newer build. |
| `installFileRootDir`                            | Root directory within the artifact for install files (`/install/dataplane-kernel-system`).                                                                                                                                              |
| `installFile`, `applicationsFile`, `valuesFile` | Install manifests and Helm-style values within the artifact.                                                                                                                                                                            |
| `onDemandCloudKernel`                           | When infrastructure provisioning generates a new output, `dataos-manager` creates a new Cloud Kernel JSON. **This value must remain `true`.**                                                                                           |
| `envPurpose`                                    | Environment purpose for the install (`prd`).                                                                                                                                                                                            |

```yaml
installFromPrime:
  artifactIdentifier: ${ARTIFACT_IDENTIFIER}
  installFileRootDir: /install/dataplane-kernel-system
  installFile: release-prd-install.yaml
  applicationsFile: install.applications.yaml
  valuesFile: install.values.yaml
  onDemandCloudKernel: true
envPurpose: prd
```

#### PostgreSQL databases

Databases provisioned on a PostgreSQL server: one for System HealthD and one for Vesta, plus one for the gateway in ingress mode. Supply the connection details and credentials provisioned during infrastructure setup.

| Attribute                                                                | Description                                                       |
| ------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| `pgHost`, `pgPort`                                                       | PostgreSQL host and port.                                         |
| `systemHealthDbName`, `systemHealthDbUsername`, `systemHealthDbPassword` | Database name and credentials for System HealthD.                 |
| `vestaDbName`, `vestaDbUsername`, `vestaDbPassword`                      | Database name and credentials for Vesta.                          |
| `gatewayDbName`, `gatewayDbUsername`, `gatewayDbPassword`                | Ingress mode only. Database name and credentials for the gateway. |

```yaml
pgHost: <PG_HOST>
pgPort: "5432"
systemHealthDbName: <SYSTEM_HEALTHD_DB_NAME>
systemHealthDbUsername: <SYSTEM_HEALTHD_DB_USERNAME>
systemHealthDbPassword: <SYSTEM_HEALTHD_DB_PASSWORD>
vestaDbName: <VESTA_DB_NAME>
vestaDbUsername: <VESTA_DB_USERNAME>
vestaDbPassword: <VESTA_DB_PASSWORD>
# Ingress mode only
gatewayDbName: <GATEWAY_DB_NAME>
gatewayDbUsername: <GATEWAY_DB_USERNAME>
gatewayDbPassword: <GATEWAY_DB_PASSWORD>
```

#### Vesta object store

The S3 bucket where Vesta keeps its objects.

| Attribute      | Description                              |
| -------------- | ---------------------------------------- |
| `region`       | AWS region of the S3 bucket.             |
| `bucket`       | Bucket name.                             |
| `relativePath` | Folder inside the bucket.                |
| `accessKey`    | Access key with access to the bucket.    |
| `secretKey`    | Secret key for the access key.           |
| `edgeEndpoint` | Cloud endpoint used to reach the bucket. |

```yaml
vestaObjectStore:
  s3:
    region: <BUCKET_NAME_REGION>
    bucket: <BUCKET_NAME>
    relativePath: <FOLDER_NAME>
    accessKey: <ACCESS_KEY>
    secretKey: <SECRET_KEY>
    edgeEndpoint: <CLOUD_ENDPOINT>
```

#### Cloud Endpoints

Used for cloud integrations and endpoint resolution.

```yaml
azureEndpointSuffix: core.windows.net
awsEndpointSuffix: amazonaws.com
gcpEndpointSuffix: googleapis.com
```

#### Node Selectors

Ensures platform components schedule onto the intended node pool.

| Attribute                | Description                                   |
| ------------------------ | --------------------------------------------- |
| `coreKernelNodeSelector` | Label selector for the core kernel node pool. |

```yaml
coreKernelNodeSelector:
  dataos.io/purpose: core-kernel
```

{% hint style="info" %}
**Do not change this:** the DataOS installer applies the same node label to node pools.
{% endhint %}

#### Features

Optional platform feature toggles.

| Attribute                   | Description                                                                          |
| --------------------------- | ------------------------------------------------------------------------------------ |
| `clusterAutoScaler.enable`  | Enables the cluster autoscaler. Keep `false` unless your deployment requires it.     |
| `cloudnativePg.enable`      | Enables the CloudNativePG operator. Keep `false` unless your deployment requires it. |
| `trinoEngineGateway.enable` | Enables the Trino engine gateway. Set to `true` for both AWS modes.                  |

```yaml
features:
  clusterAutoScaler:
    enable: false
  cloudnativePg:
    enable: false
  trinoEngineGateway:
    enable: true
```

## Troubleshooting

<details>

<summary><code>user apikey create</code> fails</summary>

The personal user's credentials or the target Instance are wrong. Verify `USER_ID`, your personal API key, and `DATAOS_FQDN`.

</details>

<details>

<summary><code>domain apply</code> returns forbidden</summary>

The application user is missing a role, or you are authenticating as yourself. Confirm the three roles on the application user, and that `USER_ID` and `APIKEY` belong to it.

</details>

<details>

<summary>Lint succeeds but nothing is provisioned</summary>

The `-l` flag is still on the command. Rerun the apply without it.

</details>

<details>

<summary>Manifest decryption fails</summary>

The identity decrypting the manifest lacks access to the key. Check its AWS credentials and KMS key permissions.

</details>

<details>

<summary>Forge task fails</summary>

The target region or availability zones are wrong. Validate `REGION` and that the availability zones in use exist in that region.

</details>

<details>

<summary>Apply fails with an auth error</summary>

Invalid or expired kubeconfig, or insufficient RBAC. Regenerate the kubeconfig with the required RBAC, re-encode it, and re-apply. If the failure is on the CLI authentication itself, confirm the application user's API key has not expired.

</details>

<details>

<summary>Pods stuck <code>Pending</code></summary>

The `dataos.io/purpose: core-kernel` label is missing, or the node pool lacks capacity. Label the nodes and scale the node pool.

</details>

<details>

<summary>Image pull failures</summary>

Egress to the registry is blocked, or image pull secrets are missing for a private mirror. Allow TCP 443 to the registry and configure image pull secrets.

</details>

<details>

<summary>Tunnel does not establish (tunnel mode)</summary>

Outbound egress to the Cloudflare tunnel edge is blocked, or DNS resolution is failing. The tunnel connects to `region1.v2.argotunnel.com` and `region2.v2.argotunnel.com` on port `7844`, with UDP (QUIC) as the primary and TCP `7844` as the fallback. If both are blocked, the tunnel cannot connect. If your firewall uses TLS inspection or SNI-based filtering, allow the SNI hostname pattern `*.cftunnel.com`. Confirm NAT Gateway egress and DNS resolution from a worker node; the tunnel client retries automatically once egress is restored.

</details>

<details>

<summary>Reconciliation stalls (cluster unreachable)</summary>

The EKS API is unreachable, the IAM access entry is missing, or `aws-auth` is mis-bound. Verify the kubeconfig's IAM identity is mapped to a Kubernetes RBAC scope on the cluster.

</details>

<details>

<summary>Install step fails</summary>

The Prime `artifactIdentifier` is wrong or unavailable for this release. Confirm the artifact identifier with Modern and that the Data Plane can reach Prime.

</details>

<details>

<summary>Ingress endpoints do not respond (ingress mode)</summary>

The DNS names do not match `fqdn` and `tcpFqdn`, or the subnet and Elastic IP counts differ. Confirm the DNS records point to the values you set, and that `subnets`, `eipIds`, and `tcpEipIds` all have the same number of entries.

</details>

<details>

<summary>Lint reports a conflicting field</summary>

The manifest mixes tunnel and ingress fields. For ingress, remove the `configs` block and make sure `inputs.ingress` is present. For tunnel, remove `inputs.ingress`. Also confirm `kernelRequest.region` is not set.

</details>

## Best practices

* Dedicate the `core-kernel` node pool to platform components. Do not co-schedule user Data Product workloads on it.
* Monitor AWS quotas and EKS node-group capacity ahead of production Data Product workload growth.
* Capture provisioning logs from `dataos-ctl` and AWS CloudWatch during the apply for assertion and troubleshooting.
* Store the manifest encrypted in version control; decrypt it only at apply time.
* Apply exactly one manifest per Data Plane: tunnel or ingress. Do not leave `configs` on an ingress manifest, and do not omit `inputs.ingress` on one.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/operate/dataos-operator/provision-data-plane/aws.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
