> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/operate/phase-1-provision-data-plane/aws.md).

# AWS

Deploy a **DataOS Data Plane** onto an **existing Amazon EKS cluster** in your AWS account. Your SRE team provisions and operates the AWS infrastructure using Modern-published requirements. DataOS installs the platform components on top through a dataplane YAML applied with `dataos-ctl`. The Data Plane connects to the DataOS **Control Plane (Instance)** in **Modern Data Cloud** over an outbound **tunnel**.

***

## Prerequisites

### Instance

Use a working DataOS **Instance FQDN**.

For example: [**`pacific-051426.dataos.cloud`**](https://pacific-051426.dataos.cloud/gateus)

### DataOS CLI

Install the DataOS CLI and set the Instance domain in your CLI context.

{% hint style="info" %}
Install the CLI from the **Setup CLI** section on the home page after you sign in to the Instance.
{% endhint %}

<details>

<summary><strong>Initialize the CLI context</strong></summary>

Open the Instance FQDN and sign in.

Point `dataos-ctl` at your DataOS Instance. Run:

```bash
dataos-ctl init
```

The CLI walks you through a short interactive prompt. When it asks for the **Tenant Identifier**, enter the default tenant `system` (not your application tenant): provisioning runs against `system`:

```bash
INFO[0000] The DataOS® is not initialized, do you want to proceed with initialization? (Y,n)
->Y

INFO[0005] Please enter a name for the current DataOS® Context?
-><instance-name>

INFO[0012] Please enter the Fully Qualified Domain Name of the DataOS® instance?
->your-org.dataos.app

INFO[0037] Please enter the Tenant Identifier to use for the DataOS® instance?
->system

INFO[0041] entered DataOS®: <instance-name> : your-org.dataos.app : system
INFO[0041] 🚀 initialization...complete
```

Then log in:

```bash
dataos-ctl login
```

{% hint style="info" %}
Enter the FQDN without a protocol prefix: use `your-org.dataos.app`, not `https://your-org.dataos.app`. The Instance domain comes from your Modern Data representative.
{% endhint %}

</details>

### Version alignment

The **Control Plane** and **Data Plane** versions must be compatible. Use the `imageTag` value from the Modern release notes in `dataOsManagerConfigs`.

### Operator permissions

You need **`roles:id:operator`** to apply the Data Plane manifest configuration.

### AWS credentials and kubeconfig access

**Credentials**

* **AWS\_ACCESS\_KEY\_ID**
* **AWS\_SECRET\_ACCESS\_KEY**

**Kubeconfig access**

Before deployment:

* Keep a valid kubeconfig file for the provisioned EKS cluster.
* Verify `kubectl` works against the cluster with that file.
* Base64-encode the kubeconfig for `kubeConfigBase64` in the YAML and `KUBECONFIG_BASE64` in your shell.

```bash
export KUBECONFIG_BASE64=$(base64 -w 0 < kubeconfig.yaml)   # Linux
# macOS: base64 -i kubeconfig.yaml | tr -d '\n'
```

***

Work through the steps in order. Each step prepares one input the final `dataos-ctl domain apply` needs: a CLI context pointed at your Instance, a base64 kubeconfig for the EKS cluster, the environment variables the YAML resolves, and the dataplane YAML itself.

### Deploy the Data Plane

{% stepper %}
{% step %}

#### Base64-encode the kubeconfig

{% hint style="info" %}
If the file is already base64-encoded, you can skip this step.
{% endhint %}

Obtain the **kubeconfig file** for your EKS cluster from the Cluster handover, confirm `kubectl` reaches the cluster, then base64-encode it for the YAML and your shell:

```bash
kubectl get nodes                                          # confirm access first
```

{% tabs %}
{% tab title="Linux" %}

```bash
export KUBECONFIG_BASE64=$(base64 -w 0 < kubeconfig.yaml)  # Linux
```

{% endtab %}

{% tab title="macOS" %}

```bash
export KUBECONFIG_BASE64=$(base64 -i kubeconfig.yaml | tr -d '\n')
```

{% endtab %}

{% tab title="Windows" %}

{% endtab %}
{% endtabs %}

This value gives the Control Plane access to the EKS API. Treat it as a secret.
{% endstep %}

{% step %}

#### Export the environment variables

`dataos-ctl domain apply` resolves `${...}` references in the YAML from your shell. Export these before applying:

```bash
export DATAPLANE_ID=<instance-name>-dp-01
export INSTANCE_TENANT_ID=<instance-name>
export KUBECONFIG_BASE64=<base64-encoded-kubeconfig>   # from the previous step
export BASE64_AWS_ACCESS_KEY_ID=<base64-encoded-key-id>
export BASE64_AWS_SECRET_ACCESS_KEY=<base64-encoded-secret>
```

| Variable                                                   | Purpose                                                   | Example          |
| ---------------------------------------------------------- | --------------------------------------------------------- | ---------------- |
| `DATAPLANE_ID`                                             | Stable Data Plane identifier                              | `acme-prd-dp-01` |
| `INSTANCE_TENANT_ID`                                       | DataOS Instance this Data Plane attaches to               | `pacific-ocean`  |
| `KUBECONFIG_BASE64`                                        | Base64-encoded kubeconfig from the previous step          | :                |
| `BASE64_AWS_ACCESS_KEY_ID`, `BASE64_AWS_SECRET_ACCESS_KEY` | Base64-encoded AWS credentials for ongoing AWS operations | :                |
| {% endstep %}                                              |                                                           |                  |

{% step %}

#### Prepare the dataplane YAML

Author the dataplane YAML using the sample below. As you fill it in, a few attributes decide how the Data Plane installs and connects: get these right and the full breakdown in Dataplane configuration covers the rest:

{% code title="aws-byoc-dataplane.yaml" %}

```yaml
name: ${DATAPLANE_ID}
version: v1alpha
entity: domain
type: dataplane
description: the ${DATAPLANE_ID} dataplane for ${INSTANCE_TENANT_ID} instance
v1alpha:
  dataplane:
    name: ${DATAPLANE_ID}
    tenantDomainNames:
      - system-entities
    networkType: tunnel
    workpiece:
      name: ${DATAPLANE_ID}-${INSTANCE_TENANT_ID}.dataplane
      zone: dataos.cloud
      kernelRequest:
        template: dataplane-kernel-system-existing-compute-tunnel-v1
        cloud: aws
        region: ap-south-1
        inputs:
          dataOsManagerConfigs:
            imageTag: latest
            logLevel: info
            replicas: 1
            nodeSelector:
              dataos.io/purpose: core-kernel
          kubeConfigBase64: ${KUBECONFIG_BASE64}
        target: engineering
        directInputs:
          aws_access_key_id: ${BASE64_AWS_ACCESS_KEY_ID}
          aws_secret_access_key: ${BASE64_AWS_SECRET_ACCESS_KEY}
      kubeAuthConfigs:
        - key: AWS_ACCESS_KEY_ID
          value: ${BASE64_AWS_ACCESS_KEY_ID}
          base64: true
          envVar: true
        - key: AWS_SECRET_ACCESS_KEY
          value: ${BASE64_AWS_SECRET_ACCESS_KEY}
          base64: true
          envVar: true
    dataPlaneKernelSystemInstall:
      installFromGit:
        installFile: install.yaml
        applicationsFile: install.applications.yaml
        valuesFile: install.values.yaml
        installFileRootDir: installs/dataplane-kernel-system
        gitRepoUrl: https://bitbucket.org/rubik_/dataos-component-install.git
        gitBranch: <GIT_BRANCH>
        onDemandCloudKernel: true
        gitUsername: <GIT_USERNAME>
        gitPassword: <GIT_PASSWORD>
      azureEndpointSuffix: core.windows.net
      awsEndpointSuffix: amazonaws.com
      coreKernelNodeSelector:
        dataos.io/purpose: core-kernel
```

{% endcode %}

{% hint style="warning" %}
The kubeconfig and AWS credentials inside the YAML are sensitive. Store the dataplane YAML encrypted and limit who can decrypt it.
{% endhint %}
{% endstep %}

{% step %}

#### Apply the Dataplane manifest configuration

With the context set and the variables exported, apply the YAML:

```bash
DATAPLANE_ID=$DATAPLANE_ID INSTANCE_TENANT_ID=$INSTANCE_TENANT_ID KUBECONFIG_BASE64=$KUBECONFIG_BASE64 dataos-ctl domain apply -f aws-byoc-dataplane.yaml
```

The Control Plane validates the request and starts installing the platform onto your cluster.
{% endstep %}
{% endstepper %}

<details>

<summary>What happens after you apply</summary>

When you run the apply, the Control Plane reads the `kubeConfigBase64` you supplied and uses it to reach the EKS API for the first time. It installs `dataos-manager` onto the cluster, which pulls the install manifests defined in `installFromGit` and lays down the DataOS platform components.

Because you set `networkType: tunnel`, the Data Plane does not wait to be reached from the outside. A tunnel client inside the cluster opens an outbound, encrypted connection from a worker node out through the NAT Gateway to Modern Data Cloud. No inbound ports on your cluster are exposed. Once the tunnel is established, the Control Plane registers the Data Plane and manages it over that connection.

The kubeconfig is used only for this Control-Plane-to-cluster management path. It is not part of your data traffic. Every hop is TLS-encrypted end to end.

</details>

***

### Manage the Data Plane

Once the Data Plane is live, run these routine operations against it.

#### Check status

```bash
dataos-ctl domain get 
#or
dataos-ctl domain get -a #if created by others
```

{% hint style="info" %}
If the command above shows <mark style="color:$success;">`workpiece-forge-kernel: success`</mark>, the infrastructure setup is complete.
{% endhint %}

Then access the Data Plane cluster with `kubectl` and inspect the `dataos-manager` logs.

```bash
kubectl logs -n <instance_name>-0-dsm dataos-manager-0 -f
```

#### Describe the applied configuration

```bash
dataos-ctl domain get -t dataplane -n <instance-name>-dp-01 -d
```

#### Upgrade the Data Plane

Coordinate `imageTag`, `template`, or sizing changes with your DataOS representative before applying in production.

```bash
dataos-ctl domain task run \
  -t dataplane \
  -v v1alpha \
  -n <instance-name>-dp-01 \
  --taskType manager-install
```

If anything looks unhealthy, inspect the `dataos-manager` logs:

```bash
kubectl logs -n <instance_name>-0-dsm dataos-manager-0 -f
```

***

### Dataplane configuration

A field-by-field reference for the dataplane manifest configuration.

#### Metadata

Top-level fields that identify the Data Plane resource.

| Attribute     | Description                                                       |
| ------------- | ----------------------------------------------------------------- |
| `name`        | Keep it stable and readable: it becomes the dataplane identifier. |
| `version`     | API version for the resource spec.                                |
| `entity`      | Always `domain`.                                                  |
| `type`        | Always `dataplane`: declares what you are provisioning.           |
| `description` | Make it environment- and Tenant-specific for traceability.        |

```yaml
name: ${DATAPLANE_ID}
version: v1alpha
entity: domain
type: dataplane
description: the ${DATAPLANE_ID} dataplane for ${INSTANCE_TENANT_ID} instance
```

{% hint style="info" %}
Use a naming convention that clearly encodes the Instance and cloud. Example: `<instance-name>-<cloud>-dp-01`.
{% endhint %}

#### Dataplane

Core Data Plane identity and Control Plane connectivity.

| Attribute           | Description                                                                                             |
| ------------------- | ------------------------------------------------------------------------------------------------------- |
| `dataplane.name`    | Often includes a date suffix to ensure uniqueness.                                                      |
| `tenantDomainNames` | Default domain mappings (platform-specific).                                                            |
| `networkType`       | Decides how the Data Plane reaches the Control Plane. `tunnel` uses outbound-only Cloudflare tunneling. |

```yaml
v1alpha:
  dataplane:
    name: ${DATAPLANE_ID}
    tenantDomainNames:
      - system-entities
    networkType: tunnel
```

#### Workpiece

Represents the execution unit for the provisioning pipeline.

| Attribute        | Description                                                                  |
| ---------------- | ---------------------------------------------------------------------------- |
| `workpiece.name` | Composite identifier combining the Data Plane ID and the Instance Tenant ID. |
| `workpiece.zone` | Typically fixed for DataOS-managed provisioning.                             |

```yaml
workpiece:
  name: ${DATAPLANE_ID}-${INSTANCE_TENANT_ID}.dataplane
  zone: dataos.cloud
```

#### Kernel Request

Selects the kernel provisioning blueprint and target cloud.

| Attribute  | Description                                                                                                                                                                  |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template` | Defines which kernel provisioning blueprint is used. `dataplane-kernel-system-existing-compute-tunnel-v1` installs the platform onto an existing EKS cluster in tunnel mode. |
| `cloud`    | Selects cloud-specific behaviour. Use `aws`.                                                                                                                                 |
| `region`   | AWS region where the existing EKS cluster and Data Plane resources reside. Set it to your cluster's region (for example, `ap-south-1`).                                      |

```yaml
kernelRequest:
  template: dataplane-kernel-system-existing-compute-tunnel-v1
  cloud: aws
  region: ap-south-1
```

#### Inputs (DataOS Manager + kubeconfig)

| Attribute                           | Description                                                                              |
| ----------------------------------- | ---------------------------------------------------------------------------------------- |
| `dataOsManagerConfigs.imageTag`     | `dataos-manager` image version. Refer to the release notes.                              |
| `dataOsManagerConfigs.logLevel`     | Prefer `info` for production, `debug` temporarily during incidents.                      |
| `dataOsManagerConfigs.replicas`     | Keep as `1`; a single replica is sufficient.                                             |
| `dataOsManagerConfigs.nodeSelector` | Schedules `dataos-manager` onto the node pool labelled `dataos.io/purpose: core-kernel`. |
| `kubeConfigBase64`                  | Base64-encoded kubeconfig that gives the Control Plane access to the EKS cluster.        |

```yaml
inputs:
  dataOsManagerConfigs:
    imageTag: latest
    logLevel: info
    replicas: 1
    nodeSelector:
      dataos.io/purpose: core-kernel
  kubeConfigBase64: ${KUBECONFIG_BASE64}
```

#### Config Bundle

| Attribute | Description                                 |
| --------- | ------------------------------------------- |
| `target`  | Platform-managed configuration bundle name. |

```yaml
target: engineering
```

#### Direct Inputs (AWS credentials)

Base64-encoded AWS credentials passed directly to the kernel provisioning inputs so the platform can reach the AWS APIs during provisioning.

| Attribute               | Description                                                                   |
| ----------------------- | ----------------------------------------------------------------------------- |
| `aws_access_key_id`     | Base64-encoded AWS access key ID, from `${BASE64_AWS_ACCESS_KEY_ID}`.         |
| `aws_secret_access_key` | Base64-encoded AWS secret access key, from `${BASE64_AWS_SECRET_ACCESS_KEY}`. |

```yaml
directInputs:
  aws_access_key_id: ${BASE64_AWS_ACCESS_KEY_ID}
  aws_secret_access_key: ${BASE64_AWS_SECRET_ACCESS_KEY}
```

#### Auth (AWS credentials)

AWS credentials used by the platform for ongoing AWS API operations.

| Attribute | Description                                                               |
| --------- | ------------------------------------------------------------------------- |
| `key`     | Environment variable name.                                                |
| `value`   | Credential value (base64-encoded), from the corresponding shell variable. |
| `base64`  | Set to `true` to indicate the value is base64-encoded.                    |
| `envVar`  | Set to `true` to inject the value as an environment variable.             |

```yaml
kubeAuthConfigs:
  - key: AWS_ACCESS_KEY_ID
    value: ${BASE64_AWS_ACCESS_KEY_ID}
    base64: true
    envVar: true
  - key: AWS_SECRET_ACCESS_KEY
    value: ${BASE64_AWS_SECRET_ACCESS_KEY}
    base64: true
    envVar: true
```

#### Installation (Git-based)

Defines which installation manifests are applied for Data Plane components and the source repository for installation artifacts.

| Attribute                                       | Description                                                                                                                                   |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `installFile`, `applicationsFile`, `valuesFile` | Install manifests and Helm-style values.                                                                                                      |
| `installFileRootDir`                            | Root directory within the Git repository for install artifacts.                                                                               |
| `gitRepoUrl`                                    | Source repository URL.                                                                                                                        |
| `gitBranch`                                     | Pin to a specific branch for production stability.                                                                                            |
| `gitUsername`, `gitPassword`                    | Git authentication credentials.                                                                                                               |
| `onDemandCloudKernel`                           | When infrastructure provisioning generates a new output, `dataos-manager` creates a new Cloud Kernel JSON. **This value must remain `true`.** |

```yaml
installFromGit:
  installFile: install.yaml
  applicationsFile: install.applications.yaml
  valuesFile: install.values.yaml
  installFileRootDir: installs/dataplane-kernel-system
  gitRepoUrl: https://bitbucket.org/rubik_/dataos-component-install.git
  gitBranch: <GIT_BRANCH>
  onDemandCloudKernel: true
  gitUsername: <GIT_USERNAME>
  gitPassword: <GIT_PASSWORD>
```

#### Cloud Endpoints

Used for cloud integrations and endpoint resolution.

```yaml
azureEndpointSuffix: core.windows.net
awsEndpointSuffix: amazonaws.com
```

#### Node Selectors

Ensures platform components schedule onto the intended node pool.

| Attribute                | Description                                   |
| ------------------------ | --------------------------------------------- |
| `coreKernelNodeSelector` | Label selector for the core kernel node pool. |

```yaml
coreKernelNodeSelector:
  dataos.io/purpose: core-kernel
```

{% hint style="info" %}
**Do not change this:** the DataOS installer applies the same node label to node pools.
{% endhint %}

***

### Troubleshooting

<details>

<summary>Apply fails with an auth error</summary>

Invalid or expired kubeconfig, or insufficient RBAC. Regenerate the kubeconfig with the required RBAC, re-encode it, and re-apply.

</details>

<details>

<summary>Pods stuck <code>Pending</code></summary>

The `dataos.io/purpose: core-kernel` label is missing, or the node pool lacks capacity. Label the nodes and scale the node pool.

</details>

<details>

<summary>Image pull failures</summary>

Egress to the registry is blocked, or image pull secrets are missing for a private mirror. Allow TCP 443 to the registry and configure image pull secrets.

</details>

<details>

<summary>Tunnel does not establish</summary>

Egress to `*.cloudflare.com:443` or `cfargotunnel.com:443` is blocked, or DNS resolution is failing. Confirm NAT egress and DNS resolution from a worker node. The tunnel client retries automatically once egress is restored.

</details>

<details>

<summary>Reconciliation stalls (cluster unreachable)</summary>

The EKS API is unreachable, the IAM access entry is missing, or `aws-auth` is mis-bound. Verify the kubeconfig's IAM identity is mapped to a Kubernetes RBAC scope on the cluster.

</details>

<details>

<summary>Install step fails</summary>

Git credentials or `installFromGit` paths are invalid. Validate Git access from a worker node.

</details>

***

### Best practices

* Dedicate the `core-kernel` node pool to platform components. Do not co-schedule user workloads on it.
* Monitor AWS quotas and EKS node-group capacity ahead of production workload growth.
* Capture provisioning logs from `dataos-ctl` and AWS CloudWatch during apply and upgrade operations for assertion and troubleshooting.
* Keep the YAML encrypted before pushing it to the Git repository.
* Use decrypted YAML only at apply time.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/operate/phase-1-provision-data-plane/aws.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.