> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/concepts/resources/secret/projections.md).

# Projections

## What are projections?

Projections are a DataOS templating system that injects configuration files, environment variables, and command arguments into your containers at runtime.

Instead of hardcoding values such as database URLs, API keys, Spark settings, configuration files, or command arguments directly inside a resource, you can define them as templates. At runtime, DataOS fills those templates using values from secrets, depots, addresses, default DataOS context, and application configuration.

### What Can Projections Do?

* Generate configuration files dynamically
* Inject environment variables from secrets
* Modify command arguments
* Access data from depots (data sources)
* Adapt to different environments automatically

## When should you use projections?

Use projections when your workload needs configuration that changes across environments, tenants, depots, secrets, or runtime contexts.

In simple terms:

```
Template + Runtime context = Container-ready configuration
```

Use projections when you need to:

* Inject secret values into a container without hardcoding them.
* Generate config files at runtime.
* Build environment-specific settings from the DataOS context.
* Use depot connection details inside an application or stack.
* Pass dynamic command arguments to a container.
* Keep the same stack or resource reusable across different tenants or environments.

#### The Problem Projections Solve

Without projections, you’d need to hardcode configuration:

```yaml
# ❌ Static, environment-specific configuration
spec:
environmentVars:
DATABASE_URL:"postgresql://user:pass@prod-db:5432/mydb"
API_KEY:"sk-1234567890abcdef"
TENANT_ID:"production-tenant"
```

**Problems with static configuration:**

* **Security risk**: Credentials exposed in YAML
* **Environment coupling**: Different configs for dev/staging/prod
* **Maintenance overhead**: Manual updates for each environment
* **No dynamic adaptation**: Can’t adapt to runtime context

#### The Projection Solution

With projections, configuration becomes dynamic and secure:

```yaml
# ✅ Dynamic, environment-agnostic configuration
spec:
use:
projection:
secrets:
-id:"database-credentials"
contextAlias:"db"
projections:
envVars:
-key:"DATABASE_URL"
value:"postgresql://{{.secrets.db.username}}:{{.secrets.db.password}}@{{.secrets.db.host}}:5432/{{.secrets.db.database}}"
-key:"TENANT_ID"
value:"{{.defaultProjections.dataOsTenantId}}"
```

**Benefits of projections:**

* **Secure**: Credentials come from DataOS secrets
* **Environment-agnostic**: Same config works across environments
* **Dynamic**: Adapts to runtime context automatically
* **Maintainable**: Single source of truth

### Template Engine Support

DataOS projections support **three template engines**:

{% tabs %}
{% tab title="Go Template" %}
Go Template is the default for stack-level projections.

```yaml
{{.secrets.db.url}}
{{.defaultProjections.dataOsTenantId}}
```

Use Go Template when you need default DataOS templating behavior, conditionals, loops, and functions.
{% endtab %}

{% tab title="Liquid" %}
Liquid uses a simpler syntax without the leading dot.

```yaml
{{secrets.db.url}}
{{defaultProjections.dataOsTenantId}}
```

Use Liquid when defining individual projection items that benefit from Liquid-style syntax.
{% endtab %}

{% tab title="None" %}
Use `none` when the value should be treated as static text.

```yaml
templateType: "none"
template: "production-mode"
```

{% endtab %}
{% endtabs %}

## Projection building blocks

A projection has three main parts:

```
1. Template engine
   Defines how variables are resolved.

2. Data sources
   Defines where runtime values come from.

3. Generated outputs
   Defines what gets created for the container.
```

Example:

```yaml
projection:
  templateType: "go-template"
  secrets:
    - id: "postgres-credentials"
      contextAlias: "db"
  projections:
    envVars:
      - key: "DATABASE_URL"
        value: "postgresql://{{.secrets.db.username}}:{{.secrets.db.password}}@{{.secrets.db.host}}:{{.secrets.db.port}}/{{.secrets.db.database}}"
```

In this example:

* `templateType` tells DataOS to use Go Template syntax.
* `secrets` tells DataOS which secret to load.
* `contextAlias: "db"` makes the secret available as `.secrets.db`.
* `envVars` generates an environment variable for the container.
* `DATABASE_URL` is built dynamically at runtime.

## Developer workflow

{% stepper %}
{% step %}

### Decide what needs to be dynamic

Start by identifying which values should not be hardcoded.

Common examples:

* Database username and password
* API tokens
* Tenant-specific URLs
* Spark configuration
* Depot connection details
* Environment-specific log levels
* Config files used by an application
  {% endstep %}

{% step %}

### Choose where to define the projection

Projections can be defined at two levels:

**Stack-level projection:** The projection should apply to all resources using the stack.

```yaml
# In your stack definition
projection:
  templateType: "go-template"  # Default: go-template
  secrets: [...]
  depots: [...]
  projections:
    envVars: [...]
    files: [...]
```

**Application-level projection:** The projection is specific to one workload or application.

```yaml
# In your application spec
spec:
  use:
    projection:
      templateType: "go-template"  # Default: go-template
      secrets: [...]
      projections:
        envVars: [...]
        files: [...]
```

{% endstep %}

{% step %}

### Add data sources

Add the runtime data your template needs.

Common sources include:

* Secrets
* Depots
* Addresses
* Default DataOS context
* Application context
* Stack spec values
  {% endstep %}

{% step %}

### Define what the projection generates

A projection can generate one or more of the following:

#### a. Environment Variables

**Individual Variables (Go Template)**

```yaml
projections:
  envVars:
    - key: "DATABASE_URL"
      value: "{{.secrets.db.url}}"
    - key: "API_KEY" 
      value: "{{.secrets.api.key}}"
    - key: "INSTANCE_ID"
      value: "{{.defaultProjections.dataOsInstanceTenantId}}"
```

#### b. Configuration Files

**Individual Files (Go Template)**

```yaml
projections:
  files:
    - name: "database.conf"
      directory: "/etc/app/config"  # Optional: defaults to /etc/dataos/secret
      value: |
        [database]
        url = {{.secrets.db.url}}
        username = {{.secrets.db.username}}
        password = {{.secrets.db.password}}
        
    - name: "spark-defaults.conf"
      directory: "/opt/spark/conf"
      value: |
        spark.sql.adaptive.enabled={{.stackSpec.adaptiveQueryExecution | default "true"}}
        spark.sql.adaptive.coalescePartitions.enabled=true
```

#### c. Command Arguments

```yaml
projections:
  args:
    - order: 1
      value: "--config=/etc/app/config.yaml"
    - order: 2  
      value: "--env={{.defaultProjections.dataOsInstanceTenantId}}"
    - order: 3
      value: "--database-url={{.secrets.db.url}}"
  replaceArgs: false  # true = replace all args, false = append
```

#### d. Dictionary Values

```yaml
projections:
  dictionary:
    - key: "tenant"
      value: "{{.defaultProjections.dataOsTenantId}}"
    - key: "namespace"
      value: "{{.defaultProjections.namespace}}"
```

{% endstep %}

{% step %}

## Test locally

The DataOS CLI provides two commands for developing and testing projections locally. Use these to prototype and test:

**`dataos develop use` - Test Projection Logic**

**`dataos develop resource` - Test Full Resource Definitions**
{% endstep %}

{% step %}

## Deploy after validation

Apply your tested projections to actual resources.
{% endstep %}
{% endstepper %}

## Quick start with DataOS CLI

Before adding projections to a real workload, test them locally.

### Test projection logic

Use `dataos develop use` when you want to test only the projection template.

```bash
# Test a projection with local template files
dataos develop use \
  --template-file ./my-projection.yaml \
  --binding-context ./context.json \
  --secrets-file ./secrets.json \
  --depots-file ./depots.json

# Test with inline context
dataos develop use \
  --template-string "DATABASE_URL: {{.secrets.db.url}}" \
  --binding-context '{"secrets":{"db":{"url":"localhost:5432"}}}'
```

### Test a full resource

Use `dataos develop resource` when you want to validate a complete resource definition that includes projections.

```bash
# Test a complete workflow with projections
dataos develop resource \
  --resource-file ./my-workflow.yaml \
  --context-file ./test-context.json \
  --dry-run

# Preview generated Kubernetes manifests
dataos develop resource \
  --resource-file ./my-service.yaml \
  --show-generated-manifests
```

### Recommended development loop

```
Write a projection template
        ↓
Test projection output locally
        ↓
Validate full resource definition
        ↓
Preview generated output
        ↓
Deploy a tested resource that includes projection
```

## Data sources

### Secrets

Use secrets for sensitive values such as passwords, tokens, usernames, and API keys.

```yaml
projection:
  secrets:
    - id: "my-database-secret"
      contextAlias: "db"        # Optional: use as {{.secrets.db.key}} or {{secrets.db.key}}
    - id: "api-credentials"
      contextAlias: "api"
```

With the alias `db`, the secret can be referenced as:

```yaml
{{.secrets.db.key}}
```

### Depots

Use depots when the projection needs information about a data source.

```yaml
projection:
  depots:
    - name: "my-postgres-depot"
      contextAlias: "postgres"
      includeSecrets: true      # Include depot secrets
      secretPurposes: ["r"]     # Only read secrets
```

Use `includeSecrets: true` only when the projected output needs depot secret values.

### Addresses

Use addresses when the projection needs to reference DataOS dataset or depot addresses.

```yaml
projection:
  addresses:
    - "dataos://icebase?purpose=query"
```

## Available template variables

### Default DataOS context

Use `defaultProjections` when you need platform-provided values such as tenant IDs, namespace, resource details, service URLs, cloud provider details, Spark event log settings, or standard DataOS paths.

```yaml
# Core DataOS Context
{{.defaultProjections.dataOsInstanceTenantId}}    # Instance tenant ID
{{.defaultProjections.dataOsTenantId}}            # Current tenant ID
{{.defaultProjections.namespace}}                 # Kubernetes namespace
{{.defaultProjections.dataOsName}}                # Resource name
{{.defaultProjections.dataOsType}}                # Resource type (workflow, service, worker)
{{.defaultProjections.dataOsRunId}}               # Unique run ID
{{.defaultProjections.owner}}                     # Resource owner

# DataOS Service URLs
{{.defaultProjections.dataOsFqdn}}                # DataOS FQDN
{{.defaultProjections.heimdallUrl}}               # Heimdall Service
{{.defaultProjections.metisUrl}}                  # Metis Service
{{.defaultProjections.depotServiceUrl}}           # Depot Service
{{.defaultProjections.gatewayUrl}}                # Gateway URL

# Cloud Detection
{{.defaultProjections.isAzureCloud}}              # true/false for Azure
{{.defaultProjections.isGcpCloud}}                # true/false for GCP
{{.defaultProjections.isAwsCloud}}               # true/false for AWS

# Spark Configuration (if enabled)
{{.defaultProjections.sparkEventLogEnabled}}      # Spark event logging enabled
{{.defaultProjections.sparkEventLogDir}}          # Event log directory
{{.defaultProjections.sparkEventLogBucket}}       # S3 bucket for logs

# Directory Paths
{{.defaultProjections.dataOsSecretDir}}           # /etc/dataos/secret
{{.defaultProjections.dataOsConfigDir}}           # /etc/dataos/config
{{.defaultProjections.runnableArtifactDir}}       # /etc/dataos/work
```

### Application context

```yaml
{{.envs.MY_VAR}}              # User-defined environment variables
{{.stackSpec.driver.memory}}  # Stack specification values
{{.serviceName}}              # Service name (if applicable)
{{.namespace}}                # Kubernetes namespace
{{.certs.truststorePath}}     # Certificate context (if generateCert=true)
```

### Secrets context

```yaml
{{.secrets.myAlias.key}}      # Secret values by alias (recommended)
{{.secrets.secret-id.key}}    # Secret values by ID
{{.projectionSecrets}}        # Array of Secret objects
```

### Depots context

```yaml
{{.depots.myAlias}}           # Depot object by alias (recommended)
{{.depots.depot-name}}        # Depot object by name
{{.projectionDepots}}         # Array of Depot objects
{{.depotsPurposesSecrets}}    # Map of depot->purpose->secrets
```

## Complete examples

### Example 1: Database configuration with Go Template

This example builds a database URL and a database config file from a secret. Use this pattern when your application needs database connection settings but the credentials should stay in DataOS secrets.

```yaml
projection:
  templateType: "go-template"
  secrets:
    - id: "postgres-credentials"
      contextAlias: "db"
  projections:
    envVars:
      - key: "DATABASE_URL"
        value: "postgresql://{{.secrets.db.username}}:{{.secrets.db.password}}@{{.secrets.db.host}}:{{.secrets.db.port}}/{{.secrets.db.database}}"
    files:
      - name: "database.conf"
        directory: "/etc/app"
        value: |
          [database]
          host = {{.secrets.db.host}}
          port = {{.secrets.db.port}}
          database = {{.secrets.db.database}}
          username = {{.secrets.db.username}}
          password = {{.secrets.db.password}}
          ssl_mode = require
```

### Example 2: Multi-cloud Spark configuration

This example generates Spark configuration based on the cloud environment and Spark event log settings. Use this pattern when the same workload needs to run across AWS, Azure, or GCP without maintaining separate config files.

```yaml
projection:
  templateType: "go-template"
  secrets:
    - id: "spark-history-server"
      contextAlias: "spark"
  projections:
    files:
      - name: "spark-defaults.conf"
        directory: "/opt/spark/conf"
        value: |
          # Event logging configuration
          {{if .defaultProjections.sparkEventLogEnabled}}
          spark.eventLog.enabled=true
          spark.eventLog.dir={{.defaultProjections.sparkEventLogDir}}
          
          {{if .defaultProjections.isAzureCloud}}
          # Azure Blob Storage configuration
          spark.hadoop.fs.azure.account.key.{{.defaultProjections.sparkEventLogStorageAccountName}}.{{.defaultProjections.sparkEventLogStorageEndpointSuffix}}={{.defaultProjections.sparkEventLogStorageAccountKey}}
          {{else if .defaultProjections.isAwsCloud}}
          # AWS S3 configuration
          spark.hadoop.fs.s3a.bucket.{{.defaultProjections.sparkEventLogBucket}}.access.key={{.defaultProjections.sparkEventLogAccessKey}}
          spark.hadoop.fs.s3a.bucket.{{.defaultProjections.sparkEventLogBucket}}.secret.key={{.defaultProjections.sparkEventLogAccessSecret}}
          spark.hadoop.fs.s3a.bucket.{{.defaultProjections.sparkEventLogBucket}}.region={{.defaultProjections.sparkEventLogRegion}}
          {{else if .defaultProjections.isGcpCloud}}
          # GCP configuration
          spark.hadoop.fs.s3a.bucket.{{.defaultProjections.sparkEventLogBucket}}.endpoint={{.defaultProjections.sparkEventLogEndpoint}}
          {{end}}
          {{else}}
          spark.eventLog.enabled=false
          {{end}}
          
          # DataOS integration
          spark.metrics.namespace=flare
          spark.driverEnv.DATAOS_FQDN={{.defaultProjections.dataOsFqdn}}
          spark.driverEnv.HEIMDALL_URL={{.defaultProjections.heimdallUrl}}
```

### Example 3: DataOS-specific patterns

#### Icebase connection

This pattern references an Icebase address and uses depot connection details in the generated output.

```yaml
projection:
  addresses:
    - "dataos://icebase:retail/customer?purpose=r"
  projections:
    envVars:
      - key: "ICEBASE_ENDPOINT"
        value: "{{.depots.icebase.spec.connectionSecret.endpoint}}"
```

#### Depot integration

This pattern loads depot metadata and secrets, then writes them into a generated config file.

```yaml
projection:
  depots:
    - name: "my-fastbase"
      contextAlias: "fastbase"
      includeSecrets: true
      secretPurposes: ["rw"]
  projections:
    files:
      - name: "fastbase-config.yaml"
        directory: "/etc/dataos/config"
        value: |
          fastbase:
            endpoint: {{.depots.fastbase.spec.endpoint}}
            database: {{.depots.fastbase.spec.database}}
```

## How it works under the hood

See [How projections work](/concepts/resources/secret/projections/how-projections-work.md) for the runtime flow, generated output, and default mount paths.

### Additional guidance

Use these companion pages when you move from setup to production use:

* [Best practices](/concepts/resources/secret/projections/best-practices.md): design secure, reusable, and maintainable projections.
* [Troubleshooting](/concepts/resources/secret/projections/troubleshooting.md): diagnose template, context, and file-mount issues.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/concepts/resources/secret/projections.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.