> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/projections.md). # Projections Use this guide to understand and build projections in DataOS for runtime dynamic configurations. ## What are projections? Projections are a DataOS templating system that injects configuration files, environment variables, and command arguments into your containers at runtime. Instead of hardcoding values such as database URLs, API keys, Spark settings, configuration files, or command arguments directly inside a resource, you can define them as templates. At runtime, DataOS fills those templates using values from secrets, depots, addresses, default DataOS context, and application configuration. ### What Can Projections Do? * Generate configuration files dynamically * Inject environment variables from secrets * Modify command arguments * Access data from depots (data sources) * Adapt to different environments automatically ## When should you use projections? Use projections when your workload needs configuration that changes across environments, tenants, depots, secrets, or runtime contexts. In simple terms: ``` Template + Runtime context = Container-ready configuration ``` Use projections when you need to: * Inject secret values into a container without hardcoding them. * Generate config files at runtime. * Build environment-specific settings from the DataOS context. * Use depot connection details inside an application or stack. * Pass dynamic command arguments to a container. * Keep the same stack or resource reusable across different tenants or environments. #### The Problem Projections Solve Without projections, you’d need to hardcode configuration: ```yaml # ❌ Static, environment-specific configuration spec: environmentVars: DATABASE_URL: "postgresql://user:pass@prod-db:5432/mydb" API_KEY: "sk-1234567890abcdef" TENANT_ID: "production-tenant" ``` **Problems with static configuration:** * **Security risk**: Credentials exposed in YAML * **Environment coupling**: Different configs for dev/staging/prod * **Maintenance overhead**: Manual updates for each environment * **No dynamic adaptation**: Can’t adapt to runtime context #### The Projection Solution With projections, configuration becomes dynamic and secure: ```yaml # Dynamic, environment-agnostic configuration spec: use: projection: secrets: - id: "database-credentials" contextAlias: "db" projections: envVars: - key: "DATABASE_URL" template: "postgresql://{{ secrets['db'].username | base64_decode }}:{{ secrets['db'].password | base64_decode }}@{{ secrets['db'].host | base64_decode }}:5432/{{ secrets['db'].database | base64_decode }}" - key: "TENANT_ID" template: "{{ defaultProjections.dataOsTenantId }}" ``` **Benefits of projections:** * **Secure**: Credentials come from DataOS secrets * **Environment-agnostic**: Same config works across environments * **Dynamic**: Adapts to runtime context automatically * **Maintainable**: Single source of truth ### Template Engine Support Every projection item carries its content in the `template` field. There is no `value` field: an item without `template` fails validation with `envVar template is required`. Set the optional `templateType` field to choose how `template` is processed. DataOS supports two values: {% tabs %} {% tab title="Liquid (default)" %} Liquid is the default. Reference context values without a leading dot. ```yaml template: "{{ secrets['db'].url | base64_decode }}" template: "{{ defaultProjections.dataOsTenantId }}" ``` Use Liquid for secrets, runtime context, filters such as `base64_decode` and `default`, and conditionals. {% endtab %} {% tab title="None" %} Use `none` when the value should be treated as static text. ```yaml templateType: "none" template: "production-mode" ``` {% endtab %} {% endtabs %} {% hint style="warning" %} Go Template syntax with a leading dot, such as `{{.secrets.db.url}}`, is not supported. Use the Liquid syntax shown above. {% endhint %} ## Projection building blocks A projection has three main parts: ``` 1. Template engine Defines how variables are resolved. 2. Data sources Defines where runtime values come from. 3. Generated outputs Defines what gets created for the container. ``` Example: ```yaml projection: templateType: "liquid" secrets: - id: "postgres-credentials" contextAlias: "db" projections: envVars: - key: "DATABASE_URL" template: "postgresql://{{ secrets['db'].username | base64_decode }}:{{ secrets['db'].password | base64_decode }}@{{ secrets['db'].host | base64_decode }}:{{ secrets['db'].port | base64_decode }}/{{ secrets['db'].database | base64_decode }}" ``` In this example: * `templateType` tells DataOS to use Liquid syntax. Liquid is the default, so you can omit this line. * `secrets` tells DataOS which secret to load. * `contextAlias: "db"` makes the secret available as `secrets['db']`. * `envVars` generates an environment variable for the container. `key` sets the variable name and `template` sets its content. * `DATABASE_URL` is built dynamically at runtime. ## Developer workflow ### 1. Decide what needs to be dynamic Start by identifying which values should not be hardcoded. Common examples: * Database username and password * API tokens * Tenant-specific URLs * Spark configuration * Depot connection details * Environment-specific log levels * Config files used by an application ### 2. Choose where to define the projection Projections can be defined at two levels: **Stack-level projection:** The projection applies to every job, Service, and Worker that uses the stack. ```yaml # In your stack definition projection: templateType: "liquid" # Default: liquid secrets: [...] depots: [...] projections: envVars: [...] files: [...] ``` **Application-level projection:** The projection is specific to one workload. ```yaml # In your application spec spec: use: projection: templateType: "liquid" # Default: liquid secrets: [...] projections: envVars: [...] files: [...] ``` If both levels define a projection, both run. When both produce an environment variable with the same name, the application-level value wins. {% hint style="warning" %} **Workflows:** Put `use.projection` under each DAG step (`spec.dag[].spec.use.projection`), not under the Workflow's top-level `spec`. The Workflow top-level `spec` has no `use` field, so a projection placed there is silently dropped: validation can pass, but the environment variables are missing at runtime. Each step needs its own projection unless the stack's projection already covers it. See [Workflow core concepts](/references/resources/workflow/core-concepts.md#projections-and-storage). {% endhint %} ### 3. Add data sources Add the runtime data your template needs. Common sources include: * Secrets * Depots * Addresses * Default DataOS context * Application context * Stack spec values ### 4. Define what the projection generates A projection can generate one or more of the following: #### a. Environment Variables **Individual Variables** Use `key` (or `keyTemplate`) for the variable name and `template` for its content. ```yaml projections: envVars: - key: "DATABASE_URL" template: "{{ secrets['db'].url | base64_decode }}" - key: "API_KEY" template: "{{ secrets['api'].key | base64_decode }}" - key: "INSTANCE_ID" template: "{{ defaultProjections.dataOsInstanceTenantId }}" ``` #### b. Configuration Files **Individual Files** ```yaml projections: files: - name: "database.conf" directory: "/etc/app/config" # Optional: defaults to /etc/dataos/secret template: | [database] url = {{ secrets['db'].url | base64_decode }} username = {{ secrets['db'].username | base64_decode }} password = {{ secrets['db'].password | base64_decode }} - name: "spark-defaults.conf" directory: "/opt/spark/conf" template: | spark.sql.adaptive.enabled={{ stackSpec.adaptiveQueryExecution | default: "true" }} spark.sql.adaptive.coalescePartitions.enabled=true ``` #### c. Command Arguments ```yaml projections: args: - order: 1 template: "--config=/etc/app/config.yaml" - order: 2 template: "--env={{ defaultProjections.dataOsInstanceTenantId }}" - order: 3 template: "--database-url={{ secrets['db'].url | base64_decode }}" replaceArgs: false # true = replace all args, false = append ``` #### d. Dictionary Values ```yaml projections: dictionary: - key: "tenant" template: "{{ defaultProjections.dataOsTenantId }}" - key: "namespace" template: "{{ defaultProjections.namespace }}" ``` ### 5. Test locally The DataOS CLI provides two commands for developing and testing projections locally. Use these to prototype and test: **`dataos develop use` - Test Projection Logic** **`dataos develop resource` - Test Full Resource Definitions** ### 6. Deploy after validation Apply your tested projections to actual resources. ## Quick start with DataOS CLI Before adding projections to a real workload, test them locally. ### Test projection logic Use `dataos develop use` when you want to test only the projection template. ```bash # Test a projection with local template files dataos develop use \ --template-file ./my-projection.yaml \ --binding-context ./context.json \ --secrets-file ./secrets.json \ --depots-file ./depots.json # Test with inline context dataos develop use \ --template-string "DATABASE_URL: {{ secrets['db'].url }}" \ --binding-context '{"secrets":{"db":{"url":"localhost:5432"}}}' ``` ### Test a full resource Use `dataos develop resource` when you want to validate a complete resource definition that includes projections. ```bash # Test a complete workflow with projections dataos develop resource \ --resource-file ./my-workflow.yaml \ --context-file ./test-context.json \ --dry-run # Preview generated Kubernetes manifests dataos develop resource \ --resource-file ./my-service.yaml \ --show-generated-manifests ``` ### Recommended development loop ``` Write a projection template ↓ Test projection output locally ↓ Validate full resource definition ↓ Preview generated output ↓ Deploy a tested resource that includes projection ``` ## Data sources ### Secrets Use secrets for sensitive values such as passwords, tokens, usernames, and API keys. ```yaml projection: secrets: - id: "my-database-secret" contextAlias: "db" # Optional: use as {{ secrets['db'].key }} - id: "api-credentials" contextAlias: "api" ``` With the alias `db`, the secret can be referenced as: ```yaml {{ secrets['db'].key | base64_decode }} ``` ### Depots Use depots when the projection needs information about a data source. ```yaml projection: depots: - name: "my-postgres-depot" contextAlias: "postgres" includeSecrets: true # Include depot secrets secretPurposes: ["r"] # Only read secrets ``` Use `includeSecrets: true` only when the projected output needs depot secret values. ### Addresses Use addresses when the projection needs to reference DataOS dataset or depot addresses. ```yaml projection: addresses: - "dataos://icebase?purpose=query" ``` ## Available template variables ### Default DataOS context Use `defaultProjections` when you need platform-provided values such as tenant IDs, namespace, resource details, service URLs, cloud provider details, Spark event log settings, or standard DataOS paths. ```yaml # Core DataOS Context {{ defaultProjections.dataOsInstanceTenantId }} # Instance tenant ID {{ defaultProjections.dataOsTenantId }} # Current tenant ID {{ defaultProjections.namespace }} # Kubernetes namespace {{ defaultProjections.dataOsName }} # Resource name {{ defaultProjections.dataOsType }} # Resource type (workflow, service, worker) {{ defaultProjections.dataOsRunId }} # Unique run ID {{ defaultProjections.owner }} # Resource owner # DataOS Service URLs {{ defaultProjections.dataOsFqdn }} # DataOS FQDN {{ defaultProjections.heimdallUrl }} # Heimdall Service {{ defaultProjections.metisUrl }} # Metis Service {{ defaultProjections.depotServiceUrl }} # Depot Service {{ defaultProjections.gatewayUrl }} # Gateway URL # Cloud Detection {{ defaultProjections.isAzureCloud }} # true/false for Azure {{ defaultProjections.isGcpCloud }} # true/false for GCP {{ defaultProjections.isAwsCloud }} # true/false for AWS # Spark Configuration (if enabled) {{ defaultProjections.sparkEventLogEnabled }} # Spark event logging enabled {{ defaultProjections.sparkEventLogDir }} # Event log directory {{ defaultProjections.sparkEventLogBucket }} # S3 bucket for logs # Directory Paths {{ defaultProjections.dataOsSecretDir }} # /etc/dataos/secret {{ defaultProjections.dataOsConfigDir }} # /etc/dataos/config {{ defaultProjections.runnableArtifactDir }} # /etc/dataos/work ``` ### Application context ```yaml {{ envs.MY_VAR }} # User-defined environment variables {{ stackSpec.driver.memory }} # Stack specification values {{ serviceName }} # Service name (if applicable) {{ namespace }} # Kubernetes namespace {{ certs.truststorePath }} # Certificate context (if generateCert=true) ``` ### Secrets context ```yaml {{ secrets['myAlias'].key }} # Secret values by alias (recommended) {{ secrets['secret-id'].key }} # Secret values by ID {{ projectionSecrets }} # Array of Secret objects ``` Secret values are base64-encoded in the projection context. Add `| base64_decode` to get the plain value, for example `{{ secrets['myAlias'].key | base64_decode }}`. ### Depots context ```yaml {{ depots.myAlias }} # Depot object by alias (recommended) {{ depots['depot-name'] }} # Depot object by name {{ projectionDepots }} # Array of Depot objects {{ depotsPurposesSecrets }} # Map of depot->purpose->secrets ``` ## Complete examples ### Example 1: Database configuration This example builds a database URL and a database config file from a secret. Use this pattern when your application needs database connection settings but the credentials should stay in DataOS secrets. ```yaml projection: secrets: - id: "postgres-credentials" contextAlias: "db" projections: envVars: - key: "DATABASE_URL" template: "postgresql://{{ secrets['db'].username | base64_decode }}:{{ secrets['db'].password | base64_decode }}@{{ secrets['db'].host | base64_decode }}:{{ secrets['db'].port | base64_decode }}/{{ secrets['db'].database | base64_decode }}" files: - name: "database.conf" directory: "/etc/app" template: | [database] host = {{ secrets['db'].host | base64_decode }} port = {{ secrets['db'].port | base64_decode }} database = {{ secrets['db'].database | base64_decode }} username = {{ secrets['db'].username | base64_decode }} password = {{ secrets['db'].password | base64_decode }} ssl_mode = require ``` ### Example 2: Multi-cloud Spark configuration This example generates Spark configuration based on the cloud environment and Spark event log settings. Use this pattern when the same workload needs to run across AWS, Azure, or GCP without maintaining separate config files. ```yaml projection: secrets: - id: "spark-history-server" contextAlias: "spark" projections: files: - name: "spark-defaults.conf" directory: "/opt/spark/conf" template: | # Event logging configuration {% if defaultProjections.sparkEventLogEnabled %} spark.eventLog.enabled=true spark.eventLog.dir={{ defaultProjections.sparkEventLogDir }} {% if defaultProjections.isAzureCloud %} # Azure Blob Storage configuration spark.hadoop.fs.azure.account.key.{{ defaultProjections.sparkEventLogStorageAccountName }}.{{ defaultProjections.sparkEventLogStorageEndpointSuffix }}={{ defaultProjections.sparkEventLogStorageAccountKey }} {% elsif defaultProjections.isAwsCloud %} # AWS S3 configuration spark.hadoop.fs.s3a.bucket.{{ defaultProjections.sparkEventLogBucket }}.access.key={{ defaultProjections.sparkEventLogAccessKey }} spark.hadoop.fs.s3a.bucket.{{ defaultProjections.sparkEventLogBucket }}.secret.key={{ defaultProjections.sparkEventLogAccessSecret }} spark.hadoop.fs.s3a.bucket.{{ defaultProjections.sparkEventLogBucket }}.region={{ defaultProjections.sparkEventLogRegion }} {% elsif defaultProjections.isGcpCloud %} # GCP configuration spark.hadoop.fs.s3a.bucket.{{ defaultProjections.sparkEventLogBucket }}.endpoint={{ defaultProjections.sparkEventLogEndpoint }} {% endif %} {% else %} spark.eventLog.enabled=false {% endif %} # DataOS integration spark.metrics.namespace=flare spark.driverEnv.DATAOS_FQDN={{ defaultProjections.dataOsFqdn }} spark.driverEnv.HEIMDALL_URL={{ defaultProjections.heimdallUrl }} ``` ### Example 3: DataOS-specific patterns #### Icebase connection This pattern references an Icebase address and uses depot connection details in the generated output. ```yaml projection: addresses: - "dataos://icebase:retail/customer?purpose=r" projections: envVars: - key: "ICEBASE_ENDPOINT" template: "{{ depots.icebase.spec.connectionSecret.endpoint }}" ``` #### Depot integration This pattern loads depot metadata and secrets, then writes them into a generated config file. ```yaml projection: depots: - name: "my-fastbase" contextAlias: "fastbase" includeSecrets: true secretPurposes: ["rw"] projections: files: - name: "fastbase-config.yaml" directory: "/etc/dataos/config" template: | fastbase: endpoint: {{ depots.fastbase.spec.endpoint }} database: {{ depots.fastbase.spec.database }} ``` ## How it works under the hood See [How projections work](/references/resources/projections/how-projections-work.md) for the runtime flow, generated output, and default mount paths. ### Additional guidance Use these companion pages when you move from setup to production use: * [Best practices](/references/resources/projections/best-practices.md): design secure, reusable, and maintainable projections. * [Troubleshooting](/references/resources/projections/troubleshooting.md): diagnose template, context, and file-mount issues. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://v2.dataos.info/references/resources/projections.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.