> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/trino/dedicated-trino.md).

# Dedicated cluster

Dedicated Trino means the Trino cluster is provisioned and owned by your data product deployment. The `vulcan-dg-trino` resource creates a Trino coordinator and workers for you and generates Trino catalogs from the depots you declare. You get one self-contained, federated cluster per data product that owns its own compute.

This is the right shape when a data product needs its own isolated cluster, federates across several sources, and materialises into a lakehouse it controls. To share one large central cluster instead, see [Minerva](/references/engine-guide/trino/minerva.md). To connect to a cluster that already exists outside DataOS, see [External Trino](/references/engine-guide/trino/external-trino.md). For the rules common to all three shapes, see the [Trino engine overview](/references/engine-guide/trino.md).

Use the [Connect engine → Dedicated Trino](https://v2.dataos.info/build/productize/connect-engine/trino/dedicated-trino) Build page for the quick steps.

## Core settings

| Setting        | Value                                                            |
| -------------- | ---------------------------------------------------------------- |
| Resource type  | `vulcan-dg-trino`                                                |
| Engine field   | `spec.engine: trino`                                             |
| Model dialect  | `trino`                                                          |
| VDE            | `false` (not supported on Trino)                                 |
| Catalog source | Generated from `spec.depots[]` and `spec.trino.catalog.config[]` |

## Architecture

The domain renders a dedicated Trino data product into cooperating resources: a coordinator service (`<name>-trino`, 1 replica), a workers service (`<name>-trino-workers`, default 2 replicas), a plan workflow (`vulcan migrate` plus `plan --auto-apply`), and a run workflow (scheduled `vulcan run`, depending on plan). The plan and run pods carry a readiness gate that blocks until the coordinator answers `SELECT 1` and the configured worker count is active before Vulcan starts.

{% hint style="info" %}
The stack owns cluster identity. It auto-injects `discovery.uri` and `internal-communication.shared-secret`. The plan pod waits for the coordinator and workers to be active before it starts, so a depot name mismatch in `spec.depots[]` passes `apply` but fails at every run.
{% endhint %}

## Platform prerequisites

Request these from your DataOS SRE team before you write model code:

| Requirement                              | Notes                                                                                            |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------ |
| Trino-capable compute pool               | Verify with `dataos-ctl resource -t compute get -a`; set `spec.compute`.                         |
| Tenant-level `vulcan-trino` stack        | Provides the runtime image, serving sidecars, catalog templating, and per-role config rendering. |
| One depot per source you read            | Each becomes a Trino catalog.                                                                    |
| A writable lakehouse depot, listed first | The first depot in `spec.depots[]` is the default catalog and materialization target.            |
| `vulcan-state-connection` secret         | External Postgres for Vulcan plan/interval state.                                                |
| `vulcan-object-store-connection` secret  | S3 bucket for Vulcan artifacts and result spooling.                                              |
| Git-sync secret                          | Used by the resource to pull model code.                                                         |

Three roles are required: an admin role (SRE installs the stack, provisions the pool, creates depots and tenant secrets), the Vulcan service role (the `runAsUser`; runs the cluster and workflows, reads sources, writes the materialization catalog), and a consumer role (read-only via API endpoints). Minimum source grants by connector:

```
Iceberg lakehouse:  rw on the lakehouse depot; storage creds on the depot
Postgres:           a read (or rw) role on the target database; credentials on the depot
Snowflake:          a role/warehouse with USAGE + SELECT; user/password or key-pair on the depot secret
```

## Catalog priority

The default catalog determines where unqualified model names resolve and where the gateway connects by default.

| Catalogs present               | Default catalog                                                           |
| ------------------------------ | ------------------------------------------------------------------------- |
| Depot catalogs only            | First depot in `spec.depots[]`.                                           |
| Secret catalogs only           | First secret catalog in `spec.trino.catalog.config[]`.                    |
| Both depot and secret catalogs | First secret catalog (secret catalogs take priority over depot catalogs). |

Order each list deliberately. Set the default catalog explicitly with `use.projection` to inject `TRINO_CATALOG`, which the gateway reads via `{{ env_var('TRINO_CATALOG') }}`. The value must match an actual catalog name from your depot or secret catalog list.

{% hint style="warning" %}
Materialised tables (`FULL` and incremental kinds) land in the first depot's catalog. Make it a writable Iceberg lakehouse depot, and put source-only systems (Postgres, Snowflake) later in the list.
{% endhint %}

## config.yaml

The gateway connection is templated from environment variables the stack injects at runtime.

```yaml
name: my-trino-dp
display_name: "My Managed Trino DP"
tenant: "<tenant>"
version: "0.1.0"

gateways:
  default:
    connection:
      type: trino
      catalog: "{{ env_var('TRINO_CATALOG') }}"
default_gateway: default

model_defaults:
  dialect: trino
  start: <YYYY-MM-DD>
  cron: "@daily"

linter:
  enabled: true
  rules:
    - ambiguousorinvalidcolumn
    - invalidselectstarexpansion
    - noambiguousprojections

ignore_patterns:
  - "trino-server-deploy.yaml"
```

Add the deploy manifest filename to `ignore_patterns`, or Vulcan tries to parse it as a model.

## Deploy manifest

The `vulcan-dg-trino` resource provisions the cluster and the plan and run workflows.

```yaml
version: v1alpha
type: vulcan-dg-trino
name: my-trino-dp
owner: <owner>
spec:
  runAsUser: "<owner>"
  compute: <trino-compute-pool>
  engine: trino
  repo:
    url: <https://your-vcs/your-repo>
    syncFlags:
      - "--ref=<branch>"
      - "--submodules=off"
    baseDir: <path/to/my-trino-dp>
    secret: <tenant>:<git-sync-secret>

  depots:
    - dataos://<lakehousedepot>?purpose=rw   # FIRST = default catalog + materialization target
    - dataos://<postgresdepot>?purpose=rw    # source

  trino:
    overideCatalogConfig:
      - name: <lakehousedepot>
        properties:
          iceberg.max-partitions-per-writer: 100
    coordinator:
      trinoServerConfig:
        jvmConfig: |
          -server
          -Xmx4G
          -XX:+UseG1GC
          -XX:G1HeapRegionSize=32M
        logProperties: |
          io.trino=INFO
    workers:
      replicas: 1
      trinoServerConfig:
        jvmConfig: |
          -server
          -Xmx4G
          -XX:+UseG1GC
          -XX:G1HeapRegionSize=32M
        logProperties: |
          io.trino=INFO

  workflow:
    logLevel: INFO
    schedule:
      crons: ["0 */6 * * *"]
      endOn: "<YYYY-01-01T00:00:00-00:00>"
      timezone: "UTC"
      concurrencyPolicy: Forbid
    resource:
      request: { cpu: "1000m", memory: "2Gi" }
      limit:   { cpu: "2000m", memory: "4Gi" }
    plan:
      command: [vulcan]
      arguments: ["--log-to-stdout", "plan", "--auto-apply"]
    run:
      command: [vulcan]
      arguments: ["--log-to-stdout", "run"]

  use:
    projection:
      projections:
        envVars:
          - key: TRINO_CATALOG
            template: "<lakehousedepot>"

  api:
    replicas: 1
    resource:
      request: { cpu: "1000m", memory: "2Gi" }
      limit:   { cpu: "2000m", memory: "4Gi" }
```

### spec.depots

The first depot becomes the default catalog and materialization target, so make it a writable lakehouse depot. The catalog name is derived from the depot name. Use `?purpose=rw` for depots Vulcan writes to and `?purpose=r` for read-only sources.

### spec.trino

| Field                                     | Description                                                                                       |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `coordinator.trinoServerConfig.jvmConfig` | Full content of `jvm.config` for the coordinator. Set `-Xmx` to fit the pod memory limit.         |
| `workers.trinoServerConfig.jvmConfig`     | Full content of `jvm.config` for workers. Set the same `-Xmx` as the coordinator.                 |
| `workers.replicas`                        | Number of worker pods (default 2). Add workers for more query parallelism.                        |
| `catalog.config`                          | List of secret-backed catalogs in `<tenant>:<secret>` format.                                     |
| `overideCatalogConfig`                    | Per-catalog property overrides appended to the generated `.properties`. Spelled exactly as shown. |

## Cluster configuration

You ship the per-role server config under `spec.trino.coordinator.trinoServerConfig` and `spec.trino.workers.trinoServerConfig`. Each accepts the full file contents for `jvmConfig`, `configProperties`, `nodeProperties`, and `logProperties`. Unless you have a reason to override, ship the defaults: they form a working cluster.

### jvm.config and the heap rule

{% hint style="warning" %}
The number-one deployment failure: if the JVM heap is left unmanaged it can exceed the pod memory limit, so workers run out of memory or never register and the cluster forms with 0 workers. Always set `-Xmx` in `jvmConfig` for both the coordinator and every worker, with the pod `resource.limit.memory` above it. `4G` is the validated value for a `4Gi` pod limit; size yours to your own limit.
{% endhint %}

The validated working `jvmConfig` is the stack default plus an explicit `-Xmx`:

```properties
-server
-Xmx4G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-Djdk.attach.allowAttachSelf=true
--enable-native-access=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--sun-misc-unsafe-memory-access=allow
```

The default property set deliberately omits memory-cap properties (`query.max-memory`, `query.max-total-memory`, `query.max-memory-per-node`, `memory.heap-headroom-per-node`). An incorrect cap relative to the heap kills queries with out-of-memory errors. Run on defaults; add caps only after testing on your own cluster, keeping them below the heap.

### config.properties and the override rule

Leave `configProperties` unset to use the validated default. When you must set it, the stack still prepends the two cluster-identity lines (`discovery.uri`, `internal-communication.shared-secret`). The default block is replaced, not merged, so re-supply the role lines yourself and do not duplicate the injected lines.

```yaml
# coordinator: explicit baseline equivalent to the default
configProperties: |
  coordinator=true
  node-scheduler.include-coordinator=false
  http-server.http.port=8080
```

```yaml
# worker: explicit baseline equivalent to the default
configProperties: |
  coordinator=false
  http-server.http.port=8080
```

### node.properties

The default sets `node.data-dir=/data/trino` and `node.environment=dataos`, and auto-generates `node.id` per pod. `node.environment` must be identical on coordinator and workers (use simple lowercase names; the stack normalises them at startup), or the cluster will not form. Never set `node.id` on workers: it is shared across replicas and collides.

### Cluster-identity guard rails

Do not change these: `discovery.uri` and `internal-communication.shared-secret` are stack-injected; a worker must not have `coordinator=true`; `node.environment` must match across roles; and `node.id` must not be set on workers.

### Catalog .properties

You do not hand-author catalog files. The stack generates one `<catalog>.properties` per depot in `spec.depots[]` (the catalog name is derived from the depot name) and drops it into `/usr/trino/etc/catalog/`. The three validated connectors:

Iceberg lakehouse (ABFSS storage):

```properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.file-format=PARQUET
iceberg.rest-catalog.uri=<metastoreUrl><metastoreRelativePath>
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.token=<dataos run-as apikey>
iceberg.rest-catalog.warehouse=abfss://<container>@<account>.<suffix>/<relativePath>
fs.native-azure.enabled=true
azure.auth-type=ACCESS_KEY
azure.access-key=<from depot secret>
```

For S3-backed lakehouse depots the storage lines become `fs.native-s3.enabled=true`, `s3.region`, `s3.aws-access-key`, and `s3.aws-secret-key`.

Postgres:

```properties
connector.name=postgresql
connection-url=jdbc:postgresql://<host>:<port>/<database>
connection-user=<from depot/secret>
connection-password=<from depot/secret>
```

Snowflake:

```properties
connector.name=snowflake
connection-url=jdbc:snowflake://<account>.snowflakecomputing.com
connection-user=<from depot/secret>
connection-password=<from depot/secret>
snowflake.account=<account>
snowflake.database=<database>
snowflake.role=<role>
snowflake.warehouse=<warehouse>
```

To append extra properties to a generated catalog, use `spec.trino.overideCatalogConfig` keyed by the catalog (depot) name.

### Secret-backed catalog

A secret catalog defines Trino catalog properties via a DataOS secret instead of a depot. The secret keys must be valid Trino catalog property names. The catalog name is the secret name, and a data product can run with only secret catalogs and no depots.

```yaml
spec:
  trino:
    catalog:
      config:
        - "<tenant>:postgresrr"   # secret key/values → catalog .properties
  use:
    projection:
      projections:
        envVars:
          - key: TRINO_CATALOG
            template: "postgresrr"
```

### Custom plugins

Custom connector, UDF, or plugin JARs go under `dependencies/java/<plugin>/`, which the stack copies to `/usr/lib/trino/plugin/` on startup. A loose JAR placed directly at `dependencies/java/foo.jar` will not load; it must live in a subdirectory, which becomes the plugin folder name.

## Materialisation behaviour per model kind

| Model kind                  | Trino operation                    | Notes                                                            |
| --------------------------- | ---------------------------------- | ---------------------------------------------------------------- |
| `VIEW`                      | `CREATE VIEW` (or logical staging) | Federated staging layer; read across depot catalogs.             |
| `FULL`                      | `CREATE TABLE AS` / replace        | Dimensions, derived aggregates, rebuildable marts.               |
| `SEED`                      | File-backed reference data         | Small reference tables.                                          |
| `INCREMENTAL_BY_TIME_RANGE` | Insert/overwrite by time window    | Set `time_column`; filter with `>= @start_dt AND < @end_dt`.     |
| `INCREMENTAL_BY_PARTITION`  | Insert/overwrite by partition key  | Best when the partition key is the natural restatement boundary. |

```sql
MODEL (
  name lakehouse.gold.orders_daily,
  kind INCREMENTAL_BY_TIME_RANGE (time_column order_date),
  start '2026-06-01',
  cron '@daily',
  grain (order_id),
  partitioned_by (order_date),
  columns (
    order_id BIGINT,
    customer_id BIGINT,
    order_date TIMESTAMP(6)
  )
);
SELECT order_id, customer_id, order_date
FROM lakehouse.staging.orders
WHERE order_date >= @start_dt
  AND order_date <  @end_dt;
```

SCD Type 2 and unique-key MERGE are not exercised in the reference projects; use `FULL` or partition/time-range incrementals, and validate Iceberg MERGE separately if you need it.

## Operational boundaries

### Compute sizing

The reference sizing that runs the full multi-depot demo (VIEW staging, FULL plus INCREMENTAL models, semantics, metrics, and DQ across three depot catalogs):

| Setting                       | Value                       |
| ----------------------------- | --------------------------- |
| `spec.trino.workers.replicas` | `1`                         |
| Coordinator / worker request  | `cpu: 1000m`, `memory: 2Gi` |
| Coordinator / worker limit    | `cpu: 2000m`, `memory: 4Gi` |
| `-Xmx`                        | `4G`                        |

Coordinator and worker resources fall back to `spec.workflow.resource` if not set per role. Add workers (not coordinators) for more query parallelism.

### Concurrency and scheduling

Use `concurrencyPolicy: Forbid` to prevent overlapping writers to the same Iceberg target, add workers for query throughput (the coordinator stays at 1), constrain federated joins, and keep `retry-policy=NONE` (fault-tolerant execution needs an exchange manager that is not enabled in the reference deployment). Schedule after upstream sources land, set `timezone: UTC`, and set `endOn` 1 to 2 years out. The reference deploy uses `crons: ["0 */6 * * *"]`.

### Latency floor and cost

The latency floor is cluster cold start (coordinator plus worker registration), the Vulcan readiness gate (waits for `SELECT 1` and the configured worker count), remote source round trips, federated exchange (cross-catalog joins shuffle data between workers), and API query overhead. For cost, right-size `spec.trino.workers.replicas` (idle workers cost compute), materialise hot repeatedly joined data into the lakehouse instead of re-federating, keep `iceberg.max-partitions-per-writer` sane (for example, 100) via `overideCatalogConfig`, and push predicates to remote sources.

## Failure modes and troubleshooting

| Symptom                                               | Likely cause                                                            | Fix                                                                                                             |
| ----------------------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Cluster forms with 0 workers / workers OOM at startup | Unmanaged JVM heap exceeds the pod limit                                | Set `-Xmx` to fit the pod limit for both roles (`4G` for `4Gi`).                                                |
| Workers never register / cluster will not form        | `node.environment` mismatch or `coordinator=true` on workers            | Make `node.environment` identical; `coordinator=false` on workers.                                              |
| Duplicate node errors across worker pods              | `node.id` set in worker `nodeProperties`                                | Remove `node.id` from workers; it auto-generates per pod.                                                       |
| `Table not found` for a source                        | Cross-catalog source not declared / wrong catalog name                  | Add to `external_models.yaml`; use `catalog.schema.table`.                                                      |
| Plan works locally, fails in DataOS                   | Depot missing/misnamed in `spec.depots[]` or wrong default catalog      | Verify with `SHOW CATALOGS`; first depot is the default catalog.                                                |
| Iceberg timestamp precision mismatch                  | `TIMESTAMP` precision differs from the catalog                          | Declare the precision the catalog expects (`TIMESTAMP(6)` preferred).                                           |
| Catalog present but queries fail                      | Missing creds or permission/schema/warehouse issues surfacing on access | `SHOW SCHEMAS FROM <catalog>`, then `SELECT 1 FROM <catalog>.<schema>.<table> LIMIT 1`; add creds and re-apply. |
| Custom plugin/UDF not loaded                          | JAR at `dependencies/java/` root, not in a subdir                       | Move it into `dependencies/java/<plugin>/`.                                                                     |
| Query OOM-killed                                      | JVM heap too small (or memory caps set above heap)                      | Raise `-Xmx` and the pod limit; keep any caps below the heap.                                                   |
| `configProperties` override breaks the cluster        | Duplicated identity lines or missing role lines                         | Re-supply role lines; do not duplicate the injected lines.                                                      |
| Production `vulcan migrate` fails                     | Missing/wrong `vulcan-state-connection` (Postgres)                      | Fix the tenant state secret; ensure Postgres is reachable.                                                      |
| plan/run pod hangs at startup                         | Readiness gate waiting for the configured worker count                  | Check worker pods are `active`; verify `TRINO_EXPECTED_WORKERS`.                                                |

### Logs and recovery

Inspect coordinator health with `ds resource -t service -n <name>-trino logs -l 100`, worker health with `ds resource -t service -n <name>-trino-workers logs -l 100`, and the rendered config inside a pod with `ds resource -t service -n <name>-trino exec -- cat /usr/trino/etc/config.properties`. Check cluster membership with `trino --catalog system --schema runtime --execute "SELECT node_id, coordinator, state FROM nodes"`. For recovery: if the cluster formed with 0 workers, set `-Xmx`, re-apply, and confirm workers are `active`; re-run failed time-range windows (partition overwrite is safe to replay); update rotated catalog credentials on the depot/secret and re-apply (this regenerates `.properties`); and for bad data materialised to Iceberg, use snapshot rollback with the platform SRE team.

## Related

* [Connect engine → Dedicated Trino](https://v2.dataos.info/build/productize/connect-engine/trino/dedicated-trino) for the quick steps.
* [Trino engine overview](/references/engine-guide/trino.md) for the common rules.
* [Minerva](/references/engine-guide/trino/minerva.md) and [External Trino](/references/engine-guide/trino/external-trino.md) for the other deployment shapes.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/engine-guide/trino/dedicated-trino.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.