> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/v1/engine-guide/trino-1/dedicated-cluster.md).

# Dedicated cluster

Dedicated Trino means the Trino cluster is provisioned and owned by your Data Product deployment. The `vulcan-dg-trino` resource creates a Trino coordinator and workers for you and generates Trino catalogs from the depots you declare. You get one self-contained, federated cluster per Data Product that owns its own compute.

This is the right shape when a Data Product needs its own isolated cluster, federates across several sources, and materializes into a lakehouse it controls. To share one large central cluster instead, see [Minerva](/references/v1/engine-guide/trino-1/minerva-1.md). To connect to a cluster that already exists outside DataOS, see [External cluster](/references/v1/engine-guide/trino-1/external-trino.md). For the rules common to all three shapes, see [Trino](/references/v1/engine-guide/trino-1.md).

{% hint style="info" %}
Tested Vulcan image: `tmdcio/vulcan-trino:0.228.1.26`. Tested Trino server image: `tmdcio/trino:5.1.12`. Stack: `vulcan+trino:1.0`.
{% endhint %}

## Core settings <a href="#core-settings" id="core-settings"></a>

| Setting        | Value                                                            |
| -------------- | ---------------------------------------------------------------- |
| Resource type  | `vulcan-dg-trino`                                                |
| Engine field   | `spec.engine: trino`                                             |
| Model dialect  | `trino`                                                          |
| VDE            | `false` (not supported on Trino)                                 |
| Catalog source | Generated from `spec.depots[]` and `spec.trino.catalog.config[]` |

## Architecture

A dedicated Trino Data Product renders as five cooperating resources:

```
vulcan resource (spec.engine: trino)
├── <name>-trino           (service)  → Trino coordinator (replicas: 1)
├── <name>-trino-workers   (service)  → Trino workers (spec.trino.workers.replicas, default 2)
├── <name>-plan            (workflow) → vulcan migrate, then plan --auto-apply (waits for cluster ready)
├── <name>-run             (workflow) → vulcan run (scheduled; depends on plan)
└── <name>-api             (service)  → Vulcan API — REST/GraphQL/MySQL-wire access for consumers (spec.api)
```

The plan/run pods carry a **trino-ready init container**. This Vulcan readiness gate blocks until the coordinator answers `SELECT 1` and the configured worker count is `active`, before Vulcan starts. (Trino itself can run with fewer workers; this gate is a Vulcan operational rule, not a Trino requirement.)

## Prerequisites

Three roles are involved, each with a distinct scope:

| Role                | Who holds it                    | Purpose                                                                                                                   |
| ------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Admin role          | DataOS / platform SRE           | Installs the `vulcan-trino` stack, provisions the compute pool, creates depots, and the tenant state/object-store secrets |
| Vulcan service role | The `runAsUser` on the resource | Runs the coordinator/workers and the plan/run workflows; reads source catalogs and writes the materialization catalog     |
| Consumer role       | BI users, endpoint consumers    | Read-only access to Data Product tables via Vulcan API endpoints                                                          |

**Ask your DataOS SRE team for:**

| Requirement                                    | Notes                                                                                                                                |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Trino-capable compute pool                     | Verify with `dataos-ctl resource -t compute get -a`; set `spec.compute` to this pool                                                 |
| Tenant-level `vulcan-trino` stack              | Runtime image, serving sidecars, catalog templating, per-role Trino config rendering                                                 |
| Depots for every source you read               | One depot per source system. Each becomes a Trino catalog                                                                            |
| A materialization depot                        | The **first** depot in `spec.depots[]` is the default catalog where FULL/INCREMENTAL models land. Make it a writable lakehouse depot |
| Tenant `vulcan-state-connection` secret        | External Postgres for Vulcan plan/interval state                                                                                     |
| Tenant `vulcan-object-store-connection` secret | S3 bucket for Vulcan artifacts / result spooling                                                                                     |
| Git-sync secret                                | Used to pull model code (`spec.repo.secret`)                                                                                         |

**Minimum source grants:**

| Source                 | Grant                                                                                                |
| ---------------------- | ---------------------------------------------------------------------------------------------------- |
| Iceberg lakehouse      | `rw` on the lakehouse depot; storage creds (S3 access/secret or ABFSS account key) live on the depot |
| Postgres               | A read (or `rw`) role on the target database; credentials on the depot                               |
| Snowflake              | A role/warehouse with `USAGE` + `SELECT`; user/password or key-pair on the depot secret              |
| Databricks (read-only) | PAT token on the secret catalog (Unity Catalog Iceberg REST endpoint)                                |

{% hint style="info" %}
Local dev only needs Python 3.10 (`python --version`). The runtime version inside the image is managed for you.
{% endhint %}

### Before you start

Check:

**Trino / source side**

* [ ] Trino-capable compute pool exists; name noted for `spec.compute`
* [ ] Tenant-level `vulcan-trino` stack installed (`vulcan+trino:1.0`)
* [ ] One depot per source system you'll federate
* [ ] The first depot in `spec.depots[]` is a writable lakehouse depot
* [ ] Source credentials live on each depot/secret; never in project code
* [ ] Tenant state and object-store secrets exist
* [ ] Git-sync secret exists and matches `spec.repo.secret`

**Local development**

* [ ] Local Trino cluster starts (coordinator + workers) via Docker Compose
* [ ] Vulcan wheel matches the deployed `vulcan-trino` image line
* [ ] Local `config.yaml` points to the local coordinator and a DuckDB state DB
* [ ] `vulcan plan` and at least one representative `vulcan run` succeed locally

**Production**

* [ ] Resource has `type: vulcan` and `spec.engine: trino`
* [ ] `spec.trino.coordinator`/`workers` JVM heap set with `-Xmx` sized to the pod memory limit (`4G` is the validated reference value)
* [ ] `spec.trino.workers.replicas` set to the intended worker count
* [ ] If overriding `configProperties`, role lines included and discovery/shared-secret **not** duplicated
* [ ] `node.environment` identical on coordinator and workers; `node.id` **not** set on workers
* [ ] `timezone: UTC`; `endOn` ≥ 1–2 years out; `concurrencyPolicy: Forbid`
* [ ] API replicas and resources sized separately from the cluster

## Local development

Install Vulcan as a Python wheel, then point it at the Dockerized Trino cluster:

```bash
pip install "/path/to/vulcan-<version>-py3-none-any.whl[trino]"
vulcan --version
```

Local development runs a real Trino coordinator + worker in Docker, plus a local Postgres container that Trino federates against. This is what the hello-world starter below reads from and writes to. Trino needs no local JVM, so there's no separate "Python in Docker" variant and no `vulcan-cli` container: Vulcan runs as a plain Python install on your host and talks to the Dockerized cluster over HTTP.

**`docker/docker-compose.yml`** - coordinator and worker:

```yaml
services:
  trino-coordinator:
    image: docker.io/tmdcio/trino:5.1.12
    platform: linux/amd64
    hostname: trino-coordinator
    ports:
      - "18080:8080"
    volumes:
      - ./trino/coordinator/etc:/usr/lib/trino/etc:ro
  trino-worker-1:
    image: docker.io/tmdcio/trino:5.1.12
    platform: linux/amd64
    hostname: trino-worker-1
    volumes:
      - ./trino/worker/etc:/usr/lib/trino/etc:ro
    depends_on: [trino-coordinator]
```

{% hint style="warning" %}
Mount catalog config at `/usr/lib/trino/etc`. That is Trino's real config path inside the image. A different mount path fails silently: the container starts, but your `coordinator/etc`/`worker/etc` files are never read.
{% endhint %}

Each `etc/` directory needs the base Trino config files, not just `catalog/`. Trino won't start without them:

**`trino/coordinator/etc/node.properties`**

```properties
node.environment=production
node.data-dir=/data/trino
```

**`trino/coordinator/etc/jvm.config`**

```properties
-server
-Xmx2G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
```

**`trino/coordinator/etc/config.properties`**

```properties
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
discovery.uri=http://trino-coordinator:8080
```

**`trino/coordinator/etc/log.properties`**

```properties
io.trino=INFO
```

Mirror `node.properties`, `jvm.config`, and `log.properties` under `trino/worker/etc/` unchanged, except `config.properties`, which flips the role line:

**`trino/worker/etc/config.properties`**

```properties
coordinator=false
http-server.http.port=8080
discovery.uri=http://trino-coordinator:8080
```

**`docker/docker-compose.warehouse.yml`** - local Postgres source, backing the `postgresvulcantest` catalog:

```yaml
services:
  warehouse:
    image: postgres:15-alpine
    container_name: trino-local-warehouse
    environment:
      POSTGRES_DB: warehouse
      POSTGRES_USER: vulcan
      POSTGRES_PASSWORD: vulcan
      POSTGRES_HOST_AUTH_METHOD: trust
    ports:
      - "5435:5432"
    volumes:
      - warehouse-data:/var/lib/postgresql/data

volumes:
  warehouse-data:
```

**`trino/coordinator/etc/catalog/postgresvulcantest.properties`** (mirror under `trino/worker/etc/catalog/`):

```properties
connector.name=postgresql
connection-url=jdbc:postgresql://host.docker.internal:5435/warehouse
connection-user=vulcan
connection-password=vulcan
```

{% hint style="warning" %}
`host.docker.internal` resolves on Docker Desktop (macOS/Windows). On native Linux Docker, add `extra_hosts: ["host.docker.internal:host-gateway"]` to the `trino-coordinator`/`trino-worker-1` services in `docker-compose.yml`, or point `connection-url` at the host's real IP instead.
{% endhint %}

Start both stacks and verify:

```bash
docker compose -f docker/docker-compose.warehouse.yml up -d
docker compose -f docker/docker-compose.yml up -d
docker exec -i trino-coordinator trino --execute "SHOW SCHEMAS FROM postgresvulcantest"
```

### config.yaml (local)

```yaml
gateways:
  default:
    connection:
      type: trino
      host: localhost
      port: 18080
      user: trino
      catalog: postgresvulcantest   # default catalog where models materialize
      http_scheme: http
      method: no-auth               # TLS verify is irrelevant over plain http
    state_connection:
      type: duckdb                  # local dev only
      database: ./.state.db

default_gateway: default

model_defaults:
  dialect: trino
  start: 2026-06-01
  cron: "@daily"

linter:
  enabled: true
  rules:
    - ambiguousorinvalidcolumn
    - invalidselectstarexpansion
    - noambiguousprojections

ignore_patterns:
  - "*-deploy.yaml"             # never let the deploy manifest be parsed as a model
```

{% hint style="warning" %}
`ignore_patterns` matters. The Trino server deploy manifest lives inside the project, so add it here or Vulcan will try to parse it as a model.
{% endhint %}

In production the gateway connection is templated from environment variables the stack injects (`TRINO_HOST`, `TRINO_PORT`, `TRINO_USER`, `TRINO_CATALOG`, `TRINO_HTTP_SCHEME`, `TRINO_METHOD`). See the deployment section below. Drop `state_connection` entirely when you move to production: DataOS provisions and wires the state store automatically from the tenant `vulcan-state-connection` secret, so there's nothing to add to `config.yaml` for it.

### Hello-world starter

Every model uses the fully-qualified three-part name: `<trino_catalog>.<schema>.<table>`. The warehouse starts empty, so seed the raw table first, then build a summary model on top of it.

**`seeds/orders.csv`**

```csv
order_id,customer_id,total_price,order_date
O001,C001,120.50,2025-01-05
O002,C002,89.99,2025-01-06
O003,C001,45.00,2025-01-07
```

**`models/seeds/raw_orders.sql`**

```sql
MODEL (
  name postgresvulcantest.raw.orders,
  kind SEED (
    path '../../seeds/orders.csv'
  ),
  columns (
    order_id VARCHAR,
    customer_id VARCHAR,
    total_price DECIMAL(10,2),
    order_date DATE
  ),
  grain order_id
);
```

**`models/full/orders_summary.sql`**

```sql
MODEL (
  name postgresvulcantest.sales.orders_summary,
  kind FULL,
  grain order_id,
  assertions (
    unique_values(columns := order_id),
    not_null(columns := (order_id, customer_id))
  ),
  columns (
    order_id VARCHAR,
    customer_id VARCHAR,
    total_price DECIMAL(10,2),
    order_date DATE
  )
);
SELECT order_id, customer_id, total_price, order_date
FROM postgresvulcantest.raw.orders;
```

**`models/semantics/orders.yml`**

```yaml
kind: semantic
name: orders
depends_on: postgresvulcantest.sales.orders_summary
dimensions:
  - order_date
measures:
  - name: total_sales
    type: sum
    expression: "{orders.total_price}"
```

**`metrics/daily_revenue.yml`**

```yaml
kind: metric
name: daily_revenue
measure: orders.total_sales
granularity: day
```

### Validate the connection

Run these once the local cluster and warehouse are up, `config.yaml` exists, and the hello-world models are in place:

```bash
vulcan migrate        # initializes Vulcan state (DuckDB local / Postgres prod)
vulcan plan           # dry-run — you should see the seed and orders_summary staged
vulcan run            # materializes postgresvulcantest.sales.orders_summary
```

If `vulcan plan` succeeds, your local setup is complete. Call the REST endpoint to confirm end-to-end. Common failures at this step are in the troubleshooting table below.

## Cluster configuration

Per-role server config sits under `spec.trino.coordinator.trinoServerConfig` and `spec.trino.workers.trinoServerConfig`. Each field takes the **full file contents** for `jvmConfig`, `configProperties`, `nodeProperties`, and `logProperties`.

{% hint style="info" %}
Only the stack's default property files, plus the one validated change (`-Xmx4G` in `jvmConfig`), are documented here. These are the configurations known to work. Trino exposes many more tuning properties (memory caps, query timeouts, scheduler tuning, fault tolerance); they're intentionally left out until you've tested them on your own cluster.
{% endhint %}

Unless you have a reason to override, ship the defaults below as-is. They form a working cluster.

### jvm.config

Stack default (coordinator and worker):

```properties
-server
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-Djdk.attach.allowAttachSelf=true
--enable-native-access=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--sun-misc-unsafe-memory-access=allow
```

{% hint style="danger" %}
**The one change you must make:** set `-Xmx` to fit the worker memory limit. Leave the heap unmanaged and the effective heap can exceed the container limit. Workers OOM or fail to register. `4G` is the validated value for the reference deployment's `4Gi` pod limit; size yours to your own limit.
{% endhint %}

```yaml
# validated working jvmConfig for the reference deployment (both roles)
jvmConfig: |
  -server
  -Xmx4G
  -XX:+UseG1GC
  -XX:G1HeapRegionSize=32M
  -XX:+UseGCOverheadLimit
  -XX:+ExplicitGCInvokesConcurrent
  -XX:+HeapDumpOnOutOfMemoryError
  -Djdk.attach.allowAttachSelf=true
  --enable-native-access=ALL-UNNAMED
  --add-opens=java.base/java.nio=ALL-UNNAMED
  --sun-misc-unsafe-memory-access=allow
```

### config.properties

Stack default - coordinator:

```properties
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
discovery.uri=http://<name>-trino.<namespace>.svc.cluster.local:8080
internal-communication.shared-secret=dataos_trino_internal_shared_secret_<name>_cluster
```

Stack default - worker:

```properties
coordinator=false
http-server.http.port=8080
discovery.uri=http://<name>-trino.<namespace>.svc.cluster.local:8080
internal-communication.shared-secret=dataos_trino_internal_shared_secret_<name>_cluster
```

This is the validated, working default. Leave `configProperties` unset to use it.

If you must set `configProperties`, know that the stack still prepends the two cluster-identity lines automatically, and the platform default block is **replaced, not merged**. Re-supply the role lines yourself, and don't duplicate the two injected lines:

```yaml
# coordinator — explicit baseline equivalent to the default
configProperties: |
  coordinator=true
  node-scheduler.include-coordinator=false
  http-server.http.port=8080
```

```yaml
# worker — explicit baseline equivalent to the default
configProperties: |
  coordinator=false
  http-server.http.port=8080
```

### log.properties

Stack default (coordinator and worker), validated:

```properties
io.trino=INFO
```

### node.properties

Stack default (coordinator and worker):

```properties
node.data-dir=/data/trino
node.environment=dataos
# node.id is auto-generated per pod when omitted
```

`node.environment` must be **identical** on coordinator and workers, or the cluster won't form (the stack normalizes it to lowercase/alphanumerics at startup, so keep it simple). Never set `node.id` on workers. It is shared across replicas and would collide; a unique `node.id` is auto-generated per pod at startup.

### Cluster-identity guard rails - don't change these

| Line                                     | Why                                                                                      |
| ---------------------------------------- | ---------------------------------------------------------------------------------------- |
| `discovery.uri`                          | Stack-injected; wrong value breaks cluster membership                                    |
| `internal-communication.shared-secret`   | Stack-injected; mismatch blocks inter-node RPC                                           |
| `coordinator=true` on a worker           | A coordinator-configured node can't act as worker-only; the cluster won't form correctly |
| `node.environment` mismatch coord↔worker | Workers won't register                                                                   |
| `node.id` on workers                     | Shared across replicas → duplicate IDs                                                   |

## How catalogs get created

You never hand-author catalog files on a dedicated cluster. There are two ways a catalog gets mounted, and the difference determines which one wins as the default gateway catalog.

|                    | Depot catalog                                                           | Secret catalog                                                                                     |
| ------------------ | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Defined in         | `spec.depots[]`                                                         | `spec.trino.catalog.config[]`                                                                      |
| Catalog name       | The depot name (`dataos://<name>` → catalog `<name>`)                   | The DataOS secret name (last segment after `:`)                                                    |
| `.properties` file | Generated by the stack from depot type + credentials                    | Built directly from the secret's key/value pairs: keys must be valid Trino catalog property names  |
| Credential storage | Managed by the DataOS depot (attached secret, not inline)               | A DataOS `Secret` resource holding raw Trino property keys                                         |
| Typical use        | Lakehouse depots (Iceberg/S3/ABFSS), managed Postgres, Snowflake depots | Sources without a DataOS depot; cross-tenant or external endpoints; fine-grained connector control |
| Example            | `dataos://<lakehouse>?purpose=rw`                                       | `"<tenant>:<secret-name>"`                                                                         |

Both can coexist on the same DP, and a DP can run entirely on secret catalogs with no depots at all.

**Default catalog priority.** The stack resolves the default catalog. It is what Vulcan uses when no catalog is specified and what is surfaced as the gateway catalog on the API. The order is:

```
1. First entry in spec.trino.catalog.config[]   (secret catalog — highest priority)
2. First entry in spec.depots[]                  (depot catalog — only if no secret catalogs)
```

Secret catalogs always win over depot catalogs. If your DP has both, order each list deliberately, or better, set the default explicitly:

```yaml
# In config.yaml — gateway reads from env var
gateways:
  default:
    connection:
      type: trino
      catalog: "{{ env_var('TRINO_CATALOG') }}"
      # ... other connection fields

# In the deploy resource — set the env var value
spec:
  use:
    projection:
      projections:
        envVars:
          - key: TRINO_CATALOG
            template: "<catalog-name>"   # must match an existing catalog: depot name or secret name
```

This survives any future reordering of `spec.depots[]` or `spec.trino.catalog.config[]`.

{% hint style="warning" %}
**Postgres via Trino - disable comment registration.** The Postgres connector rejects the DDL Vulcan uses to register column comments. Set `register_comments: false` on the gateway connection whenever Postgres is the gateway catalog:

```yaml
gateways:
  default:
    connection:
      type: trino
      register_comments: false
```

{% endhint %}

## Connector reference

A secret catalog is a DataOS `Secret` whose key/value pairs map 1:1 to a Trino `<catalog>.properties` file. The secret name (last segment of `<tenant>:<secret-name>`) becomes the Trino catalog name:

```yaml
version: v1
kind: Secret
name: <secret-name>
layer: user
type: key-value
data:
  connector.name: <trino-connector-type>
  # ... remaining .properties key/value pairs for that connector
```

Apply it, then reference it in the deploy resource:

```bash
dataos-ctl resource apply -f secret.yaml
```

```yaml
spec:
  trino:
    catalog:
      config:
        - "<tenant>:<secret-name>"
```

**Postgres**: Exposes a Postgres database as a Trino catalog.

```yaml
connector.name: postgresql
connection-url: jdbc:postgresql://<host>:<port>/<database>
connection-user: <username>
connection-password: <password>
```

Optional tuning (secret, or via `overideCatalogConfig`):

```yaml
case-insensitive-name-matching: "true"      # match tables regardless of case
metadata.cache-ttl: "30s"                   # cache schema metadata for 30 seconds
```

Postgres doesn't support partitioned incremental writes through Trino. Use `kind FULL` or `kind VIEW` for Postgres-backed models. It also rejects comment DDL, so always set `register_comments: false` when Postgres is the gateway catalog.

**BigQuery**: Connects to a Google BigQuery project as a read and write source. Managed Trino has no file mount, so provide the service account key inline as a base64-encoded string.

```yaml
connector.name: bigquery
bigquery.project-id: <gcp-project-id>
bigquery.credentials-key: <base64-encoded-service-account-json>
```

**Snowflake**: All compute runs on the Trino cluster; Snowflake is a data source only.

```yaml
connector.name: snowflake
connection-url: jdbc:snowflake://<account-identifier>.snowflakecomputing.com
connection-user: <username>
connection-password: <password>
snowflake.account: <account-identifier>
snowflake.database: <database>
snowflake.role: <role>
snowflake.warehouse: <snowflake-virtual-warehouse>
```

Optional: `snowflake.jdbc.override-url: jdbc:snowflake://<account>.snowflakecomputing.com/?<params>`

**Databricks (Delta Lake)**: Read-only source, connected via Iceberg REST against Unity Catalog. Vulcan writes materialized output to a separate lakehouse depot; writing into Databricks tables through Trino isn't supported.

```yaml
connector.name: iceberg
iceberg.catalog.type: rest
iceberg.rest-catalog.uri: https://<workspace-host>/api/2.1/unity-catalog/iceberg
iceberg.rest-catalog.security: OAUTH2
iceberg.rest-catalog.oauth2.token: <databricks-personal-access-token>
iceberg.rest-catalog.warehouse: <unity-catalog-name>
iceberg.rest-catalog.case-insensitive-name-matching: "true"
```

Delta Lake tables read through the Iceberg REST interface can fan out wide splits. Size the coordinator and workers with larger JVM heaps (`-Xmx12G`) and more CPU/memory when Databricks is a primary source:

```yaml
spec:
  trino:
    coordinator:
      resource:
        request: { cpu: "2000m", memory: "8Gi" }
        limit:   { cpu: "4000m", memory: "16Gi" }
      trinoServerConfig:
        jvmConfig: |
          -server
          -Xmx12G
          -XX:+UseG1GC
          ...
    workers:
      replicas: 2
      resource:
        request: { cpu: "2000m", memory: "8Gi" }
        limit:   { cpu: "4000m", memory: "16Gi" }
```

Typical pattern: read Databricks, write to a lakehouse depot. Because the secret catalog takes priority by default, override `TRINO_CATALOG` to the depot name when you want the lakehouse to be the default write target:

```yaml
spec:
  depots:
    - dataos://<lakehouse-depot>?purpose=rw   # write target; first depot = default catalog
  trino:
    catalog:
      config:
        - "<tenant>:<databricks-secret>"      # read-only Databricks catalog (secret takes priority)
  use:
    projection:
      projections:
        envVars:
          - key: TRINO_CATALOG
            template: "<lakehouse-depot>"     # override to make depot the gateway/write target
```

**Lakehouse depot (AWS S3 / Azure ABFSS)**: Supports both S3 and ABFSS backends. This is the primary write/materialization target for a dedicated cluster. The DataOS depot carries all storage credentials; the stack generates the Trino catalog from it automatically. There is nothing to hand-author. Add it as the **first** entry in `spec.depots[]` to make it the default materialization catalog:

```yaml
spec:
  depots:
    - dataos://<lakehouse-depot>?purpose=rw   # first depot → default Trino catalog for writes
```

Add extra Iceberg tuning on top of the auto-generated catalog with `overideCatalogConfig`:

```yaml
spec:
  trino:
    overideCatalogConfig:
      - name: <lakehouse-depot>
        properties:
          iceberg.max-partitions-per-writer: "100"
          iceberg.rest-catalog.case-insensitive-name-matching: "true"
```

Snowflake and Postgres are also supported as depots. When added to `spec.depots[]`, the stack auto-generates the catalog the same way. No manually authored secret is needed.

### Per-catalog property overrides (`overideCatalogConfig`)

`spec.trino.overideCatalogConfig` appends extra key/value pairs to a generated catalog's `.properties` file, after the stack renders the base properties from the depot or secret. It works on both depot-generated and secret catalogs, keyed by depot/secret name.

{% hint style="warning" %}
Spelling note: the field is `overideCatalogConfig`. It has one `r` in "overide" and matches the actual field name in the `spec.trino` schema. This is not a typo.
{% endhint %}

```yaml
spec:
  trino:
    overideCatalogConfig:
      - name: <catalog-name>            # depot name or secret name (not the full "tenant:secret" form)
        properties:
          <trino-property-key>: "<value>"   # always quote values as strings
```

Multiple catalogs can each have their own override block:

```yaml
overideCatalogConfig:
  - name: <iceberg-catalog>
    properties:
      iceberg.max-partitions-per-writer: "100"
      iceberg.rest-catalog.case-insensitive-name-matching: "true"
      iceberg.rest-catalog.case-insensitive-name-matching.cache-ttl: "30s"
  - name: <postgres-catalog>
    properties:
      case-insensitive-name-matching: "true"
      metadata.cache-ttl: "30s"
```

Use this for tuning concerns that aren't credentials, such as `iceberg.max-partitions-per-writer`, `case-insensitive-name-matching`, and `metadata.cache-ttl`. Keep these separate from the secret when they differ across dev/prod.

Confirm the properties actually loaded after deploy:

```bash
dataos-ctl resource -t vulcan -n <resource-name> logs -l 2000 2>&1 \
  | grep -E "case-insensitive-name-matching|max-partitions-per-writer|metadata\.cache-ttl"
```

If a property doesn't show up in the logs, it wasn't applied. Double-check the field spelling and that `overideCatalogConfig.name` matches the exact catalog name.

## Java plugins and UDFs

Trino loads plugins from `/usr/lib/trino/plugin/` on coordinator and workers. The `vulcan-trino` stack copies each **direct subdirectory** of `dependencies/java/` in your repo into that path at pod startup:

```
dependencies/java/
└── <plugin-folder>/         ← each direct subdir is copied as a plugin
    ├── plugin.jar
    └── ...other jars...
```

Only direct subdirectories are copied. A loose JAR at `dependencies/java/*.jar` won't load. Each JAR inside a plugin folder must contain `META-INF/services/io.trino.spi.Plugin` to count as a valid Trino SPI plugin.

{% hint style="danger" %}
Don't name your plugin folder `udfs/`. The `vulcan-trino` image already ships its own `udfs/` plugin (Minerva's `FunctionsPlugin`) at `/usr/lib/trino/plugin/udfs/`. Naming yours the same overwrites it.
{% endhint %}

**Fetching an official Trino plugin.** Commit a `fetch-plugins.sh` alongside the plugin directory to reproducibly pull JARs from the official Trino GitHub release:

```bash
#!/usr/bin/env bash
# Fetches the official Trino <plugin-name> plugin for Trino version <TRINO_VERSION>.
# The vulcan-trino stack copies each direct subdir of dependencies/java/ to /usr/lib/trino/plugin/.
set -euo pipefail

ROOT="$(cd "$(dirname "$0")" && pwd)"
TRINO_VERSION="${TRINO_VERSION:-480}"        # override: TRINO_VERSION=490 ./fetch-plugins.sh
BASE="https://github.com/trinodb/trino/releases/download/${TRINO_VERSION}"
ARTIFACT="trino-<plugin-name>-${TRINO_VERSION}"
PLUGIN_DIR="$ROOT/<plugin-folder>"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT

rm -rf "$PLUGIN_DIR"
curl -fsSL -o "$WORK/${ARTIFACT}.zip" "${BASE}/${ARTIFACT}.zip"
unzip -q "$WORK/${ARTIFACT}.zip" -d "$WORK"
mkdir -p "$PLUGIN_DIR"
cp "$WORK/${ARTIFACT}"/*.jar "$PLUGIN_DIR/"

# Validate the plugin JAR has the SPI entry point
if ! sh -c 'for jar in "$1"/*.jar; do jar tf "$jar" | grep -qx "META-INF/services/io.trino.spi.Plugin" && exit 0; done; exit 1' sh "$PLUGIN_DIR"; then
  echo "Invalid plugin payload: no JAR contains META-INF/services/io.trino.spi.Plugin" >&2
  exit 1
fi

echo "Installed $ARTIFACT -> <plugin-folder>/"
echo "Done. Commit dependencies/java/<plugin-folder>/ for git-sync."
```

```bash
bash dependencies/java/fetch-plugins.sh
git add dependencies/java/<plugin-folder>/
git commit -m "vendor trino <plugin-name> plugin"
```

**Confirm it loaded.** After `dataos-ctl resource apply`, check the coordinator and each worker's log for the plugin load line:

```bash
dataos-ctl resource -t vulcan -n <resource-name> \
  --container-group '<coordinator-container-group>' logs -l 4000 2>&1 \
  | grep -E "Loading plugin|FunctionsPlugin"
```

Expect one pair of lines per plugin folder:

```
-- Loading plugin <plugin-folder> --
Installing <plugin-class>
```

The built-in `udfs` plugin logs `-- Loading plugin udfs --` followed by `FunctionsPlugin`. If yours doesn't show up, check the subdirectory layout and the SPI service file.

**Using a plugin function.** Once loaded, reference it in any model's SQL exactly like a built-in function. No import or registration is needed:

```sql
MODEL (
  name <catalog>.<schema>.<model-name>,
  kind FULL,
  columns (id INTEGER, encoded VARCHAR)
);

SELECT id, char2hexint(name) AS encoded
FROM <source-table>;
```

`Function not registered` means the plugin didn't load. Check the log proof above.

Reference example: `trino/java-plugins-udfs/`. It validates the `teradata-functions` plugin with a FULL model and semantic model smoke test.

## Production deployment

Do this after local `plan` succeeds. Apply order matters. Each step depends on the previous:

```
Source depots → Materialization (lakehouse) depot → Git secret + tenant state/object-store secrets → config.yaml → Vulcan deploy resource
```

| Manifest                         | Purpose                                                                     | Key fields                                                                                     |
| -------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Source depots                    | One per federated source                                                    | `name`, `spec.type`, source credentials, `purpose: rw`/`r`                                     |
| Lakehouse depot (first)          | Default catalog for materialized models                                     | lakehouse `storageType` (`abfss`/`s3`) + creds                                                 |
| `secret-git-sync.yaml`           | Repo credentials for git-sync                                               | `GITSYNC_USERNAME`, `GITSYNC_PASSWORD`                                                         |
| Tenant secrets                   | `vulcan-state-connection` (Postgres), `vulcan-object-store-connection` (S3) | provisioned once per tenant by SRE                                                             |
| `config.yaml`                    | Project config + gateway templated from env                                 | `engine: trino`, env-var connection, `model_defaults`                                          |
| Deploy resource (`type: vulcan`) | Cluster + workflows + API + schedule                                        | `spec.engine: trino`, `spec.compute`, `spec.depots`, `spec.trino`, `spec.workflow`, `spec.api` |

Generate the deploy manifest from the project root:

```bash
vulcan create_deploy_yaml --output workspace-deploy.yaml
```

```
Generated deploy YAML: <project-path>/workspace-deploy.yaml
```

It is always `type: vulcan` with `spec.engine: trino`. Never use a separate `vulcan-trino` resource type. Fill in:

| Field                                           | What to set                                                                                                       |
| ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `type`                                          | `vulcan`                                                                                                          |
| `name`                                          | Lowercase DataOS resource name                                                                                    |
| `owner` / `spec.runAsUser`                      | DataOS user/service account to run the workflow                                                                   |
| `spec.compute`                                  | Trino-capable DataOS compute pool                                                                                 |
| `spec.engine`                                   | `trino`                                                                                                           |
| `spec.repo.url`                                 | Git repository URL containing the project                                                                         |
| `spec.repo.syncFlags`                           | Branch/tag ref, e.g. `--ref=<branch>`                                                                             |
| `spec.repo.baseDir`                             | Path from repo root to the Vulcan project                                                                         |
| `spec.repo.secret`                              | Git-sync secret, if the repo is private                                                                           |
| `spec.depots`                                   | One entry per source, plus the lakehouse depot **first**                                                          |
| `spec.trino.coordinator` / `spec.trino.workers` | Per-role JVM/log config                                                                                           |
| `spec.trino.overideCatalogConfig`               | Per-catalog property overrides                                                                                    |
| `spec.workflow.schedule`                        | Cron, `endOn`, timezone, `concurrencyPolicy`                                                                      |
| `spec.workflow.resource`                        | Workflow pod CPU/memory                                                                                           |
| `spec.workflow.migrate` / `.plan` / `.run`      | Commands for the migrate, plan, and run steps: migrate initializes state, then plan runs before the scheduled run |
| `spec.api`                                      | API replicas and request/limit resources                                                                          |

A complete example: daily federated DP, small project:

```yaml
version: v1alpha
type: vulcan
name: managed-trino-dp
owner: <owner>
spec:
  runAsUser: "<owner>"
  compute: <trino-compute-pool>
  engine: trino
  repo:
    url: <git-repo-url>
    syncFlags: ["--ref=<branch>", "--submodules=off"]
    baseDir: <path/to/your/project>
    secret: <tenant>:<git-sync-secret>
  depots:
    - dataos://<lakehouse-depot-name>?purpose=rw   # FIRST = default catalog + materialization target
    - dataos://<source-depot-name>?purpose=rw      # federated source
  trino:
    overideCatalogConfig:
      - name: <depot-name>
        properties:
          <property.key>: <value>
    coordinator:
      trinoServerConfig:
        jvmConfig: |
          -server
          -Xmx4G
          -XX:+UseG1GC
          -XX:G1HeapRegionSize=32M
        logProperties: |
          io.trino=INFO
    workers:
      replicas: 1
      trinoServerConfig:
        jvmConfig: |
          -server
          -Xmx4G
          -XX:+UseG1GC
        logProperties: |
          io.trino=INFO
  workflow:
    schedule:
      crons: ["0 */6 * * *"]
      endOn: "<YYYY-MM-DDT00:00:00-00:00>"
      timezone: "UTC"
      concurrencyPolicy: Forbid
    resource:
      request: { cpu: "1000m", memory: "2Gi" }
      limit:   { cpu: "2000m", memory: "4Gi" }
    migrate:
      command: [vulcan]
      arguments: ["--log-to-stdout", "migrate"]
    plan:
      command: [vulcan]
      arguments: ["--log-to-stdout", "plan", "--auto-apply"]
    run:
      command: [vulcan]
      arguments: ["--log-to-stdout", "run"]
  api:
    replicas: 1
    resource:
      request: { cpu: "1000m", memory: "2Gi" }
      limit:   { cpu: "2000m", memory: "4Gi" }
```

**Why order matters.** Each catalog `.properties` is generated from a depot that must already exist; the plan/run workflows can only sync code once the Git secret exists; and the plan pod waits for the coordinator and workers to be active before it starts. A depot name mismatch in `spec.depots[]` passes `apply` but fails at every run.

## Sizing and operational boundaries

{% hint style="danger" %}
**The #1 deployment failure:** an unmanaged JVM heap can exceed the pod memory limit, so workers OOM or never register and the cluster forms with 0 workers. Always set `-Xmx` in `trinoServerConfig.jvmConfig` to fit the pod limit, with headroom.
{% endhint %}

The default property set deliberately ships without memory-cap properties (`query.max-memory`, `query.max-total-memory`, `query.max-memory-per-node`, `memory.heap-headroom-per-node`). An incorrect cap relative to the heap OOM-kills queries, so run on defaults and add caps only when a specific workload calls for them.

**Sizing that works.** This runs the full multi-depot demo (VIEW staging, FULL + INCREMENTAL models, semantics, metrics, DQ) across three depot catalogs:

| Setting                       | Value                       |
| ----------------------------- | --------------------------- |
| `spec.trino.workers.replicas` | `1`                         |
| Coordinator / worker request  | `cpu: 1000m`, `memory: 2Gi` |
| Coordinator / worker limit    | `cpu: 2000m`, `memory: 4Gi` |
| `-Xmx`                        | `4G`                        |

Set the pod's `resource.limit.memory` **above** `-Xmx` (the reference uses a `4Gi` limit for `-Xmx4G`). Coordinator/worker resources fall back to `spec.workflow.resource` if not set per role. Add workers, not coordinators, for more query parallelism.

**Concurrency**

| Recommendation                                | Why                                                                               |
| --------------------------------------------- | --------------------------------------------------------------------------------- |
| `concurrencyPolicy: Forbid`                   | Prevent overlapping runs writing to the same Iceberg target                       |
| Add workers, not coordinators, for throughput | Coordinator stays at 1; workers execute splits                                    |
| Constrain federated joins                     | Cross-catalog joins move data over the network                                    |
| Keep `retry-policy=NONE`                      | Fault-tolerant execution needs an exchange manager not enabled in this deployment |

**API replicas.** The reference deploy runs a single API replica for REST, GraphQL, and MySQL-wire. Scale replicas for request concurrency. Scale Trino workers for query execution. These are different bottlenecks.

```yaml
api:
  replicas: 1
  resource:
    request: { cpu: "1000m", memory: "2Gi" }
    limit:   { cpu: "2000m", memory: "4Gi" }
```

**Scheduling**

| Recommendation                       | Why                                  |
| ------------------------------------ | ------------------------------------ |
| Schedule after upstream sources land | Avoid partial reads and restatements |
| `timezone: UTC`                      | Avoid DST-shifted intervals          |
| `concurrencyPolicy: Forbid`          | Prevent overlapping writers          |
| Set `endOn` ≥ 1–2 years out          | Expired schedules stop silently      |

The reference deploy schedules every 6 hours: `crons: ["0 */6 * * *"]`.

**Latency floor:** Cold cluster start (coordinator + worker registration), the Vulcan trino-ready gate, remote source round trips, federated exchange, and API query overhead all stack up before the first result. If your downstream SLO is sub-second on a cold cluster, keep the cluster warm and pre-aggregate hot marts.

## Performance tuning

Trino performance is driven by federation width, source latency, exchange size, and worker heap.

**What to measure:** cluster formation time, per-source scan time, exchange (shuffle) bytes, spilled bytes, peak query memory, worker GC pause.

**Validation loop:**

1. Start from the reference sizing above
2. Run a representative `vulcan plan` + `vulcan run`
3. Inspect the Trino UI for spill, exchange size, peak memory, stragglers
4. Apply one tuning change (heap, `query.max-memory`, worker count, or join shape)
5. Re-run and compare
6. Record the stable combination as your baseline

**Fix order for concurrency issues:** prevent overlapping writers (`concurrencyPolicy: Forbid`) → constrain federated join shapes (declare joins, push filters down) → add workers → raise the JVM heap and pod limit together, introducing memory-cap properties only after testing.

**Performance ceilings:**

| Ceiling                                      | Tune via                                                                    |
| -------------------------------------------- | --------------------------------------------------------------------------- |
| Wide federated joins shuffle large exchanges | Reduce join width; pre-aggregate; push predicates to sources                |
| Remote source is the bottleneck              | Cache/stage into the lakehouse; read the materialized copy                  |
| High-cardinality `COUNT(DISTINCT)`           | Pre-aggregate, or use `approx_distinct` where an exact count isn't required |
| Worker OOM on big queries                    | Raise `-Xmx` and pod limit; reduce query/join width                         |
| Cold-cluster latency                         | Keep the cluster warm; pre-warm before scheduled peaks                      |

## Cost guardrails

* Right-size `spec.trino.workers.replicas` to the real concurrency need. Idle workers cost compute.
* Trino supports query timeouts (`query.max-run-time`, `query.max-execution-time`, `query.client.timeout`) via `configProperties`, but these aren't in the validated default set. Setting any `configProperties` block replaces the working default, so introduce timeouts only after testing an explicit baseline. The validated guardrail today is the API-side `VULCAN_API_QUERY_TIMEOUT` (default 300s).
* Use `concurrencyPolicy: Forbid` for all scheduled workflows.
* Materialize hot, repeatedly-joined data into the lakehouse instead of re-federating every query.
* Keep `iceberg.max-partitions-per-writer` sane (e.g. 100) via `overideCatalogConfig` to avoid small-file explosions.
* Push filters/predicates to remote sources so Trino reads less.

## Deployment recipes

**Multi-depot federation with a per-catalog override** (mirrors the validated reference deploy: three depot catalogs, a per-catalog Iceberg override, default `config.properties`, and the only validated server change, `-Xmx4G`):

```yaml
spec:
  engine: trino
  depots:
    - dataos://abfsslhdepot?purpose=rw     # Iceberg (default catalog)
    - dataos://arabledepot?purpose=rw      # Postgres
    - dataos://snowflakevulcan?purpose=rw  # Snowflake
  trino:
    overideCatalogConfig:
      - name: abfsslhdepot
        properties:
          iceberg.max-partitions-per-writer: "100"
    coordinator:
      trinoServerConfig:
        # config.properties left to the stack default (validated working path)
        jvmConfig: |
          -server
          -Xmx4G
          -XX:+UseG1GC
          -XX:G1HeapRegionSize=32M
          -XX:+UseGCOverheadLimit
          -XX:+ExplicitGCInvokesConcurrent
          -XX:+HeapDumpOnOutOfMemoryError
        logProperties: |
          io.trino=INFO
    workers:
      replicas: 1
      trinoServerConfig:
        jvmConfig: |
          -server
          -Xmx4G
          -XX:+UseG1GC
          -XX:G1HeapRegionSize=32M
          -XX:+UseGCOverheadLimit
          -XX:+ExplicitGCInvokesConcurrent
          -XX:+HeapDumpOnOutOfMemoryError
        logProperties: |
          io.trino=INFO
```

**Secret-backed (non-depot) catalog:** Mount a catalog whose `.properties` come straight from a DataOS secret:

```yaml
spec:
  engine: trino
  depots:
    - dataos://lakehousedepot?purpose=rw
  trino:
    catalog:
      config:
        - "engineering:my-trino-catalog-secret"   # secret key/values → catalog .properties
```

**End-to-end: Postgres via secret catalog.** Use this when your Postgres source has no DataOS depot, you want fine-grained control over catalog properties, or you're connecting to an external Postgres not managed by DataOS.

*Step 1 - create the secret* (its keys are Trino catalog property names; the secret name becomes the catalog name):

```yaml
version: v1
kind: Secret
name: <your-catalog-secret>
layer: user
type: key-value
data:
  connector.name: postgresql
  connection-url: jdbc:postgresql://<host>:<port>/<database>
  connection-user: <username>
  connection-password: <password>
```

```bash
dataos-ctl resource apply -f secret.yaml
```

*Step 2 - `config.yaml`:*

```yaml
name: <your-dp-name>
display_name: "<Display Name>"
tenant: "<tenant>"
description: >
  <Description of the data product.>
discoverable: true
version: "1.0.0"
alignment: source_aligned
domain: "<domain>"

tags:
  - trino
  - postgres

users:
  - username: "<owner-username>"
    email: "<owner-email>"
    type: "OWNER"

gateways:
  default:
    connection:
      type: trino
      # Postgres connector rejects comment DDL — disable to prevent plan errors.
      register_comments: false

default_gateway: default

model_defaults:
  dialect: trino
  start: "<YYYY-MM-DD>"
  cron: "@daily"

linter:
  enabled: true
  rules:
    - ambiguousorinvalidcolumn
    - invalidselectstarexpansion
    - noambiguousprojections

notification_targets:
  - type: console
    notify_on:
      - apply_start
      - apply_end
      - run_start
      - run_end
      - apply_failure
      - run_failure
      - dq_start
      - dq_end

ignore_patterns:
  - "deployment.yaml"
```

*Step 3 - deploy resource (`deploy.yaml`):*

```yaml
version: v1alpha
type: vulcan
name: <your-dp-resource-name>
owner: <owner-username>
tags:
  - trino
  - postgres
description: >
  Managed Trino DP — Postgres via secret catalog.
spec:
  runAsUser: "<owner-username>"
  compute: <compute-pool>
  engine: trino
  repo:
    url: <your-repo-url>
    syncFlags:
      - "--ref=<branch>"
      - "--submodules=off"
    baseDir: <path/to/your/project>
    secret: <tenant>:<git-sync-secret>

  # No depots — Postgres catalog from trino.catalog.config only.
  # Secret keys are Trino catalog property names; secret name becomes the catalog name.
  trino:
    # Optional: append extra properties to the generated catalog .properties.
    overideCatalogConfig:
      - name: <your-catalog-secret>
        properties:
          case-insensitive-name-matching: "true"
          metadata.cache-ttl: "30s"

    catalog:
      config:
        - "<tenant>:<your-catalog-secret>"

    coordinator:
      trinoServerConfig:
        jvmConfig: |
          -server
          -Xmx4G
          -XX:+UseG1GC
          -XX:G1HeapRegionSize=32M
          -XX:+UseGCOverheadLimit
          -XX:+ExplicitGCInvokesConcurrent
          -XX:+HeapDumpOnOutOfMemoryError
          -Djdk.attach.allowAttachSelf=true
          --enable-native-access=ALL-UNNAMED
          --add-opens=java.base/java.nio=ALL-UNNAMED
          --sun-misc-unsafe-memory-access=allow
        logProperties: |
          io.trino=INFO
    workers:
      replicas: 1
      trinoServerConfig:
        jvmConfig: |
          -server
          -Xmx4G
          -XX:+UseG1GC
          -XX:G1HeapRegionSize=32M
          -XX:+UseGCOverheadLimit
          -XX:+ExplicitGCInvokesConcurrent
          -XX:+HeapDumpOnOutOfMemoryError
          -Djdk.attach.allowAttachSelf=true
          --enable-native-access=ALL-UNNAMED
          --add-opens=java.base/java.nio=ALL-UNNAMED
          --sun-misc-unsafe-memory-access=allow
        logProperties: |
          io.trino=INFO

  # Set TRINO_CATALOG explicitly so the gateway resolves the correct catalog.
  # With only a secret catalog and no depots, this can be inferred — but be explicit.
  use:
    projection:
      projections:
        envVars:
          - key: TRINO_CATALOG
            template: "<your-catalog-secret>"

  workflow:
    logLevel: INFO
    schedule:
      crons:
        - "0 */6 * * *"
      endOn: "<YYYY-MM-DDT00:00:00-00:00>"
      timezone: "UTC"
      concurrencyPolicy: Forbid
    resource:
      request: { cpu: "1000m", memory: "2Gi" }
      limit:   { cpu: "2000m", memory: "4Gi" }
    migrate:
      command: [vulcan]
      arguments: ["--log-to-stdout", "migrate"]
    plan:
      command: [vulcan]
      arguments: ["--log-to-stdout", "plan", "--auto-apply"]
    run:
      command: [vulcan]
      arguments: ["--log-to-stdout", "run"]

  api:
    replicas: 1
    logLevel: INFO
    resource:
      request: { cpu: "1000m", memory: "2Gi" }
      limit:   { cpu: "2000m", memory: "4Gi" }
```

*Steps 4–9 - models, semantics, DQ, metrics, tests* (fill in your own schema/columns):

SEED model:

```sql
MODEL (
  name <your-catalog-secret>.<raw-schema>.<table-name>,
  kind SEED (
    path '../../seeds/<table>.csv'
  ),
  description '<Description>',
  columns (
    <column_1> <TYPE>,
    <column_2> <TYPE>
    -- ...
  ),
  grain <primary-key-column>
);
```

FULL model:

```sql
MODEL (
  name <your-catalog-secret>.<sales-schema>.<model-name>,
  kind FULL,
  cron '@daily',
  grain <primary-key-column>,
  description '<Description>',
  columns (
    <column_1> <TYPE>,
    <column_2> <TYPE>
    -- ...
  )
);

SELECT
  <column_1>,
  <column_2>
  -- transformation expressions
FROM <your-catalog-secret>.<raw-schema>.<source-table>
```

> Partitioned incrementals (`INCREMENTAL_BY_PARTITION`) aren't supported through the Postgres connector. Use `FULL` or `VIEW` for models on Postgres-backed catalogs.

Semantic model:

```yaml
kind: semantic
name: <semantic-model-name>
depends_on: <your-catalog-secret>.<schema>.<model-name>
description: <Description>

dimensions:
  - name: <primary-key>
    behavior:
      type: identifier
  - <dimension-1>
  - name: <dimension-2>
    behavior:
      type: categorical

measures:
  - name: <measure-name>
    type: count
    expression: "{<semantic-model-name>.<primary-key>}"
    description: <Description>

segments:
  - name: <segment-name>
    expression: "{<semantic-model-name>.<dimension>} = '<value>'"
```

Data quality:

```yaml
kind: dq
name: <model-name>_dq
depends_on: <your-catalog-secret>.<schema>.<model-name>

rules:
  - failed rows:
      name: <rule-name>
      dimension: validity
      fail query: |
        SELECT <key-column>, <checked-column>
        FROM <your-catalog-secret>.<schema>.<model-name>
        WHERE <condition>
      samples limit: 10
      description: "<What this rule checks>"
```

Business metric:

```yaml
kind: metric
name: <metric-name>
measure: <semantic-model-name>.<measure-name>
ts: <semantic-model-name>.<timestamp-dimension>
granularity: day    # day | week | month
description: <Description>
```

Model test:

```yaml
<test-name>:
  model: <your-catalog-secret>.<schema>.<model-name>
  description: >
    <What this test validates>

  inputs:
    <your-catalog-secret>.<raw-schema>.<source-table>:
      rows:
        - <column_1>: <value>
          <column_2>: <value>

  outputs:
    query:
      rows:
        - <output-column-1>: <expected-value>
          <output-column-2>: <expected-value>
```

*Step 10 - apply and verify:*

```bash
# 1. Apply the secret (once)
dataos-ctl resource apply -f secret.yaml

# 2. Apply the deploy resource
dataos-ctl resource apply -f deployment.yaml

# 3. Verify the cluster formed
dataos-ctl resource -t vulcan -n <resource-name> get runtime -r

# 4. Check coordinator logs — confirm catalog loaded
dataos-ctl resource -t vulcan -n <resource-name> logs -l 2000 2>&1 \
  | grep -E "Loading catalog <your-catalog-secret>|connector.name"
```

After a successful `plan` run, models are available via REST, GraphQL, and MySQL-wire under the Vulcan API.

## Failure modes specific to a dedicated cluster

| Symptom                                               | Likely cause                                                     | Fix                                                                             |
| ----------------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Cluster forms with 0 workers / workers OOM at startup | Unmanaged JVM heap exceeds pod limit                             | Set `-Xmx` to fit the pod limit in `trinoServerConfig.jvmConfig` for both roles |
| Workers never register / cluster won't form           | `node.environment` mismatch or `coordinator=true` on workers     | Make `node.environment` identical; `coordinator=false` on workers               |
| Duplicate node errors across worker pods              | `node.id` set in worker `nodeProperties`                         | Remove `node.id` from workers. It auto-generates per pod                        |
| Custom plugin/UDF not loaded                          | JAR at `dependencies/java/` root, not in a subdir                | Move it into `dependencies/java/<plugin>/`                                      |
| Query OOM-killed                                      | JVM heap too small for the query (or memory caps set above heap) | Raise `-Xmx` and pod limit; keep any memory caps below heap                     |
| `configProperties` override breaks the cluster        | Duplicated discovery/shared-secret, or missing role lines        | Don't duplicate injected lines; include role lines                              |
| plan/run pod hangs at startup                         | Vulcan trino-ready gate waiting for the configured worker count  | Check worker pods are `active`; verify `TRINO_EXPECTED_WORKERS`                 |
| Snowflake/Databricks catalog won't mount              | Credentials missing from the secret                              | Check `SHOW CATALOGS` in coordinator logs                                       |

**Recovery procedures**

| Situation                               | Procedure                                                                |
| --------------------------------------- | ------------------------------------------------------------------------ |
| Cluster formed with 0 workers           | Set `-Xmx`, re-apply, confirm workers `active` in `system.runtime.nodes` |
| Time-range run failed mid-window        | Re-run the affected window; partition overwrite is safe to replay        |
| Source temporarily unavailable          | Re-run after the source recovers; declare retries at the workflow level  |
| Catalog credentials rotated             | Update depot/secret, re-apply (regenerates `.properties`)                |
| Workflow halted because `endOn` expired | Update `endOn` and re-apply                                              |
| Bad data materialized to Iceberg        | Use Iceberg snapshot rollback (coordinate with platform SRE)             |

**Where to look**

| Symptom                       | Where                   | How                                                                                                 |
| ----------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------- |
| Local cluster startup         | Docker compose services | `docker compose -f docker/docker-compose.yml logs <svc>`                                            |
| Coordinator/cluster health    | Coordinator service     | `dataos-ctl resource -t service -n <name>-trino logs -l 100`                                        |
| Worker health / OOM           | Workers service         | `dataos-ctl resource -t service -n <name>-trino-workers logs -l 100`                                |
| Rendered Trino config (proof) | Inside the pods         | `dataos-ctl resource -t service -n <name>-trino exec -- cat /usr/lib/trino/etc/config.properties`   |
| Plan/run failures             | Workflow pods           | `dataos-ctl resource -t vulcan -n <name> logs -l 500`                                               |
| SQL smoke                     | Coordinator             | `dataos-ctl resource -t service -n <name>-trino exec -- trino --execute "SELECT 1"`                 |
| Cluster membership            | Coordinator             | `trino --catalog system --schema runtime --execute "SELECT node_id, coordinator, state FROM nodes"` |

## Version compatibility

This combination is confirmed to work together:

| Vulcan image                     | Trino server image    | DataOS compute     | Storage / sources                    |
| -------------------------------- | --------------------- | ------------------ | ------------------------------------ |
| `tmdcio/vulcan-trino:0.228.1.26` | `tmdcio/trino:5.1.12` | Trino-capable pool | Iceberg (ABFSS), Postgres, Snowflake |

## Full implementation example

Use this complete Data Product as a production-shaped starting point.

{% file src="/files/bJx758F25A4Cy047VdPm" %}

## Related

* [Trino](/references/v1/engine-guide/trino-1.md) for rules shared across all cluster shapes.
* [Minerva](/references/v1/engine-guide/trino-1/minerva-1.md) for the shared DataOS-managed cluster.
* [External cluster](/references/v1/engine-guide/trino-1/external-trino.md) for a bring-your-own Trino-compatible cluster.
* [Trino properties reference](https://trino.io/docs/current/admin/properties.html)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/v1/engine-guide/trino-1/dedicated-cluster.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.