For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs.
Skip to content

Dedicated Trino

Ship a Trino cluster as part of the Data Product. Vulcan provisions and owns one cluster per product.

Use this path when the Trino cluster should be part of the data product deployment itself. You don't manage the cluster separately from the product: a single vulcan trino resource provisions the coordinator and workers, generates catalogs from your depots or DataOS-provisioned secret catalogs, and runs the plan and run workflows.

This page covers the connection and deploy shape. For tuning, failure modes, and local development, see the Trino engine manual in References.

How it works

One vulcan trino resource creates 5 cooperating resources:

<name>-trino           Trino coordinator (replicas: 1)
<name>-trino-workers   Trino workers
<name>-plan            vulcan migrate, then plan --auto-apply (waits for the cluster)
<name>-run             vulcan run (scheduled; depends on plan)
<name>-api             Vulcan API for consumers

Check their status with:

dataos-ctl resource -t vulcan -n <name> get runtime -r

Vulcan generates catalogs automatically: one per depot in deployment.depots[], plus one per secret catalog you define.

Prerequisites

Request these from your platform team:

  • A Trino-capable compute pool.

  • The tenant-level vulcan-trino stack (vulcan+trino:1.0) installed.

  • One depot per source you read. The first depot must be a writable lakehouse depot; it's the materialization target.

  • Tenant secrets vulcan-state-connection (Postgres) and vulcan-object-store-connection (S3), and a git-sync secret for your repo.

  • Python 3.10 to 3.13 locally for the Vulcan CLI.

Confirm your depots are visible with dataos-ctl resource -t depot get -a.

Permissions

There are no Trino SQL grants to configure here: Vulcan connects to the internal coordinator over the Kubernetes network with method: no-auth. Access control lives entirely at each depot's underlying data source: whatever the depot's credentials permit there is what Trino can do through that catalog.

Depot type
Trino connector
Permission guide

lakehouse

iceberg (REST)

DataOS-native; no external grants

bigquery

bigquery

Governed by IAM roles on the depot's service account

databricks

iceberg (REST, read-only)

delta_lake

delta_lake

AWS IAM (S3 + Glue/Hive Metastore)

config.yaml

The gateway catalog is templated from an environment variable the stack injects, so the same file works locally and in production:

Tenant can be set in config.yaml or via the environment

tenant is a valid config.yaml key, and domain becomes required once tenant is set. Alternatively, set DATAOS_TENANT_ID in your shell for local runs; the platform injects it in production, where it overrides tenant. Vulcan refuses to load the project without a tenant set either way.

Cluster settings for this product live under deployment.trino: in this same file (see The deploy configuration below). Keep ignore_patterns: ["*-deploy.yaml"] so that a rendered deploy manifest saved inside the project is never parsed as a model.

Models: always use three-part names

Use fully-qualified catalog.schema.name for the model name and every source you read. The default catalog is the first secret catalog if any exist, otherwise the first depot. Unqualified names are a common cause of models materializing in the wrong place, so always spell out all 3 parts:

The deploy configuration

There's no separate manifest to hand-author for the Trino cluster. Coordinator and worker settings live under a deployment.trino: block, inside the same config.yaml you already have, alongside deployment.repo, deployment.compute, and the other keys every engine uses. See References → Vulcan → Deployment for the full deployment: walkthrough; this section covers only the Trino-specific parts.

The fields you'll change most often are deployment.depots (order matters: the first one is the default catalog and write target), deployment.trino.coordinator.resource, and deployment.trino.workers:

Size coordinator.resource and workers.resource to fit the compute pool; an undersized coordinator or worker limit is the most common deployment failure. TRINO_CATALOG must match a real catalog name.

Each deployment.trino.catalog.config entry is a "<tenant>:<secret-name>" reference to a DataOS secret whose key/value pairs are Trino catalog properties; the secret name becomes the catalog name. Secret catalogs take priority over depot catalogs as the default catalog, which is why the example pins TRINO_CATALOG to the lakehouse depot. Omit catalog if every source has a depot.

Set the JVM heap explicitly for both coordinator and workers, sized to the pod memory limit — an unmanaged heap is the most common cause of workers that OOM or never register. 4G is the validated -Xmx for a 4Gi pod limit; size yours to your own. If Postgres is the gateway catalog, set register_comments: false on the connection — the Postgres connector rejects the DDL Vulcan uses to register column comments.

Local development

Local development runs a real Trino coordinator and worker via Docker Compose (mounted at /usr/lib/trino/etc, Trino's real config path — a different mount path fails silently), plus a local Postgres container Trino federates against. Point config.yaml at the local coordinator (host: localhost, method: no-auth) with a DuckDB state_connection, add ignore_patterns: ["*-deploy.yaml"] so the deploy manifest is never parsed as a model, then validate with vulcan migrate && vulcan plan && vulcan run. See the Trino engine manual for the full Docker Compose setup.

Troubleshooting

Symptom
Likely cause
Fix

Cluster forms with 0 workers / workers OOM at startup

Unmanaged JVM heap exceeds pod limit

Set -Xmx to fit the pod limit for both coordinator and workers.

Workers never register / cluster won't form

node.environment mismatch, or coordinator=true on a worker

Make node.environment identical on both roles; never set node.id on workers.

Custom plugin/UDF not loaded

JAR at dependencies/java/ root, not in a subdirectory

Move it into dependencies/java/<plugin>/.

Postgres catalog: vulcan plan fails on comment DDL

register_comments not disabled

Set register_comments: false on the gateway connection.

For the full failure-mode table, recovery procedures, and log locations, see Troubleshooting in the Trino engine manual.

Apply and verify

Deploying a dedicated Trino product uses the same single flow as every other engine: render the manifest from config.yaml and apply it; no separate dataos-ctl resource apply -f trino-deploy.yaml step is required.

Confirm the cluster is up by looking for SERVER STARTED in the coordinator logs. The Trino engine manual covers common issues (workers out of memory, wrong-catalog materialization, schedules expiring silently) and their fixes.

With the cluster running, continue to Assets.

Last updated

Was this helpful?