> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/v1/engine-guide/trino-1/external-trino/starburst.md). # Starburst Starburst Galaxy is a managed, Trino-compatible query service. you can build Vulcan's full productization layer: models, semantics, metrics, data quality, versioned lifecycle, and auto-generated REST/GraphQL/SQL endpoints. It runs on infrastructure your team already operates and funds, with zero duplication. Vulcan does not provision or manage Galaxy compute. {% hint style="info" %} Engine adapter type: `trino`. Model dialect: `trino`. Tested Vulcan package: `vulcan[trino]`, matching the `tmdcio/vulcan-trino` image line. Tested cluster: Starburst Galaxy custom cluster (`floki`) with `tpch` for reads and `s3lhdepot` Iceberg/S3 lakehouse for writes. Reference project: `trino-usecase/external-starburst`. Last updated: July 2026. {% endhint %} {% hint style="warning" %} This manual is based on one fully worked reference project and the Trino engine manual. See Scope and validation status for tested scenarios and current limits. {% endhint %} ### Scope and validation status This first-pass manual reflects one Galaxy cluster and the `external-starburst` project. Use the validated patterns for production. Test excluded scenarios in your environment before adopting them. | Area | Validated scope | Current limit | | -------------------- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | Connection | Basic authentication over HTTPS against a Galaxy custom cluster (`floki`). | JWT, mTLS with client certificates, and Kerberos are not included in the current scope. | | Catalogs | `tpch` as a read-only source and `s3lhdepot` Iceberg/S3 as a materialization target. | Hive, Delta Lake, and JDBC catalogs are not included in the current scope. | | Workload behavior | End-to-end `vulcan plan` and `vulcan run`, models, semantics, metrics, DQ, audits, and profiling. | Multi-tenant concurrency and multi-cluster deployments are not included in the current scope. | | Performance and cost | No benchmark claims. | Compute-sizing tables, concurrency curves, and cost baselines require Galaxy-specific measurement. | Starburst uses the Trino adapter. Shared model, semantic, metric, DQ, lineage, endpoint, and MCP behavior follows the [Trino](/references/v1/engine-guide/trino-1.md) manual. ## When to use Starburst Choose Starburst when your team already operates Starburst Galaxy. Use Vulcan for governed transformations, data quality, and semantic serving without managing Galaxy compute through DataOS. Starburst uses the `type: trino` gateway against the cluster's HTTPS endpoint. It always follows the external Trino deployment shape: Vulcan connects to an existing cluster and never provisions Starburst compute. ## Starburst at a glance Use this section to confirm the deployment and catalog pattern. ### Deployment model | | Starburst Galaxy via Vulcan | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | Resource type | `vulcan` (`spec.engine: trino`, **no** `spec.trino` block, **no** `spec.depots`) | | Cluster ownership | Pre-existing Starburst Galaxy cluster (created/managed in the Galaxy console, outside DataOS) | | Gateway connection | `type: trino`, `basic` auth, `https`, port `443` | | Catalogs | Already attached to the cluster in Galaxy (console → Clusters → Catalogs tab). Vulcan does not create them | | Materialization target | Whichever attached catalog you set as `catalog` in the gateway (must be writable, e.g. an Iceberg/S3 catalog) | | Credentials | Injected via a DataOS `Secret` + `spec.use.projection`, not `spec.depots` | | Best for | Teams already running Starburst Galaxy who want Vulcan's modeling/DQ/semantic layer on top, without Vulcan owning any compute | This is architecturally identical to the "External / Minerva Trino" row in the Trino manual's engine-shape table. Starburst is just one more Trino-compatible endpoint behind the same gateway type. ### Catalog pattern A single query can federate across every catalog attached to the cluster, but only some of those catalogs can be written to: * **`tpch`** : Trino's built-in synthetic data generator. An excellent free read-only source for learning and testing; it has no `CREATE SCHEMA`/`CREATE TABLE` support, so nothing can ever be materialized into it. * **A writable lakehouse catalog** (e.g. `s3lhdepot`, Iceberg-on-S3): This is where Vulcan's FULL/VIEW/INCREMENTAL models actually land. Every model in the reference project reads from `tpch.tiny.*` and writes into `s3lhdepot..*`, federated in one query, on one cluster. Swap `tpch` for a real source catalog (Postgres, Snowflake, another lakehouse, and so on) once you're past the hello-world stage. The pattern doesn't change. ### Engine characteristics | Item | Value | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Engine adapter type | `trino` (same adapter as every other Trino project) | | Model dialect | `trino` | | Auth validated | Basic auth over HTTPS | | Authentication and catalog scope | See [Scope and validation status](#scope-and-validation-status) | | Identifier casing | **Lowercase** (Trino default) | | Schema resolution | Always from the model's 3-part name (`catalog.schema.table`). The gateway has **no schema-level config key**, unlike Snowflake/Databricks, which default a schema | | Supported model kinds | VIEW, FULL, SEED, INCREMENTAL\_BY\_TIME\_RANGE, INCREMENTAL\_BY\_PARTITION | | Local state store | DuckDB (`./.state.db`) | | Known Iceberg gotcha | `INCREMENTAL_BY_PARTITION` needs `catalog_type_overrides` set explicitly. See Configure `catalog_type_overrides` under Starburst-specific behavior | ### Validated capabilities * A clean end-to-end `vulcan plan` / `vulcan run` against a live Galaxy cluster: staging → intermediate → mart, semantics, metrics, DQ, audits, and profiling all pass. * A federated read (`tpch`) and write (`s3lhdepot`) in the same query, on the same cluster. * Negative-path errors (bad auth, bad host, missing catalog, missing schema, unsupported table property), all captured against a real cluster. See Failure modes and troubleshooting. {% hint style="info" %} Review [Scope and validation status](#scope-and-validation-status) before using authentication, catalog, or workload patterns outside this guide's scope. {% endhint %} ## Connection options Use this section to configure authentication, prerequisites, permissions, and local requirements. ### Authentication Methods | Method | Status against Starburst | | ------------------------------------ | ------------------------------------------------------------------ | | Basic (username/password over HTTPS) | ✅ Validated | | JWT | Supported by the `type: trino` gateway; outside this guide's scope | | mTLS with client certificates | Supported by the gateway; outside this guide's scope | | Kerberos | Supported by the gateway; outside this guide's scope | {% hint style="info" %} Galaxy usernames take a `/` shape. Omitting the `/` suffix produces a confusing `404 User not found` rather than an auth error (see Failure modes and troubleshooting, F3).
{% endhint %} ### Starburst-side permissions and grants Unlike managed Trino, there's no compute pool or stack to install. Starburst Galaxy already exists as a SaaS product. Here's what you need before touching Vulcan: | Requirement | Where to get it | | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A Galaxy account and a cluster | Galaxy console → Clusters. Note the cluster's **Trino endpoint host** (`.trino.galaxy.`), not the console URL. | | At least one read source catalog attached | Console → Clusters → your cluster → Catalogs tab. `tpch` ships built in for a zero-setup hello world. | | At least one **writable** catalog attached | An Iceberg/S3 (or equivalent) catalog you've created and attached to the same cluster. This is your materialization target. | | A user with the right role suffix | Galaxy usernames used by the Trino wire protocol are `/` (e.g. `you@example.com/accountadmin`), not just the email. See Failure modes and troubleshooting, F3. | | A password (or other supported credential) | Galaxy console → your account settings, or an issued cluster password. | {% hint style="warning" %} Both catalogs must be attached to the same cluster. Vulcan cannot federate across two separate Starburst clusters in one query. {% endhint %} ### DataOS permissions These permissions must be provisioned before you create and deploy a Data Product: | **Permission** | **What it allows** | **Granted by** | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | **Tenant access** | Build and manage Data Products within your tenant. | Tenant Admin | | **`roles:id:data-dev`** *(or your tenant's equivalent role tag)* | Create and apply Vulcan resources. Verify your role using `dataos-ctl user get` and check the **TAGS** column. | Tenant Admin | | Permission to create a `Secret` resource | Store `STARBURST_*` credentials | Tenant Admin | |

depot🇷🇼
depot:r:

| Not applicable. You do not create a Depot to connect with Starburst. Vulcan receives credentials through a DataOS `Secret` and environment variables. | N/A | | **Can Use** *(Compute, Depots, Secrets)* | Use the compute pool and `Secret` resources required by the deployment. Resource access is deny-by-default. | Tenant Admin, Data Admin, or the resource owner | | **`git-sync` Secret** | Allows Vulcan to pull model code from Git using credentials stored in a `Secret`. Request a **git-sync Secret**, not raw Git credentials. You also need **Can Use** permission on the secret. | Tenant Admin, Data Admin, or the Secret owner (with **Can Use** granted) | ### Local development requirement | Requirement | Version | | -------------------------- | -------- | | Python (local development) | **3.10** | ```bash python --version # must be 3.10.x ``` ### **Before you begin** **Ensure:** * [ ] Galaxy cluster exists; Trino endpoint host noted * [ ] `tpch` (or your read source) attached to the cluster * [ ] A writable Iceberg/S3 lakehouse catalog attached to the **same** cluster * [ ] Username includes the `/` suffix * [ ] Password/credential in hand * [ ] Vulcan wheel installed locally (`vulcan[trino]`) * [ ] `MAX_FORK_WORKERS=1` set if developing on macOS (see Failure modes and troubleshooting) **Before you ship checklist (Data Product developer)** * [ ] `.env` / real credentials are **not** committed (gitignored, and never pasted into `config.yaml` directly) * [ ] `catalog_type_overrides` set for any Iceberg catalog used with `INCREMENTAL_BY_PARTITION` (see Starburst-specific behavior) * [ ] All identifiers lowercase; every model uses the fully-qualified 3-part name * [ ] `.artifacts/` (or wherever the deploy secret manifest lives) is gitignored before the first commit ## Local development Connect your workstation directly to the existing Galaxy cluster. Starburst requires no local Docker cluster because Galaxy already provides the Trino compute. ### Install Vulcan ```bash pip install "/path/to/vulcan--py3-none-any.whl[trino]" vulcan --version ``` Same wheel and extra as any other Trino project. Starburst needs no separate engine package. ### Set up credentials Keep real credentials out of `config.yaml` and out of git entirely. Use a `.env` file, gitignored: ```bash # .env (gitignored — never committed) STARBURST_HOST=.trino.galaxy. STARBURST_PORT=443 STARBURST_USER=@/ STARBURST_PASSWORD= STARBURST_CATALOG= STARBURST_HTTP_SCHEME=https STARBURST_METHOD=basic ``` ```bash set -a && source .env && set +a ``` {% hint style="warning" %} Never hardcode the password directly into `config.yaml` "just for local debugging." Even a file you intend to revert before committing is one `git add .` away from leaking a live credential. Use `.env` and `env_var(...)` templating from the start (see Configure config.yaml, next). {% endhint %} ### Configure config.yaml ```yaml gateways: trino: connection: type: trino host: "{{ env_var('STARBURST_HOST') }}" user: "{{ env_var('STARBURST_USER') }}" catalog: "{{ env_var('STARBURST_CATALOG', 'tpch') }}" port: "{{ env_var('STARBURST_PORT', '443') }}" http_scheme: "{{ env_var('STARBURST_HTTP_SCHEME', 'https') }}" method: "{{ env_var('STARBURST_METHOD', 'basic') }}" password: "{{ env_var('STARBURST_PASSWORD') }}" verify: true # Forces Iceberg-specific DDL (see Starburst-specific behavior) — Starburst's # live catalog-type detection doesn't always resolve reliably. catalog_type_overrides: : iceberg state_connection: type: duckdb # local dev only — drop this block for deployment (see Production deployment) database: ./.state.db default_gateway: trino model_defaults: dialect: trino start: cron: '@daily' linter: enabled: true rules: - ambiguousorinvalidcolumn - invalidselectstarexpansion - noambiguousprojections ignore_patterns: - "*-deploy.yaml" ``` {% hint style="warning" %} **`catalog_type_overrides` matters more here than on managed Trino.** Vulcan's Trino adapter normally auto-detects catalog type via `system.metadata.catalogs` to decide between Iceberg's `PARTITIONING` property and the Hive-style `partitioned_by` property. Against Starburst Galaxy, that live detection didn't reliably come back `'iceberg'`, which breaks any `INCREMENTAL_BY_PARTITION` model with `TrinoUserError: Table property 'partitioned_by' not supported for Iceberg tables`. Set the override explicitly for every Iceberg-backed catalog you materialize into. See Starburst-specific behavior and Troubleshooting, F1. {% endhint %} {% hint style="info" %} **No schema field.** The `type: trino` gateway takes `catalog` only. There is no default-schema connection key. Schema always comes from each model's fully-qualified `catalog.schema.table` name. {% endhint %} ### Build a hello-world project The minimum-viable Data Product against Starburst is one seed-backed VIEW over the free `tpch.tiny` sample data, materialized into your writable lakehouse catalog. No local infrastructure is needed beyond the Vulcan CLI and a network path to your Galaxy cluster. Confirm the two catalogs first. Swap in your real catalog name for `s3lhdepot` below: ```bash vulcan fetchdf "select * from tpch.tiny.customer limit 5" vulcan fetchdf "show schemas from s3lhdepot" ``` {% hint style="warning" %} **Column-naming gotcha.** Trino's built-in `tpch` connector does **not** use the prefixed TPC-H DDL column names other engines' sample datasets use (`c_custkey`, `o_orderkey`, and so on). Its columns are unprefixed: `custkey`, `orderkey`, `name`, `nationkey`, and so on. Run `DESCRIBE tpch.tiny.` before assuming a column name. {% endhint %} **`models/staging/stg_customer.sql`:** A typed, renamed pass-through: ```sql MODEL ( name s3lhdepot.staging.stg_customer, kind VIEW, owner 'you', grain customer_id, tags ('staging', 'tpch', 'customer'), description 'Renamed/typed pass-through of tpch.tiny.customer.', columns ( customer_id BIGINT, customer_name VARCHAR, nation_key BIGINT, account_balance DECIMAL(12, 2), market_segment VARCHAR ), assertions ( not_null(columns := (customer_id)) ) ); SELECT custkey AS customer_id, name AS customer_name, nationkey AS nation_key, CAST(acctbal AS DECIMAL(12, 2)) AS account_balance, mktsegment AS market_segment FROM tpch.tiny.customer; ``` **`models/staging/stg_orders.sql`:** A second source table, joined later: ```sql MODEL ( name s3lhdepot.staging.stg_orders, kind VIEW, owner 'you', grain order_id, tags ('staging', 'tpch', 'orders'), columns ( order_id BIGINT, customer_id BIGINT, total_price DECIMAL(12, 2), order_date DATE ), assertions ( not_null(columns := (order_id, customer_id)) ) ); SELECT orderkey AS order_id, custkey AS customer_id, CAST(totalprice AS DECIMAL(12, 2)) AS total_price, orderdate AS order_date FROM tpch.tiny.orders; ``` **`models/mart/mart_customer_orders.sql`:** The first materialized (`FULL`) table, actually written into your writable catalog: ```sql MODEL ( name s3lhdepot.mart.mart_customer_orders, kind FULL, owner 'you', cron '@daily', grain order_id, tags ('mart', 'orders'), description 'Orders joined to their customer, one row per order.', columns ( order_id BIGINT, customer_id BIGINT, customer_name VARCHAR, market_segment VARCHAR, total_price DECIMAL(12, 2), order_date DATE ), assertions ( not_null(columns := (order_id, customer_id)), unique_values(columns := order_id) ) ); SELECT o.order_id, o.customer_id, c.customer_name, c.market_segment, o.total_price, o.order_date FROM s3lhdepot.staging.stg_orders AS o INNER JOIN s3lhdepot.staging.stg_customer AS c ON o.customer_id = c.customer_id; ``` **`models/semantics/orders.yml`** : expose it to the semantic layer: ```yaml kind: semantic name: orders depends_on: s3lhdepot.mart.mart_customer_orders dimensions: - order_id - customer_id - market_segment - order_date measures: - name: total_revenue type: sum expression: "{orders.total_price}" - name: order_count type: count_distinct expression: "{orders.order_id}" ``` **`models/metrics/daily_revenue.yml`**: ```yaml kind: metric name: daily_revenue measure: orders.total_revenue ts: orders.order_date granularity: day ``` ### Validate your connection ```bash vulcan migrate # initializes local Vulcan state (DuckDB, ./.state.db) vulcan plan # dry run — expect stg_customer, stg_orders (VIEW) + mart_customer_orders (FULL) staged vulcan run # materializes mart_customer_orders into s3lhdepot.mart ``` Sanity check after `run`: ```bash vulcan fetchdf "select * from s3lhdepot.mart.mart_customer_orders order by total_price desc limit 10" ``` If `vulcan plan` succeeds and `run` materializes without error, your local setup is complete. Common failures at this step are covered in Failure modes and troubleshooting. ### Extend the hello-world project The reference project (`external-starburst`) takes this exact pattern and extends it to a full layered pipeline: 8 staging views, a `SEED`-backed lookup table, 4 intermediate models, 9 mart models (dimensions, a fact table, daily/monthly rollups, a window-function rank, both incremental kinds), 4 semantic models, 2 metrics, 10 DQ suites, 7 custom audits, and 3 unit tests. All run against the same `tpch` + `s3lhdepot` pair on one Galaxy cluster. See `trino-usecase/external-starburst/README.md` for the fully worked version of every layer summarized under Starburst-specific behavior, below; it's the canonical example this manual is drawn from. ## Production deployment Deploy after local `plan` and `run` succeed. Starburst has **no depot** and **no `spec.trino` block** in the deploy resource. Credentials flow entirely through a DataOS `Secret` and `spec.use.projection`. {% stepper %} {% step %} ## Create the secret Plain-text values; DataOS base64-encodes them automatically on `ds apply` (the same convention as Kubernetes Secrets): ```yaml # starburst-secret.yaml name: starburst-sec version: v2alpha type: secret layer: user secret: type: key-value data: STARBURST_HOST: .trino.galaxy. STARBURST_PORT: "443" STARBURST_USER: @/ STARBURST_PASSWORD: STARBURST_CATALOG: STARBURST_HTTP_SCHEME: https STARBURST_METHOD: basic ``` ```bash ds resource apply -f starburst-secret.yaml --disable-interpolation ``` {% hint style="danger" %} **Don't pre-encode these values.** The deploy resource's `| base64_decode` template exists to *undo* the automatic encoding `ds apply` already applies to a `type: secret` resource. Encoding again double-encodes and corrupts the value. This bit the reference project for real: its first deployed run failed with `error 401: access_denied` on every `CREATE SCHEMA`, root-caused to exactly this double-encoding (two of the seven values contain `$`/`@`, outside the base64 alphabet, so the corruption wasn't silent-but-plausible; it errored). Store plain text; let the platform and the deploy manifest's decode step handle the rest. {% endhint %} {% hint style="danger" %} **Never commit the secret manifest.** Add its directory to `.gitignore` immediately. A real credential in an untracked file is one `git add .` away from being committed. {% endhint %} {% endstep %} {% step %} ## Reference it from the deploy resource ```yaml version: v1alpha type: vulcan name: description: "" tags: [trino, starburst, starburst-galaxy] spec: compute: # check with: dataos-ctl resource -t compute get -a engine: trino repo: url: syncFlags: - '--ref=' - '--submodules=off' baseDir: secretId: : # optional: only if the repo is private use: projection: secrets: - id: :starburst-sec contextAlias: sbg projections: envVars: - key: STARBURST_HOST template: "{{ secrets['sbg'].STARBURST_HOST | base64_decode }}" - key: STARBURST_PORT template: "{{ secrets['sbg'].STARBURST_PORT | base64_decode }}" - key: STARBURST_USER template: "{{ secrets['sbg'].STARBURST_USER | base64_decode }}" - key: STARBURST_CATALOG template: "{{ secrets['sbg'].STARBURST_CATALOG | base64_decode }}" - key: STARBURST_HTTP_SCHEME template: "{{ secrets['sbg'].STARBURST_HTTP_SCHEME | base64_decode }}" - key: STARBURST_METHOD template: "{{ secrets['sbg'].STARBURST_METHOD | base64_decode }}" - key: STARBURST_PASSWORD template: "{{ secrets['sbg'].STARBURST_PASSWORD | base64_decode }}" workflow: schedule: crons: ['0 */6 * * *'] endOn: '' timezone: 'UTC' concurrencyPolicy: Forbid resource: request: { cpu: "200m", memory: "512Mi" } limit: { cpu: "1000m", memory: "1Gi" } plan: command: [vulcan] arguments: ["--log-to-stdout", "plan", "--auto-apply"] run: command: [vulcan] arguments: ["--log-to-stdout", "run"] api: replicas: 1 resource: request: { cpu: "200m", memory: "512Mi" } limit: { cpu: "1000m", memory: "1Gi" } ``` {% hint style="warning" %} **Don't hand-write this file.** Generate the starting point with `vulcan create_deploy_yaml --output -deploy.yaml` from the project root, then fill in the placeholders. It confirms the shape for you: no `spec.trino` block and no `spec.depots`. This tells you Vulcan is connecting to an existing cluster rather than provisioning one. {% endhint %} {% endstep %} {% step %} ## Confirm startup configuration The plan/run workflow pods read `STARBURST_*` from the environment at startup. If the secret doesn't exist yet, or the `contextAlias` in `spec.use.projection.secrets` doesn't match what `envVars` templates reference, every model fails at connection time before a single query runs. {% endstep %} {% endstepper %} ## Starburst-specific behavior Use this section for Galaxy-specific configuration. Shared Vulcan behavior follows the `type: trino` adapter and the Trino manual. {% hint style="info" %} **Architectural constraints specific to Starburst:** * Vulcan never provisions or manages Starburst compute. The cluster, its catalogs, and its scaling are entirely a Galaxy-console concern. * The gateway has no schema-level config key; schema always comes from the model's fully-qualified name. * Live catalog-type detection (Iceberg vs. Hive-style DDL) isn't reliable against Galaxy. Always set `catalog_type_overrides` explicitly for any catalog you run `INCREMENTAL_BY_PARTITION` against. {% endhint %} ### Configure `catalog_type_overrides` Vulcan picks between Iceberg's `PARTITIONING` table property and the Hive-style `partitioned_by` property by calling `get_catalog_type()`, which by default queries `system.metadata.catalogs` live. Against Starburst Galaxy, this didn't reliably resolve to `'iceberg'`, so the adapter silently fell back to the Hive-style property name, which Iceberg rejects outright: ``` TrinoUserError: Table property 'partitioned_by' not supported for Iceberg tables ``` The fix is to force it explicitly in `config.yaml`, on the gateway connection. The adapter checks this override *before* the live query: ```yaml gateways: trino: connection: catalog_type_overrides: s3lhdepot: iceberg ``` This only matters for `INCREMENTAL_BY_PARTITION` models. VIEW, FULL, and `INCREMENTAL_BY_TIME_RANGE` are unaffected. ### Shared Vulcan capabilities There are no Starburst-specific differences from the Trino manual. The same `kind: dq` / `AUDIT(...)` / `assertions (...)` / `kind: semantic` / `kind: metric` mechanics apply, and all execute as ordinary Trino SQL against the Galaxy cluster. See [Trino](/references/v1/engine-guide/trino-1.md) for the full reference; only the DDL quirk described under The `catalog_type_overrides` requirement, above, is Starburst-specific. ## Engine-native feature support Use this table to map Galaxy objects to their Vulcan patterns. | Starburst/Galaxy object | Vulcan pattern | Notes | | --------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | Catalog (any connector Galaxy supports) | Read via fully-qualified 3-part model names | Must already be attached to the target cluster in the Galaxy console | | Iceberg table | `kind FULL` / `kind VIEW` / `kind INCREMENTAL_BY_TIME_RANGE` / `kind INCREMENTAL_BY_PARTITION` | Set `catalog_type_overrides` for the partitioned kind (see Configure `catalog_type_overrides`) | | Role-based auth (`/`) | `user` field in the gateway connection | Role suffix required | | Cluster-level catalog access control | N/A; managed entirely in the Galaxy console | Attach/detach catalogs there, not via Vulcan | ## Operational boundaries Use this section to configure the workflow and API resources. Galaxy owns cluster sizing and scaling. {% hint style="info" %} Galaxy-specific compute sizing, concurrency limits, and API replica guidance require measurement. Use the Trino operational guidance as a starting point, then validate it for your workload. Vulcan sizes only the plan/run workflow pod and API track here. {% endhint %} ### Known operational limitations | Limitation | Workaround | | -------------------------------------- | --------------------------------------------------------------------------------- | | No schema-level connection config | Always use fully-qualified `catalog.schema.table` names | | Catalog-type auto-detection unreliable | Set `catalog_type_overrides` explicitly (see Configure `catalog_type_overrides`) | | Two catalogs must share a cluster | Attach both source and target catalogs to the same Galaxy cluster before planning | | `tpch` has no write support | Use it for reads only; materialize elsewhere | ## Performance and cost Use Galaxy telemetry to establish workload-specific performance and cost baselines. See [Scope and validation status](#scope-and-validation-status) for current benchmark limits. For general Trino guardrails, including query timeouts and attribution tags, see [Trino](/references/v1/engine-guide/trino-1.md). ## Troubleshooting Use these entries to match observed failures with their likely causes and fixes. ### Failure modes | # | Symptom | Likely cause | Fix | | --- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | F1 | `TrinoUserError: Table property 'partitioned_by' not supported for Iceberg tables` | Catalog-type auto-detection didn't resolve to `'iceberg'` | Set `catalog_type_overrides: { : iceberg }` (see Configure `catalog_type_overrides`) | | F2 | `error 401: access_denied No description` | Bad password, **or** (in production only) a double-base64-encoded secret value | Locally: check the password. In production: confirm the secret was stored as **plain text** (see Production deployment). `ds apply` already encodes it once | | F3 | `error 404: User not found` | Username missing the `/` suffix | Use `/`, e.g. `you@example.com/accountadmin` | | F4 | `error 404: Destination not found` | Unreachable/misspelled `STARBURST_HOST` | Use the cluster's **Trino endpoint** host (`*.trino.galaxy.`), not the Galaxy console URL | | F5 | `TrinoUserError(name=CATALOG_NOT_FOUND, message="Catalog '' not found")` | Catalog not attached to the cluster `STARBURST_HOST` points at | Galaxy console → Clusters → your cluster → Catalogs tab → attach it | | F6 | `TrinoUserError(name=SCHEMA_NOT_FOUND, message="Schema '' does not exist")` | Schema doesn't exist in an otherwise-valid, attached catalog | Create the schema, or fix the model's schema name | | F7 | `COLUMN_NOT_FOUND` on every `tpch.tiny.*` staging model | Assumed prefixed TPC-H column names (`c_custkey`, and so on) | Use `DESCRIBE tpch.tiny.
`. Columns are unprefixed (`custkey`, `orderkey`, and so on) | | F8 | `BlockingIOError` during `vulcan plan` on macOS | Vulcan's default fork-worker concurrency | `export MAX_FORK_WORKERS=1` | | F9 | `Cannot specify both 'audits' and 'assertions' - use one or the other` | Tried a separate `audits (...)` model clause | There's only one clause, `assertions (...)`. It takes built-in checks and custom audit names (bare name only) together in the same list | | F10 | Editing `.env` doesn't fix an already-broken shell session | A variable was already `export`ed from an earlier `source .env` | `unset` the specific vars, re-`source .env`, and `echo` them back before retrying | | # | Symptom | Likely cause | Fix | | --- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | F1 | `TrinoUserError: Table property 'partitioned_by' not supported for Iceberg tables` | Catalog-type auto-detection didn't resolve to `'iceberg'` | Set `catalog_type_overrides: { : iceberg }` (see Configure `catalog_type_overrides`) | | F2 | `error 401: access_denied No description` | Bad password, **or** (in production only) a double-base64-encoded secret value | Locally: check the password. In production: confirm the secret was stored as **plain text** (see Production deployment). `ds apply` already encodes it once | | F3 | `error 404: User not found` | Username missing the `/` suffix | Use `/`, e.g. `you@example.com/accountadmin` | | F4 | `error 404: Destination not found` | Unreachable/misspelled `STARBURST_HOST` | Use the cluster's **Trino endpoint** host (`*.trino.galaxy.`), not the Galaxy console URL | | F5 | `TrinoUserError(name=CATALOG_NOT_FOUND, message="Catalog '' not found")` | Catalog not attached to the cluster `STARBURST_HOST` points at | Galaxy console → Clusters → your cluster → Catalogs tab → attach it | | F6 | `TrinoUserError(name=SCHEMA_NOT_FOUND, message="Schema '' does not exist")` | Schema doesn't exist in an otherwise-valid, attached catalog | Create the schema, or fix the model's schema name | | F7 | `COLUMN_NOT_FOUND` on every `tpch.tiny.*` staging model | Assumed prefixed TPC-H column names (`c_custkey`, and so on) | Use `DESCRIBE tpch.tiny.
`. Columns are unprefixed (`custkey`, `orderkey`, and so on) | | F8 | `BlockingIOError` during `vulcan plan` on macOS | Vulcan's default fork-worker concurrency | `export MAX_FORK_WORKERS=1` | | F9 | `Cannot specify both 'audits' and 'assertions' - use one or the other` | Tried a separate `audits (...)` model clause | There's only one clause, `assertions (...)`. It takes built-in checks and custom audit names (bare name only) together in the same list | | F10 | Editing `.env` doesn't fix an already-broken shell session | A variable was already `export`ed from an earlier `source .env` | `unset` the specific vars, re-`source .env`, and `echo` them back before retrying | {% hint style="info" %} **Failure clarity pattern worth knowing:** errors caught by Trino's own query engine *after* auth succeeds (catalog/schema resolution, unsupported table properties, including F1, F5, and F6) come back as Trino's structured `USER_ERROR` taxonomy: a specific error name, the exact object named, and a traceable `query_id`. Errors at the Galaxy gateway/HTTP layer *before* a query reaches Trino (auth and connection, F2 through F4) come back as generic HTTP status codes with thinner messages instead. They correctly signal the failure category and never echo the password or token back, but they don't name the specific field that was wrong. {% endhint %} ### Health-check queries ```bash vulcan fetchdf "select 1" # connection + auth vulcan fetchdf "show catalogs" # what's attached to this cluster vulcan fetchdf "show schemas from " # confirm the write target is reachable vulcan fetchdf "describe tpch.tiny.customer" # confirm real column names before writing staging SQL ``` ### Recovery procedures * **Stale local state after model changes:** Delete `.state.db` and `.cache/` (both gitignored) and re-run `vulcan migrate`. * **Suspect a stuck shell env var:** `unset STARBURST_HOST STARBURST_USER STARBURST_PASSWORD` (and so on), re-`source .env`, and `echo` each one back before retrying (see F10, above). ## Deployment recipes ### Hello-world / learning project Use Hello world through Growing past hello world, above, as-is: `tpch` read plus a single writable lakehouse catalog, scheduled `@daily` or run manually, `spec.workflow.resource` at the small defaults shown under Production deployment. ### Layered federated Data Product (staging → intermediate → mart) Follow the reference project's shape end to end: `trino-usecase/external-starburst`. Read its README's "Model layers" and "Governance & deployment" sections for the full worked pattern (macros, a SEED-backed lookup, semantics with joins, metrics, DQ suites per mart model, custom audits, unit tests, custom linter rules). ## Full implementation example `trino-usecase/external-starburst` is the complete, validated Starburst Galaxy example behind this guide. It connects to an existing Galaxy cluster, reads from `tpch`, and materializes models into a writable Iceberg catalog. Use it when you need a production-shaped starting point. It includes: * Layered staging, intermediate, and mart models. * Semantic models, metrics, data-quality suites, audits, and unit tests. * Local and DataOS deployment configuration with secret-based credentials. Start with the hello-world models in this guide. Then adapt the reference project’s catalog names, credentials, and model layers for your environment. {% file src="/files/0v3sxk1fsfJ7DHdBVk8T" %} ## Related * [Trino](/references/v1/engine-guide/trino-1.md): The full companion manual for models, semantics, metrics, DQ, lineage, endpoints, and MCP tools. * [Dedicated cluster](/references/v1/engine-guide/trino-1/dedicated-cluster.md): For a Vulcan-provisioned Trino cluster. * `trino-usecase/external-starburst/README.md`: The complete reference project, including its resolved-error changelog. * [Engine guide overview](/references/v1/engine-guide/engine-guide.md): Cross-engine guidance used by every engine manual. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/v1/engine-guide/trino-1/external-trino/starburst.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.