For the complete documentation index, see llms.txt. This page is also available as Markdown.

Amazon Athena

Configure Amazon Athena as the engine for your Data Product.

Amazon Athena is a serverless, Presto/Trino-family query engine over data registered in the AWS Glue catalog. Use the athena adapter when your data already lands in S3 as Parquet or Iceberg, is registered in Glue, and you want pay-per-scan SQL rather than a cluster to manage. This page covers the connection. For IAM details, materialization internals, cost guardrails, and the full reference, see the Athena engine manual in References.

Athena has no SQL GRANT statements. Access is enforced through AWS IAM (and optionally Lake Formation) on Glue databases and tables and the underlying S3 prefixes, not through database-side grants.

Before you start

  • An Athena workgroup, a Glue database, an S3 prefix for query results (s3_staging_dir), and an S3 prefix for model data (s3_warehouse_location), all in the same AWS region. Vulcan doesn't create S3 buckets, workgroups, or the Glue account setup: provision those out of band.

  • An IAM principal for Vulcan whose keys belong to the same account as the target Glue databases (or a trusted cross-account catalog).

  • The Vulcan CLI with the athena extra: pip install "./vulcan-0.228.1.32-py3-none-any.whl[athena]". Athena has no local container, so local development points at real AWS resources (a dev workgroup, Glue database, and S3 prefixes).

  • DataOS access: a roles:id:data-dev-equivalent role tag, depot:rw:<athena-depot-name> (or depot:r for read-only) when the deployment uses a depot, Can Use on Compute/Depots/Secrets, an Athena instance secret, a git-sync Secret, and the vulcan-athena stack or approved compute.

Permissions

Three roles, each with a distinct scope — don't collapse them into one: an AWS/data platform admin (creates S3 buckets, Glue databases, workgroups, and IAM policies), the Vulcan service role (runs Athena queries, creates Glue tables, reads and writes S3), and a consumer role (read-only Athena, Glue, and S3 on Data Product outputs).

Area
Actions for the Vulcan service principal (illustrative)

Athena

athena:StartQueryExecution, athena:GetQueryExecution, athena:GetQueryResults, athena:StopQueryExecution, athena:GetWorkGroup

Glue

glue:GetDatabase, glue:GetDatabases, glue:CreateDatabase, glue:GetTable, glue:GetTables, glue:CreateTable, glue:UpdateTable, glue:DeleteTable

S3 staging

s3:GetObject, s3:PutObject, s3:ListBucket on the s3_staging_dir bucket/prefix

S3 warehouse

The same actions on the s3_warehouse_location prefix

Tighten ARNs to your account. Give consumers glue:Get* on target databases and tables, athena:StartQueryExecution plus results read (optionally on a separate consumer workgroup), and s3:GetObject / ListBucket on published prefixes only. Verify from the Athena query editor, in the same workgroup as your config, with SELECT current_user; and SHOW DATABASES;.

Connection options

Option
Required
Description

type

Yes

Must be athena

aws_access_key_id

Yes

AWS access key for the Vulcan service principal

aws_secret_access_key

Yes

AWS secret key

region_name

Yes

AWS region. Must match the workgroup and buckets

work_group

Yes

Athena workgroup to run queries in

s3_staging_dir

Yes

Athena query output location

schema_name

Yes

Glue database (schema)

catalog_name

No

Glue catalog. Defaults to awsdatacatalog

s3_warehouse_location

Yes

Base s3:// prefix for managed table data

Authentication methods

  • IAM access key and secret (local and CI): set aws_access_key_id and aws_secret_access_key, or use a role chain per your org standard. Prefer IRSA or instance roles in production over long-lived keys where possible, and rotate keys on a schedule.

  • Instance secret (DataOS): keep the AWS_* and ATHENA_* values in an Athena instance secret and project them into the Vulcan resource's environment.

Example

Never commit the keys. Keep them in .env locally. By default the Athena adapter persists Vulcan's own state in Iceberg tables inside the warehouse; to isolate several projects on shared infrastructure, add a Postgres state_connection and a state_schema instead. Validate with vulcan migrate (initializes state) and vulcan plan.

Production: project the instance secret

In production, the typical pattern projects the instance secret straight into the Vulcan resource's environment, so the gateway above stays the same and reads its values from env vars. The projection lives under deployment: in config.yaml, and vulcan create_deploy_yaml renders it into the resource:

When you want depot-managed access purposes instead, use an Athena depot through the vulcan-athena stack: list the depot under depots in the stack spec, and the stack projects a per-depot Athena connection for you. For local development, keep the direct type: athena connection shown above.

Bind each purpose to its real use: rw for the Vulcan workflow (read sources, create and update Glue tables, write S3), scan for the metadata scanner, and query for read-only consumers. Apply the instance secret and the git-sync Secret first, then deploy from config.yaml with vulcan create_deploy_yaml.

Supported model kinds

FULL, VIEW, INCREMENTAL_BY_TIME_RANGE, INCREMENTAL_BY_UNIQUE_KEY, and INCREMENTAL_BY_PARTITION. FULL rebuilds the table (insert-overwrite per adapter strategy), VIEW compiles to CREATE VIEW, INCREMENTAL_BY_TIME_RANGE runs a windowed delete/overwrite plus insert, INCREMENTAL_BY_UNIQUE_KEY uses MERGE or an equivalent where supported, and INCREMENTAL_BY_PARTITION overwrites the partition. Python models run on the adapter's pandas path; validate them against your Vulcan image first. For the full mapping, see the Athena engine manual.

Athena has no strong REPLACE TABLE semantics and no multi-statement transactions, so a pipeline that fails mid-run may need an idempotent re-run. Match semantic identifiers to the physical column casing, which is often lowercase snake_case. The Athena parser rejects some comment patterns: avoid a leading /* ... */ block comment on statements.

Declare cross-database sources (for example retail_source.orders_ext) in inputs.yaml, or bootstrap external Parquet tables with before_all. Wrap lifecycle DDL in pre_statements / post_statements guarded with @IF(@runtime_stage = 'evaluating', ...), or vulcan plan executes it. Unit tests run on DuckDB locally even when the gateway is Athena; use audits and DQ checks for warehouse-side validation.

Limits and performance

Athena bills primarily for data scanned, so reduce scanned bytes before raising limits. Set a bytes-scanned limit on every workgroup (lower in development to catch unpartitioned queries early), set a workgroup query timeout, and partition or time-filter daily incrementals aggressively. Workgroups have concurrent query limits: align concurrent_tasks and API replicas with your workgroup's quotas, and keep api.resources.limits.memory at 3 GiB for large result sets.

Use separate workgroups for scheduled builds and interactive or API traffic when they compete, and per environment for cost attribution. The semantic API adds roughly 1.5–3 seconds of overhead on top of scan time, and every agent-driven MCP data query scans bytes like a dashboard query does.

Troubleshooting

Symptom
Likely cause
Fix

S3 Access Denied on query start

Staging or warehouse IAM

Fix bucket policies and IAM for both S3 URIs.

SCHEMA_NOT_FOUND / database missing

Glue database not created

Run the bootstrap, or create the database in Glue.

InvalidRequestException on workgroup

Wrong workgroup name or region

Align ATHENA_WORK_GROUP and AWS_REGION.

s3_warehouse_location was expected

Option omitted from the connection

Set ATHENA_S3_WAREHOUSE_LOCATION.

Secret projection empty on DataOS

Secret not applied, or wrong id

Verify <tenant>:athena-secret and the base64-encoded keys.

Parse error on commented SQL

Athena comment limitations

Remove leading block comments on statements.

vulcan plan runs DDL

Lifecycle DDL not guarded

Wrap it in @IF(@runtime_stage = 'evaluating', ...).

High bill after a test run

Full table scan

Add predicates; use columnar Parquet.

For the full failure-mode table, health-check queries, and recovery procedures, see Troubleshooting in the Athena engine manual.

Once vulcan info succeeds, continue to Assets.

Last updated

Was this helpful?