Amazon Athena
Configure Amazon Athena as the engine for your Data Product.
Amazon Athena is a serverless, Presto/Trino-family query engine over data registered in the AWS Glue catalog. Use the athena adapter when your data already lands in S3 as Parquet or Iceberg, is registered in Glue, and you want pay-per-scan SQL rather than a cluster to manage. This page covers the connection. For IAM details, materialization internals, cost guardrails, and the full reference, see the Athena engine manual in References.
Before you start
An Athena workgroup, a Glue database, an S3 prefix for query results (
s3_staging_dir), and an S3 prefix for model data (s3_warehouse_location), all in the same AWS region. Vulcan doesn't create S3 buckets, workgroups, or the Glue account setup: provision those out of band.An IAM principal for Vulcan whose keys belong to the same account as the target Glue databases (or a trusted cross-account catalog).
The Vulcan CLI with the
athenaextra:pip install "./vulcan-0.228.1.32-py3-none-any.whl[athena]". Athena has no local container, so local development points at real AWS resources (a dev workgroup, Glue database, and S3 prefixes).DataOS access: a
roles:id:data-dev-equivalent role tag,depot:rw:<athena-depot-name>(ordepot:rfor read-only) when the deployment uses a depot, Can Use on Compute/Depots/Secrets, an Athena instance secret, a git-sync Secret, and thevulcan-athenastack or approved compute.
Permissions
Three roles, each with a distinct scope — don't collapse them into one: an AWS/data platform admin (creates S3 buckets, Glue databases, workgroups, and IAM policies), the Vulcan service role (runs Athena queries, creates Glue tables, reads and writes S3), and a consumer role (read-only Athena, Glue, and S3 on Data Product outputs).
Athena
athena:StartQueryExecution, athena:GetQueryExecution, athena:GetQueryResults, athena:StopQueryExecution, athena:GetWorkGroup
Glue
glue:GetDatabase, glue:GetDatabases, glue:CreateDatabase, glue:GetTable, glue:GetTables, glue:CreateTable, glue:UpdateTable, glue:DeleteTable
S3 staging
s3:GetObject, s3:PutObject, s3:ListBucket on the s3_staging_dir bucket/prefix
S3 warehouse
The same actions on the s3_warehouse_location prefix
Tighten ARNs to your account. Give consumers glue:Get* on target databases and tables, athena:StartQueryExecution plus results read (optionally on a separate consumer workgroup), and s3:GetObject / ListBucket on published prefixes only. Verify from the Athena query editor, in the same workgroup as your config, with SELECT current_user; and SHOW DATABASES;.
Connection options
type
Yes
Must be athena
aws_access_key_id
Yes
AWS access key for the Vulcan service principal
aws_secret_access_key
Yes
AWS secret key
region_name
Yes
AWS region. Must match the workgroup and buckets
work_group
Yes
Athena workgroup to run queries in
s3_staging_dir
Yes
Athena query output location
schema_name
Yes
Glue database (schema)
catalog_name
No
Glue catalog. Defaults to awsdatacatalog
s3_warehouse_location
Yes
Base s3:// prefix for managed table data
Authentication methods
IAM access key and secret (local and CI): set
aws_access_key_idandaws_secret_access_key, or use a role chain per your org standard. Prefer IRSA or instance roles in production over long-lived keys where possible, and rotate keys on a schedule.Instance secret (DataOS): keep the
AWS_*andATHENA_*values in an Athena instance secret and project them into the Vulcan resource's environment.
Example
Never commit the keys. Keep them in .env locally. By default the Athena adapter persists Vulcan's own state in Iceberg tables inside the warehouse; to isolate several projects on shared infrastructure, add a Postgres state_connection and a state_schema instead. Validate with vulcan migrate (initializes state) and vulcan plan.
Production: project the instance secret
In production, the typical pattern projects the instance secret straight into the Vulcan resource's environment, so the gateway above stays the same and reads its values from env vars. The projection lives under deployment: in config.yaml, and vulcan create_deploy_yaml renders it into the resource:
When you want depot-managed access purposes instead, use an Athena depot through the vulcan-athena stack: list the depot under depots in the stack spec, and the stack projects a per-depot Athena connection for you. For local development, keep the direct type: athena connection shown above.
Bind each purpose to its real use: rw for the Vulcan workflow (read sources, create and update Glue tables, write S3), scan for the metadata scanner, and query for read-only consumers. Apply the instance secret and the git-sync Secret first, then deploy from config.yaml with vulcan create_deploy_yaml.
Supported model kinds
FULL, VIEW, INCREMENTAL_BY_TIME_RANGE, INCREMENTAL_BY_UNIQUE_KEY, and INCREMENTAL_BY_PARTITION. FULL rebuilds the table (insert-overwrite per adapter strategy), VIEW compiles to CREATE VIEW, INCREMENTAL_BY_TIME_RANGE runs a windowed delete/overwrite plus insert, INCREMENTAL_BY_UNIQUE_KEY uses MERGE or an equivalent where supported, and INCREMENTAL_BY_PARTITION overwrites the partition. Python models run on the adapter's pandas path; validate them against your Vulcan image first. For the full mapping, see the Athena engine manual.
Athena has no strong
REPLACE TABLEsemantics and no multi-statement transactions, so a pipeline that fails mid-run may need an idempotent re-run. Match semantic identifiers to the physical column casing, which is often lowercase snake_case. The Athena parser rejects some comment patterns: avoid a leading/* ... */block comment on statements.
Declare cross-database sources (for example retail_source.orders_ext) in inputs.yaml, or bootstrap external Parquet tables with before_all. Wrap lifecycle DDL in pre_statements / post_statements guarded with @IF(@runtime_stage = 'evaluating', ...), or vulcan plan executes it. Unit tests run on DuckDB locally even when the gateway is Athena; use audits and DQ checks for warehouse-side validation.
Limits and performance
Athena bills primarily for data scanned, so reduce scanned bytes before raising limits. Set a bytes-scanned limit on every workgroup (lower in development to catch unpartitioned queries early), set a workgroup query timeout, and partition or time-filter daily incrementals aggressively. Workgroups have concurrent query limits: align concurrent_tasks and API replicas with your workgroup's quotas, and keep api.resources.limits.memory at 3 GiB for large result sets.
Use separate workgroups for scheduled builds and interactive or API traffic when they compete, and per environment for cost attribution. The semantic API adds roughly 1.5–3 seconds of overhead on top of scan time, and every agent-driven MCP data query scans bytes like a dashboard query does.
Troubleshooting
S3 Access Denied on query start
Staging or warehouse IAM
Fix bucket policies and IAM for both S3 URIs.
SCHEMA_NOT_FOUND / database missing
Glue database not created
Run the bootstrap, or create the database in Glue.
InvalidRequestException on workgroup
Wrong workgroup name or region
Align ATHENA_WORK_GROUP and AWS_REGION.
s3_warehouse_location was expected
Option omitted from the connection
Set ATHENA_S3_WAREHOUSE_LOCATION.
Secret projection empty on DataOS
Secret not applied, or wrong id
Verify <tenant>:athena-secret and the base64-encoded keys.
Parse error on commented SQL
Athena comment limitations
Remove leading block comments on statements.
vulcan plan runs DDL
Lifecycle DDL not guarded
Wrap it in @IF(@runtime_stage = 'evaluating', ...).
High bill after a test run
Full table scan
Add predicates; use columnar Parquet.
For the full failure-mode table, health-check queries, and recovery procedures, see Troubleshooting in the Athena engine manual.
Once vulcan info succeeds, continue to Assets.
Last updated
Was this helpful?