> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/productize/connect-engine/athena.md).

# Amazon Athena

Configure Amazon Athena as the engine for your Data Product.

Amazon Athena is a serverless, Presto/Trino-family query engine over data registered in the AWS Glue catalog. Use the `athena` adapter when your data already lands in S3 as Parquet or Iceberg, is registered in Glue, and you want pay-per-scan SQL rather than a cluster to manage. This page covers the connection. For IAM details, materialization internals, cost guardrails, and the full reference, see the [Athena engine manual](https://v2.dataos.info/references/engine-guide/athena) in References.

{% hint style="info" %}
Athena has no SQL `GRANT` statements. Access is enforced through AWS IAM (and optionally Lake Formation) on Glue databases and tables and the underlying S3 prefixes, not through database-side grants.
{% endhint %}

## Before you start

* An Athena workgroup, a Glue database, an S3 prefix for query results (`s3_staging_dir`), and an S3 prefix for model data (`s3_warehouse_location`), all in the same AWS region. Vulcan doesn't create S3 buckets, workgroups, or the Glue account setup: provision those out of band.
* An IAM principal for Vulcan whose keys belong to the same account as the target Glue databases (or a trusted cross-account catalog).
* The Vulcan CLI with the `athena` extra: `pip install "./vulcan-0.228.1.32-py3-none-any.whl[athena]"`. Athena has no local container, so local development points at real AWS resources (a dev workgroup, Glue database, and S3 prefixes).
* DataOS access: a `roles:id:data-dev`-equivalent role tag, `depot:rw:<athena-depot-name>` (or `depot:r` for read-only) when the deployment uses a depot, **Can Use** on Compute/Depots/Secrets, an Athena instance secret, a git-sync Secret, and the `vulcan-athena` stack or approved compute.

## Permissions

Three roles, each with a distinct scope — don't collapse them into one: an AWS/data platform **admin** (creates S3 buckets, Glue databases, workgroups, and IAM policies), the **Vulcan service role** (runs Athena queries, creates Glue tables, reads and writes S3), and a **consumer role** (read-only Athena, Glue, and S3 on Data Product outputs).

| Area         | Actions for the Vulcan service principal (illustrative)                                                                                                       |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Athena       | `athena:StartQueryExecution`, `athena:GetQueryExecution`, `athena:GetQueryResults`, `athena:StopQueryExecution`, `athena:GetWorkGroup`                        |
| Glue         | `glue:GetDatabase`, `glue:GetDatabases`, `glue:CreateDatabase`, `glue:GetTable`, `glue:GetTables`, `glue:CreateTable`, `glue:UpdateTable`, `glue:DeleteTable` |
| S3 staging   | `s3:GetObject`, `s3:PutObject`, `s3:ListBucket` on the `s3_staging_dir` bucket/prefix                                                                         |
| S3 warehouse | The same actions on the `s3_warehouse_location` prefix                                                                                                        |

Tighten ARNs to your account. Give consumers `glue:Get*` on target databases and tables, `athena:StartQueryExecution` plus results read (optionally on a separate consumer workgroup), and `s3:GetObject` / `ListBucket` on published prefixes only. Verify from the Athena query editor, in the same workgroup as your config, with `SELECT current_user;` and `SHOW DATABASES;`.

## Connection options

| Option                  | Required | Description                                      |
| ----------------------- | :------: | ------------------------------------------------ |
| `type`                  |    Yes   | Must be `athena`                                 |
| `aws_access_key_id`     |    Yes   | AWS access key for the Vulcan service principal  |
| `aws_secret_access_key` |    Yes   | AWS secret key                                   |
| `region_name`           |    Yes   | AWS region. Must match the workgroup and buckets |
| `work_group`            |    Yes   | Athena workgroup to run queries in               |
| `s3_staging_dir`        |    Yes   | Athena query output location                     |
| `schema_name`           |    Yes   | Glue database (schema)                           |
| `catalog_name`          |    No    | Glue catalog. Defaults to `awsdatacatalog`       |
| `s3_warehouse_location` |    Yes   | Base `s3://` prefix for managed table data       |

### Authentication methods

* **IAM access key and secret** (local and CI): set `aws_access_key_id` and `aws_secret_access_key`, or use a role chain per your org standard. Prefer IRSA or instance roles in production over long-lived keys where possible, and rotate keys on a schedule.
* **Instance secret** (DataOS): keep the `AWS_*` and `ATHENA_*` values in an Athena instance secret and project them into the Vulcan resource's environment.

{% hint style="warning" %}
`s3_warehouse_location` is required for managed table materialization. Without it, the adapter can't resolve warehouse paths and fails with `s3_warehouse_location was expected`.
{% endhint %}

## Example

```yaml
gateways:
  athena:
    connection:
      type: athena
      aws_access_key_id: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
      aws_secret_access_key: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
      region_name: "{{ env_var('AWS_REGION') }}"
      work_group: "{{ env_var('ATHENA_WORK_GROUP') }}"
      s3_staging_dir: "{{ env_var('ATHENA_S3_STAGING_DIR') }}"
      schema_name: "{{ env_var('ATHENA_SCHEMA_NAME') }}"
      catalog_name: "{{ env_var('ATHENA_CATALOG_NAME', default='awsdatacatalog') }}"
      s3_warehouse_location: "{{ env_var('ATHENA_S3_WAREHOUSE_LOCATION') }}"
default_gateway: athena
model_defaults:
  dialect: athena
```

Never commit the keys. Keep them in `.env` locally. By default the Athena adapter persists Vulcan's own state in Iceberg tables inside the warehouse; to isolate several projects on shared infrastructure, add a Postgres `state_connection` and a `state_schema` instead. Validate with `vulcan migrate` (initializes state) and `vulcan plan`.

### Production: project the instance secret

In production, the typical pattern projects the instance secret straight into the Vulcan resource's environment, so the gateway above stays the same and reads its values from env vars. The projection lives under `deployment:` in `config.yaml`, and `vulcan create_deploy_yaml` renders it into the resource:

```yaml
deployment:
  use:
    secrets:
      - { id: <tenant>:athena-secret, context_alias: athena }
    env_vars:
      - key: AWS_ACCESS_KEY_ID
        template: "{{ secrets['athena'].AWS_ACCESS_KEY_ID | base64_decode }}"
      # ... AWS_SECRET_ACCESS_KEY, AWS_REGION, and the remaining ATHENA_* keys
```

When you want depot-managed access purposes instead, use an Athena depot through the `vulcan-athena` stack: list the depot under `depots` in the stack spec, and the stack projects a per-depot Athena `connection` for you. For local development, keep the direct `type: athena` connection shown above.

Bind each purpose to its real use: `rw` for the Vulcan workflow (read sources, create and update Glue tables, write S3), `scan` for the metadata scanner, and `query` for read-only consumers. Apply the instance secret and the git-sync Secret first, then deploy from `config.yaml` with `vulcan create_deploy_yaml`.

## Supported model kinds

`FULL`, `VIEW`, `INCREMENTAL_BY_TIME_RANGE`, `INCREMENTAL_BY_UNIQUE_KEY`, and `INCREMENTAL_BY_PARTITION`. `FULL` rebuilds the table (insert-overwrite per adapter strategy), `VIEW` compiles to `CREATE VIEW`, `INCREMENTAL_BY_TIME_RANGE` runs a windowed delete/overwrite plus insert, `INCREMENTAL_BY_UNIQUE_KEY` uses `MERGE` or an equivalent where supported, and `INCREMENTAL_BY_PARTITION` overwrites the partition. Python models run on the adapter's pandas path; validate them against your Vulcan image first. For the full mapping, see the [Athena engine manual](https://v2.dataos.info/references/engine-guide/athena).

> Athena has no strong `REPLACE TABLE` semantics and no multi-statement transactions, so a pipeline that fails mid-run may need an idempotent re-run. Match semantic identifiers to the physical column casing, which is often lowercase snake\_case. The Athena parser rejects some comment patterns: avoid a leading `/* ... */` block comment on statements.

Declare cross-database sources (for example `retail_source.orders_ext`) in `inputs.yaml`, or bootstrap external Parquet tables with `before_all`. Wrap lifecycle DDL in `pre_statements` / `post_statements` guarded with `@IF(@runtime_stage = 'evaluating', ...)`, or `vulcan plan` executes it. Unit tests run on DuckDB locally even when the gateway is Athena; use audits and DQ checks for warehouse-side validation.

## Limits and performance

Athena bills primarily for data scanned, so reduce scanned bytes before raising limits. Set a bytes-scanned limit on every workgroup (lower in development to catch unpartitioned queries early), set a workgroup query timeout, and partition or time-filter daily incrementals aggressively. Workgroups have concurrent query limits: align `concurrent_tasks` and API replicas with your workgroup's quotas, and keep `api.resources.limits.memory` at 3 GiB for large result sets.

Use separate workgroups for scheduled builds and interactive or API traffic when they compete, and per environment for cost attribution. The semantic API adds roughly 1.5–3 seconds of overhead on top of scan time, and every agent-driven MCP `data` query scans bytes like a dashboard query does.

## Troubleshooting

| Symptom                                | Likely cause                       | Fix                                                          |
| -------------------------------------- | ---------------------------------- | ------------------------------------------------------------ |
| S3 `Access Denied` on query start      | Staging or warehouse IAM           | Fix bucket policies and IAM for both S3 URIs.                |
| `SCHEMA_NOT_FOUND` / database missing  | Glue database not created          | Run the bootstrap, or create the database in Glue.           |
| `InvalidRequestException` on workgroup | Wrong workgroup name or region     | Align `ATHENA_WORK_GROUP` and `AWS_REGION`.                  |
| `s3_warehouse_location was expected`   | Option omitted from the connection | Set `ATHENA_S3_WAREHOUSE_LOCATION`.                          |
| Secret projection empty on DataOS      | Secret not applied, or wrong `id`  | Verify `<tenant>:athena-secret` and the base64-encoded keys. |
| Parse error on commented SQL           | Athena comment limitations         | Remove leading block comments on statements.                 |
| `vulcan plan` runs DDL                 | Lifecycle DDL not guarded          | Wrap it in `@IF(@runtime_stage = 'evaluating', ...)`.        |
| High bill after a test run             | Full table scan                    | Add predicates; use columnar Parquet.                        |

For the full failure-mode table, health-check queries, and recovery procedures, see [Troubleshooting](https://v2.dataos.info/references/engine-guide/athena/troubleshooting) in the Athena engine manual.

Once `vulcan info` succeeds, continue to [Assets](https://v2.dataos.info/build/productize/assets.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/build/productize/connect-engine/athena.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
