> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/athena/troubleshooting.md).

# Troubleshooting

Diagnose common Athena and Vulcan failures, then recover production workloads.

Start here when a Data Product on Athena fails to plan, deploy, or run. Identify the symptom, validate the environment, then recover safely.

{% hint style="info" %}
**Flagged for review:** the source manual carried two overlapping tables: "Operational limitations" (nine rows) and "Failure modes & troubleshooting" (fifteen rows), where all but one limitation row duplicated a failure-modes row (IAM/S3 access, missing database, missing `s3_warehouse_location`, external table issues, semantic casing, stale state, re-run cost, expired comments). They've been merged into the single table below, the pattern every other converted engine page uses, keeping the one row that wasn't a duplicate (expired `endOn`). Check the merged table against the original if you want to confirm nothing was dropped.
{% endhint %}

### Failure modes

| #   | Symptom                                | Likely cause                              | Fix                                                              |
| --- | -------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------- |
| F1  | S3 `Access Denied` on query start      | Staging or warehouse IAM                  | Fix bucket policies and IAM for both URIs                        |
| F2  | `SCHEMA_NOT_FOUND` / database missing  | Glue database not created                 | Run the bootstrap, or create the database in Glue                |
| F3  | `InvalidRequestException` on workgroup | Wrong name or region                      | Align `ATHENA_WORK_GROUP` and `AWS_REGION`                       |
| F4  | External table not found               | Missing `before_all`, or wrong `LOCATION` | Fix the S3 path; re-run plan with bootstrap                      |
| F5  | `s3_warehouse_location was expected`   | Config omitted                            | Set `ATHENA_S3_WAREHOUSE_LOCATION`                               |
| F6  | Semantic column not found              | Casing or rename drift                    | Match the semantic YAML to the model columns                     |
| F7  | Secret projection empty on DataOS      | Secret not applied, or wrong `id`         | Verify `<workspace>:<athena-secret>` and the base64-encoded keys |
| F8  | Plan runs heavy DDL                    | Unguarded lifecycle SQL                   | Use `@runtime_stage` guards                                      |
| F9  | Incremental re-scans all history       | Wrong `time_column`                       | Fix the model kind config                                        |
| F10 | API OOM                                | Memory limit too low                      | Raise to 3 GiB                                                   |
| F11 | Duplicate apply skipped                | Stale state                               | New `state_schema`, then `vulcan migrate`                        |
| F12 | Parse error on commented SQL           | Athena comment limitations                | Remove leading block comments on statements                      |
| F13 | Cross-account catalog empty            | Catalog misconfigured                     | Fix `catalog_name` or AWS federation                             |
| F14 | High bill after test                   | Full table scan                           | Add predicates; use columnar Parquet                             |
| F15 | Python model fails                     | Unsupported UDF or library on Athena      | Run the logic in SQL, or validate the Python path                |
| F16 | Workflow halted                        | Expired `endOn`                           | Update `endOn` in `deploy.yaml` and re-apply                     |

### Health-check queries

```sql
SELECT current_user;
SHOW DATABASES;
SHOW TABLES IN goldv4;

SELECT table_schema, table_name, table_type
FROM information_schema.tables
WHERE table_schema IN ('bronzev4', 'silverv4', 'goldv4', 'retail_source')
ORDER BY 1, 2
LIMIT 100;
```

### Recovery procedures

| Situation                     | Procedure                                                           |
| ----------------------------- | ------------------------------------------------------------------- |
| Incremental failed mid-window | Re-run the workflow; safe if the model is idempotent for the window |
| Table dropped manually        | `vulcan plan` to detect, then rebuild the model                     |
| Wrong Glue database           | Fix `ATHENA_SCHEMA_NAME`, reset the state schema, re-apply          |
| Leaked IAM keys               | Rotate the keys; update the secret and projection                   |
| Workflow halted (`endOn`)     | Extend `endOn` in `deploy.yaml`                                     |

### Next step

Use [Deployment recipes](https://v2.dataos.info/references/engine-guide/athena/recipes/deployment-recipes.md) to validate your gateway, grants, models, and deployment end to end.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/athena/troubleshooting.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
