> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/spark-1/troubleshooting.md).

# Troubleshooting

Diagnose Vulcan Spark catalog, dependency, execution, and Iceberg failures.

Use this guide when a Vulcan Spark project fails during deployment, planning, model execution, or API queries. Start with the failing DataOS component, read its complete error, and change configuration only after identifying the cause.

## Locate the failure

Check the Resource status:

```bash
dataos-ctl resource get -t vulcan -n <name>
```

If the status is `error` or `error(QT)`, inspect the build:

```bash
dataos-ctl resource get -t vulcan -n <name> -d -b
```

Fix the build error and apply the Resource again. Do not rerun a quarantined build; only an apply clears `error(QT)`.

If the status is `active`, check the runtime state:

```bash
dataos-ctl resource get runtime -t vulcan -n <name>
```

Use the `SUB RESOURCE ID`, container-group name, and phase to locate the failure:

| Sub-resource  | Purpose                           | Healthy phase                              |
| ------------- | --------------------------------- | ------------------------------------------ |
| `vulcan-plan` | Plans and applies project changes | `completed`                                |
| `vulcan-run`  | Executes scheduled model runs     | `completed`, or `running` during execution |
| `vulcan-api`  | Serves API queries                | `running`                                  |

A `failing` plan or run usually requires logs from the `main` container. If `main` never starts, inspect the init container shown in the runtime output.

## Read DataOS logs

Copy the container-group and container names from the runtime output:

```bash
dataos-ctl resource log \
  -t vulcan \
  -n <name> \
  --container-group <container-group-name> \
  -c <container-name> \
  -l 10000
```

Use the component that matches the symptom:

| Failure                                          | Logs to inspect                                            |
| ------------------------------------------------ | ---------------------------------------------------------- |
| Git sync, Secret projection, or startup          | Init container                                             |
| Plan, model loading, catalog, or scheduled run   | `main` container in the plan or run group                  |
| API or semantic query                            | Primary container in the `vulcan-api` group                |
| Executor OOM, failed task, shuffle fetch, or UDF | Spark executor logs in the Spark UI                        |
| Iceberg commit failure                           | Spark driver logs for the affected plan, run, or API query |

For complete DataOS log commands and container naming, see [Monitor a Vulcan Data Product](https://v2.dataos.info/references/resources/vulcan/monitor-and-delete.md).

## Catalog and Lakehouse failures

| Error or symptom                                       | Check                                                                               | Resolution                                                                             |
| ------------------------------------------------------ | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| Catalog name cannot be resolved                        | The Lakehouse Depot is attached and the SQL uses the Depot name as the catalog      | Correct `deployment.depots` or `spec.depots`, then use `<catalog>.<namespace>.<table>` |
| Iceberg REST request returns unauthorized or forbidden | `lh_access_token` and catalog namespace access                                      | Update the Lakehouse Depot Secret or namespace permissions                             |
| Object-store request returns access denied             | S3, GCS, or ABFSS access for the exact warehouse path                               | Grant matching storage access for source and target paths                              |
| Namespace or table is not found                        | Fully qualified name, catalog spelling, and namespace access                        | Correct the identifier and confirm the catalog can list the namespace                  |
| Catalog works locally but not in DataOS                | Depot attachment, projected Secrets, connector JARs, and DataOS Spark configuration | Compare the deployed catalog configuration with the local gateway                      |
| Custom catalog class cannot be loaded                  | Connector JAR is missing or incompatible                                            | Add the compatible JAR to `dependencies/java/` and redeploy                            |

Catalog and storage permissions are separate. A successful catalog login does not prove that Spark can read or write the underlying Iceberg files. See [Spark Lakehouse permissions](https://v2.dataos.info/references/engine-guide/spark-1/permissions-and-grants.md).

## Dependency failures

### JVM dependencies

Errors such as `ClassNotFoundException` and `NoClassDefFoundError` indicate that a required JVM class is unavailable or incompatible.

Check:

1. The JAR exists directly under `dependencies/java/`.
2. The JAR matches the deployed Spark, Scala, Java, Iceberg, and connector versions.
3. Required transitive JARs are present.
4. The same class is not supplied by conflicting JAR versions.
5. Driver and executor classpaths include the dependency.

After changing JARs, redeploy the resource so Git sync and the Spark classpath use the new artifact.

### Python dependencies

For `ImportError` or `ModuleNotFoundError`:

1. Confirm the wheel exists under `dependencies/python/`.
2. Confirm the package supports the Python version used by the deployed runtime.
3. Include required transitive Python dependencies.
4. Redeploy before rerunning the model.

See [Add project dependencies](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md#add-project-dependencies).

## Model and Iceberg write failures

| Error or symptom                                | Check                                                            | Resolution                                                                                                                            |
| ----------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `MERGE INTO` is unsupported                     | Target table provider and catalog capabilities                   | Use an Iceberg target that supports Spark row-level merge operations                                                                  |
| Unique-key model fails after initialization     | Target provider, unique key, source duplicates, and merge plan   | Correct the target or input data, then test insert and update cases with controlled rows                                              |
| Partition overwrite changes an unexpected range | Target partition specification and Spark overwrite configuration | Stop the run and validate overwrite scope with controlled partitions                                                                  |
| Iceberg commit conflict                         | Another workflow or external writer updated the same table       | Allow the competing write to finish, inspect the current table state, and rerun only after confirming the failed commit did not apply |
| Schema evolution fails                          | Source/target types and catalog alter-table support              | Make the schema change compatible with the target and rerun the plan                                                                  |
| Write is denied                                 | Target namespace and object-storage permissions                  | Grant catalog and storage write access to target paths                                                                                |

`concurrencyPolicy: Forbid` prevents overlapping runs of one schedule. It does not coordinate separate Vulcan resources, API writes, or external Iceberg writers. Stagger all known writers to the same target.

## Driver and executor failures

### Driver out of memory

Check driver logs for heap exhaustion, long garbage-collection pauses, large plans, catalog metadata pressure, or data collected into the driver process.

Possible corrections:

* Avoid collecting large results in Python.
* Reduce driver-side data and metadata work.
* Process a smaller backfill interval.
* Increase driver memory only when the logs confirm driver memory pressure.

### Executor out of memory

Check failed-task input size, scan partitions, shuffle partitions, and spill.

Possible corrections:

* Reduce input partition size.
* Increase shuffle partition count when shuffle tasks are too large.
* Address skewed keys that create one oversized task.
* Increase executor memory when the task size is appropriate but memory remains insufficient.

### One task runs much longer than the others

Compare task duration and input size in the Spark UI. If one key or partition dominates:

* Filter unnecessary rows before the join.
* Review join cardinality.
* Enable adaptive skew handling when supported.
* Change the data or query design when one key remains disproportionately large.

Adding executors does not split an already oversized task.

See [Spark limits and performance](https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md) before changing resource sizes or Spark properties.

## API query failures

When model runs succeed but an API query fails:

1. Confirm the `vulcan-api` component is `running`.
2. Read the API primary-container logs.
3. Confirm that the API Spark session has the required custom catalogs and classpath.
4. Check catalog and storage permissions for every table used by the query.
5. Inspect the Spark UI for failed stages, skew, spill, or executor loss.

Spark queries can include driver startup, catalog requests, object-storage reads, planning, shuffle, and result serialization. A timeout does not identify which stage is slow; use logs and the Spark UI before tuning.

## Recover safely

Before rerunning a failed write:

1. Identify the exact model, interval, partition, and target table.
2. Check whether an Iceberg commit succeeded before the process failed.
3. Inspect the current table snapshot and affected data.
4. Confirm that no other writer is still updating the table.
5. Fix the original error.
6. Rerun the smallest affected scope.
7. Validate the target data after the rerun.

Do not assume that every failed Spark task left the table unchanged. Verify the Iceberg commit state before replaying an incremental model or `MERGE INTO`.

## Rerun after fixing the cause

Rerun the plan:

```bash
dataos-ctl resource runtime re-run \
  -t vulcan \
  -n <name> \
  -v v1alpha \
  --sub-resource-id vulcan-plan
```

Rerun the scheduled model workflow:

```bash
dataos-ctl resource runtime re-run \
  -t vulcan \
  -n <name> \
  -v v1alpha \
  --sub-resource-id vulcan-run
```

Rerunning before fixing the cause only creates another failed execution.

## Related

* [Connect Spark to catalogs](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md)
* [Spark Lakehouse permissions](https://v2.dataos.info/references/engine-guide/spark-1/permissions-and-grants.md)
* [Spark materialization behavior](https://v2.dataos.info/references/engine-guide/spark-1/materialization-behaviour.md)
* [Spark limits and performance](https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md)
* [Monitor a Vulcan Data Product](https://v2.dataos.info/references/resources/vulcan/monitor-and-delete.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/spark-1/troubleshooting.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
