> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/spark-1/spark-lakehouse-limitations.md).

# Spark lakehouse limitations

Understand the supported boundaries and operational limitations of Vulcan Spark workloads on the DataOS Lakehouse and Apache Iceberg.

This page describes limitations specific to the Vulcan Spark deployment path on DataOS Lakehouse and Apache Iceberg. These boundaries do not describe every capability of Apache Spark itself.

## Limitations summary

| Area                      | Limitation                                                                                    | Recommended approach                                                                    |
| ------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| Views                     | `VIEW` and materialized views are not supported                                               | Use `FULL` or an incremental table-backed model                                         |
| Virtual data environments | VDE is not supported when the configured engine is Spark                                      | Use separate environments and deployment controls supported by the Spark path           |
| Transactions              | Spark does not provide a transaction across the complete model lifecycle                      | Design pre-statements, writes, audits, and post-statements for partial-failure recovery |
| Concurrent writers        | Multiple writers to the same Iceberg table can produce optimistic commit conflicts            | Stagger writers and coordinate all workflows that target the table                      |
| Partition overwrite       | The scope of `INSERT OVERWRITE` depends on table partitioning and Spark/catalog configuration | Test overwrite behavior with controlled partitions before production                    |
| Unique-key updates        | Later evaluations require an Iceberg target that supports Spark `MERGE INTO`                  | Use `table_format iceberg` when the provider is not already established                 |
| Streaming                 | Streaming pipelines are outside Vulcan's model lifecycle                                      | Build streaming ingestion separately and expose its tables to Vulcan                    |
| Query latency             | Spark does not guarantee consistently low API query latency                                   | Pre-aggregate common queries or use a lower-latency serving engine                      |
| API autoscaling           | `spec.api.autoscale` is not supported for the Spark engine                                    | Size the Spark API driver and use Spark dynamic allocation when the runtime supports it |

## Views

The Spark deployment path does not support `VIEW` models or materialized views. Use `FULL` for a complete table rebuild or an incremental model for bounded table updates.

## Catalog boundaries

Spark can read from and write to multiple configured catalogs. Read and write behavior depends on the catalog implementation, connector capabilities, credentials, and source permissions.

Use fully qualified model names to select the output catalog. Validate the selected model kind and write operation against that catalog before production.

See [Connect and configure Spark](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md) for Lakehouse and custom catalog setup.

## Transactions and recovery

The Spark adapter does not support transactions across multiple statements. If a model lifecycle performs several operations and a later operation fails, Spark does not automatically roll back earlier successful operations.

Design recovery for:

* `pre_statements` that succeed before the model write fails.
* A successful write followed by a failed audit or `post_statement`.
* Seed loading failures. Vulcan attempts to drop an incomplete target when a batch fails; recovery is required if that cleanup also fails.
* Failed or conflicting Iceberg commits.

Iceberg write-audit-publish (WAP) applies only when explicitly enabled and supported by the target. It is not a general rollback mechanism.

Test recovery procedures in a non-production environment. See [Troubleshooting](https://v2.dataos.info/references/engine-guide/spark-1/troubleshooting.md) for runtime diagnosis and recovery guidance.

## Concurrent Iceberg writers

Iceberg uses optimistic concurrency. Two workflows, applications, or external writers that update the same table can conflict during commit.

`concurrencyPolicy: Forbid` prevents overlapping runs of one schedule. It does not coordinate:

* Separate Vulcan resources.
* Different schedules.
* External Spark applications.
* Other Iceberg writers.

Coordinate writers at the workload level and stagger jobs that target the same table.

## Overwrite and MERGE behavior

Time-range and partition incremental models use Spark overwrite operations. The effective replacement scope depends on:

* The target table's partition specification.
* The data selected by the model.
* Spark partition-overwrite settings.
* Catalog and Iceberg behavior.

Unique-key incremental models use `MERGE INTO` after initialization. The target must use a provider that supports Spark row-level merge operations. Validate insert, update, delete, and unmatched-row behavior with controlled source changes.

See [Spark materialization behavior](https://v2.dataos.info/references/engine-guide/spark-1/materialization-behaviour.md) for model-kind details.

## Spark API scaling

The Spark API runs as one SparkApplication with one driver. `spec.api.autoscale` is therefore not supported for the Spark engine.

When enabled and supported by the deployed runtime, Spark dynamic allocation can change executor count. It requires compatible shuffle tracking or external shuffle-service configuration. It does not scale:

* The API driver.
* API replicas.
* Request concurrency.

Size the driver for expected concurrent queries and use executor allocation settings for distributed Spark work. See [Limits and performance](https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md).

## Related

* [Spark overview](https://v2.dataos.info/references/engine-guide/spark-1.md)
* [Connect and configure Spark](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md)
* [Spark materialization behavior](https://v2.dataos.info/references/engine-guide/spark-1/materialization-behaviour.md)
* [Limits and performance](https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md)
* [Troubleshooting](https://v2.dataos.info/references/engine-guide/spark-1/troubleshooting.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/spark-1/spark-lakehouse-limitations.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
