> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md).

# Connect to Spark

Connect Vulcan Spark to DataOS Lakehouse catalogs and configure additional Spark catalogs for queries across multiple data sources.

Vulcan uses Spark catalogs to read source data and write model outputs. For a DataOS Lakehouse, attach its Depot to the Vulcan deployment. The `vulcan-spark` stack then generates the Iceberg REST catalog configuration automatically.

Use custom Spark catalog properties when a source is not exposed through a DataOS Lakehouse Depot. Spark can load multiple catalogs in the same session, allowing a model to read and join data across Lakehouse and other catalog implementations supported by Spark and their installed connectors.

For a Docker-based Spark, Iceberg, RustFS, and PostgreSQL environment, see [Local Spark development](https://v2.dataos.info/references/engine-guide/spark-1/local-development.md).

## Prerequisites

Before configuring catalogs, confirm that:

* A Spark-capable Compute resource is available.
* The `vulcan-spark` stack is available in the tenant.
* The target DataOS Lakehouse and Lakehouse Depot are active.
* The Vulcan runtime has the required Depot, Secret, and Compute permissions.
* Connector libraries and credentials are available for each custom catalog.

See [Spark Lakehouse permissions](https://v2.dataos.info/references/engine-guide/spark-1/permissions-and-grants.md) for the required access.

## Connect a DataOS Lakehouse

DataOS Lakehouse supports these object-storage backends:

* Amazon S3
* Azure Data Lake Storage through ABFSS
* Google Cloud Storage (GCS)

Create the Lakehouse and its Depot before deploying Vulcan:

1. [Create a Lakehouse](https://v2.dataos.info/references/resources/lakehouse/create-a-lakehouse.md).
2. [Create a Lakehouse Depot](https://v2.dataos.info/references/resources/lakehouse/create-a-lakehouse-depot.md).
3. Use the [storage-specific guides](https://v2.dataos.info/references/resources/lakehouse/supported-sources.md) for S3, ABFSS, or GCS configuration.
4. Grant the Vulcan runtime read/write access to the Depot. See [Spark Lakehouse permissions](https://v2.dataos.info/references/engine-guide/spark-1/permissions-and-grants.md).

Attach the Lakehouse Depot with `purpose=rw`:

```yaml
spec:
  engine: spark
  depots:
    - dataos://<lakehouse-depot-name>?purpose=rw
```

The stack uses the Depot name as the Spark catalog name. For example, a Depot named `saleslakehouse` is referenced as:

```sql
SELECT *
FROM saleslakehouse.analytics.orders;
```

Do not manually repeat the generated `spark.sql.catalog.<lakehouse-depot-name>.*` settings. The stack obtains the REST endpoint, warehouse location, and storage credentials from the Lakehouse Depot.

## Add a custom Spark catalog

Use `spec.workflow.sparkConf` to define catalogs that the Lakehouse Depot does not generate. A custom catalog needs:

* A Spark catalog implementation.
* Connector-specific properties, such as an endpoint, catalog type, or JDBC URL.
* The required connector libraries on the Spark driver and executor classpaths.
* Credentials projected from DataOS Secrets.

The example below demonstrates this pattern with PostgreSQL and Microsoft SQL Server JDBC V2 catalogs alongside a DataOS Lakehouse catalog. Other catalog implementations can use the same pattern with their own Spark properties and connector libraries.

{% hint style="info" %}
Only attached Lakehouse Depots generate Spark catalogs automatically. Attaching another Depot type does not automatically create a Spark JDBC catalog; define that catalog explicitly in `sparkConf`.
{% endhint %}

### Project credentials from DataOS Secrets

Store source credentials in DataOS Secrets and project them as environment variables. Do not place usernames or passwords directly in `sparkConf`.

The example below uses the DataOS resource-level `spec.use.projection` syntax in the Vulcan deployment manifest. This is separate from the project-level `deployment.use` generator.

Create Secrets containing lowercase `username` and `password` keys, as referenced by the templates below. Secret values are base64-encoded in the projection context, so the templates apply `base64_decode` before exposing them as environment variables.

```yaml
spec:
  use:
    projection:
      secrets:
        - id: <tenant>:postgres-secret
          contextAlias: pg
        - id: <tenant>:mssql-secret
          contextAlias: mssql
      projections:
        envVars:
          - key: POSTGRES_USERNAME
            template: "{{ secrets['pg'].username | base64_decode }}"
          - key: POSTGRES_PASSWORD
            template: "{{ secrets['pg'].password | base64_decode }}"
          - key: MSSQL_USERNAME
            template: "{{ secrets['mssql'].username | base64_decode }}"
          - key: MSSQL_PASSWORD
            template: "{{ secrets['mssql'].password | base64_decode }}"
```

See [Consume a Secret](https://v2.dataos.info/references/resources/secret/consume.md) and [Projections](https://v2.dataos.info/references/resources/projections.md) for secret creation, access, and projection syntax.

### Configure multiple catalogs

Place the JDBC driver JARs in the project's `dependencies/java/` directory. Configure each catalog under a unique `spark.sql.catalog.<name>` prefix.

```yaml
version: v1alpha
type: vulcan
name: spark-multi-catalog

spec:
  runAsUser: <dataos-user>
  compute: <spark-compute>
  engine: spark

  use:
    projection:
      secrets:
        - id: <tenant>:postgres-secret
          contextAlias: pg
        - id: <tenant>:mssql-secret
          contextAlias: mssql
      projections:
        envVars:
          - key: POSTGRES_USERNAME
            template: "{{ secrets['pg'].username | base64_decode }}"
          - key: POSTGRES_PASSWORD
            template: "{{ secrets['pg'].password | base64_decode }}"
          - key: MSSQL_USERNAME
            template: "{{ secrets['mssql'].username | base64_decode }}"
          - key: MSSQL_PASSWORD
            template: "{{ secrets['mssql'].password | base64_decode }}"

  repo:
    url: <git-repository-url>
    syncFlags:
      - --ref=<branch>
      - --submodules=off
    baseDir: <path-to-vulcan-project>
    secretId: <tenant>:git-sync

  depots:
    - dataos://<lakehouse-depot-name>?purpose=rw

  workflow:
    sparkConf:
      spark.sql.catalogImplementation: "in-memory"
      spark.sql.adaptive.enabled: "true"
      spark.sql.adaptive.coalescePartitions.enabled: "true"

      # PostgreSQL JDBC V2 catalog.
      spark.sql.catalog.postgres: "org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog"
      spark.sql.catalog.postgres.url: "jdbc:postgresql://<host>:5432/<database>"
      spark.sql.catalog.postgres.driver: "org.postgresql.Driver"
      spark.sql.catalog.postgres.user: "{{ env_var('POSTGRES_USERNAME') }}"
      spark.sql.catalog.postgres.password: "{{ env_var('POSTGRES_PASSWORD') }}"

      # Microsoft SQL Server JDBC V2 catalog.
      spark.sql.catalog.mssql: "org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog"
      spark.sql.catalog.mssql.url: "jdbc:sqlserver://<host>:1433;databaseName=<database>;encrypt=true;trustServerCertificate=false"
      spark.sql.catalog.mssql.driver: "com.microsoft.sqlserver.jdbc.SQLServerDriver"
      spark.sql.catalog.mssql.user: "{{ env_var('MSSQL_USERNAME') }}"
      spark.sql.catalog.mssql.password: "{{ env_var('MSSQL_PASSWORD') }}"

    logLevel: INFO
    schedule:
      crons:
        - "0 0 * * *"
      timezone: UTC
      concurrencyPolicy: Forbid
```

The API track uses a separate Spark session. If API queries need the same custom catalogs, repeat their catalog properties under `spec.api.sparkConf`. Repeat classpath properties only when using a custom dependency path or maintaining a raw manifest; the standard stack already adds `dependencies/java/*` to both classpaths.

## Query across catalogs

Use fully qualified three-part names: `<catalog>.<namespace>.<table>`.

```sql
SELECT
  o.order_id,
  c.customer_name
FROM <lakehouse-depot-name>.sales.orders AS o
JOIN postgres.public.customers AS c
  ON o.customer_id = c.customer_id;
```

Use the Lakehouse catalog in a model's `name` when the model output should be materialized as an Iceberg table:

```sql
MODEL (
  name <lakehouse-depot-name>.analytics.customer_orders,
  kind FULL
);

SELECT
  o.order_id,
  c.customer_name
FROM <lakehouse-depot-name>.sales.orders AS o
JOIN postgres.public.customers AS c
  ON o.customer_id = c.customer_id;
```

Read and write support for a custom catalog depends on its Spark catalog implementation, JDBC driver, source permissions, and connector capabilities. Validate each source before using it in production.

## Configure Spark properties

Use Spark properties to control query planning, shuffle behavior, file scanning, executor allocation, catalogs, and classpaths.

Add shared properties in `config.yaml`, or configure the workflow and API Spark sessions directly in `deploy.yaml`.

{% tabs %}
{% tab title="config.yaml" %}
The deployment generator writes `deployment.sparkConf` to both the workflow and API Spark sessions.

```yaml
deployment:
  runAsUser: <dataos-user>
  compute: <spark-compute>
  engine: spark
  repo:
    url: <git-repository-url>
    baseDir: <path-to-vulcan-project>
  depots:
    - dataos://<lakehouse-depot-name>?purpose=rw

  sparkConf:
    spark.sql.shuffle.partitions: "64"
    spark.sql.adaptive.enabled: "true"
    spark.sql.adaptive.coalescePartitions.enabled: "true"
    spark.sql.adaptive.skewJoin.enabled: "true"
```

Values in `deployment.sparkConf` override the generated defaults.
{% endtab %}

{% tab title="deploy.yaml" %}
When maintaining the DataOS resource directly, configure the workflow and API sessions separately:

```yaml
spec:
  workflow:
    sparkConf:
      spark.sql.shuffle.partitions: "64"
      spark.sql.adaptive.enabled: "true"

  api:
    sparkConf:
      spark.sql.shuffle.partitions: "64"
      spark.sql.adaptive.enabled: "true"
```

Properties in one session are not copied to the other. Add a property to both blocks when workflow jobs and API queries require it.
{% endtab %}
{% endtabs %}

The deployment generator supplies these defaults:

| Property                                        | Default  |
| ----------------------------------------------- | -------- |
| `spark.sql.shuffle.partitions`                  | `"16"`   |
| `spark.sql.adaptive.coalescePartitions.enabled` | `"true"` |

{% hint style="warning" %}
The `vulcan-spark` stack generates Lakehouse catalog properties from attached Depots. User-supplied `sparkConf` is merged after generated Depot catalog properties, so duplicate keys override the Depot projection. Do not redefine generated catalog keys unless that override is intentional.
{% endhint %}

### Common Spark properties

| Property                                          | Purpose                                                        |
| ------------------------------------------------- | -------------------------------------------------------------- |
| `spark.sql.adaptive.enabled`                      | Enables adaptive query execution                               |
| `spark.sql.adaptive.coalescePartitions.enabled`   | Coalesces small shuffle partitions at runtime                  |
| `spark.sql.adaptive.skewJoin.enabled`             | Optimizes skewed joins at runtime                              |
| `spark.sql.shuffle.partitions`                    | Sets the initial number of shuffle partitions                  |
| `spark.sql.files.maxPartitionBytes`               | Limits the amount of file data assigned to one input partition |
| `spark.dynamicAllocation.enabled`                 | Enables dynamic executor allocation                            |
| `spark.dynamicAllocation.shuffleTracking.enabled` | Preserves shuffle data while executors change                  |
| `spark.dynamicAllocation.initialExecutors`        | Sets the initial executor count                                |
| `spark.dynamicAllocation.minExecutors`            | Sets the minimum executor count                                |
| `spark.dynamicAllocation.maxExecutors`            | Sets the maximum executor count                                |

`sparkConf` property names are strings. Values can be strings, numbers, or booleans; quoted strings provide consistent serialization for values passed to Spark.

Dynamic allocation changes only executor count. It does not scale the API driver, API replicas, or request concurrency. Spark API autoscaling through `spec.api.autoscale` is not supported.

Use the [deployment recipe fragments](https://v2.dataos.info/references/engine-guide/spark-1/recipes/deployment-recipes.md) as starting examples. Validate all tuning values with representative data and the Spark UI.

## Configure driver and executor resources

Spark workflow resources use separate `driver` and `executor` blocks:

{% tabs %}
{% tab title="config.yaml" %}

```yaml
deployment:
  workflow:
    resources:
      driver:
        cores: 2
        coreLimit: "3"
        memory: 4g
      executor:
        cores: 2
        coreLimit: "3"
        memory: 4g
        instances: 2
```

{% endtab %}

{% tab title="deploy.yaml" %}

```yaml
spec:
  workflow:
    resources:
      driver:
        cores: 2
        coreLimit: "3"
        memory: 4g
      executor:
        cores: 2
        coreLimit: "3"
        memory: 4g
        instances: 2
```

{% endtab %}
{% endtabs %}

| Property    | Applies to          | Type    | Description                             |
| ----------- | ------------------- | ------- | --------------------------------------- |
| `cores`     | Driver and executor | Integer | CPU cores assigned to each process      |
| `coreLimit` | Driver and executor | String  | Maximum CPU allocation for each process |
| `memory`    | Driver and executor | String  | Memory assigned to each process         |
| `instances` | Executor only       | Integer | Number of executor instances            |

All fields shown are required when this Spark resource shape is used. The driver does not accept `instances`; one Spark application has one driver.

Define resources under `deployment.workflow.resources` in `config.yaml` or `spec.workflow.resources` in `deploy.yaml`, not under `sparkConf`.

## Add project dependencies

Store project-specific dependencies under the project root:

```
<project-root>/
└── dependencies/
    ├── java/
    │   ├── postgresql.jar
    │   └── custom-connector.jar
    └── python/
        └── custom_package-1.0.0-py3-none-any.whl
```

### Add Java dependencies

Place Spark connector, JDBC driver, UDF, and other JVM JARs directly in `dependencies/java/`. Keep JARs flat in this directory because Spark classpath wildcards are not recursive.

For the standard DataOS Lakehouse path, the `vulcan-spark` stack adds this directory to both classpaths:

```yaml
spark.driver.extraClassPath: /etc/dataos/work/<repo-baseDir>/dependencies/java/*
spark.executor.extraClassPath: /etc/dataos/work/<repo-baseDir>/dependencies/java/*
```

Define these properties explicitly when using a custom path or maintaining a raw manifest. The driver needs catalog and planning classes; executors need classes used while reading, transforming, or writing data.

### Add Python dependencies

Build additional Python packages as wheel files and place them under `dependencies/python/`. Nested directories are supported for Python dependencies.

```
dependencies/python/custom_package-1.0.0-py3-none-any.whl
```

Commit the wheel with the project so Git sync makes it available to the deployed runtime. Import the installed package normally from a Python model.

### Dependency compatibility

Before adding a dependency:

* Match the connector or library to the deployed Spark, Scala, Java, and Iceberg versions.
* Include transitive JARs when the selected distribution does not bundle them.
* Avoid adding duplicate versions of libraries already present in the runtime image.
* Test driver and executor loading in a non-production environment.

Missing or incompatible dependencies commonly appear as `ClassNotFoundException`, `NoClassDefFoundError`, `ImportError`, or serialization failures.

## Validate the connection

Confirm that:

* The Lakehouse Depot is attached with the required purpose.
* The projected environment variables are available without exposing their values.
* The JDBC driver JARs are available to the driver and executors.
* Python wheels required by models are available under `dependencies/python/`.
* Every catalog can list its expected namespaces and tables.
* A controlled model can read across catalogs and write its output to the Lakehouse.

Generate and inspect the deployment manifest in a non-production project:

```bash
vulcan create_deploy_yaml --overwrite
```

Confirm that the generated resource contains the expected properties under `spec.workflow.sparkConf` and, when required, `spec.api.sparkConf`.

Apply the generated resource and trigger its plan workflow:

```bash
dataos-ctl resource apply -f <generated-deploy-file>

dataos-ctl resource runtime re-run \
  -t vulcan \
  -n <resource-name> \
  -v v1alpha \
  --sub-resource-id vulcan-plan
```

Use a controlled model that references each custom source catalog. Review the deployed plan before triggering `vulcan-run`:

```bash
dataos-ctl resource runtime re-run \
  -t vulcan \
  -n <resource-name> \
  -v v1alpha \
  --sub-resource-id vulcan-run
```

Confirm from the deployed run status and logs that credentials, connector JARs, classpaths, and catalog properties load successfully, and that the model reads from the expected source catalogs and writes its output to the target Lakehouse catalog.

### Replace example placeholders

Before applying the deployment, replace every angle-bracket placeholder. In particular:

* `<repo-baseDir>` must match `spec.repo.baseDir`.
* `<git-repository-url>` must identify the repository containing the Vulcan project.
* `<path-to-vulcan-project>` must be the project path within that repository and must match `spec.repo.baseDir`.
* `<lakehouse-depot-name>` must match the attached Lakehouse Depot.
* `<host>`, `<database>`, and Secret IDs must identify the custom source.
* `<branch>`, `<dataos-user>`, and `<spark-compute>` must match the target DataOS environment.
* `<generated-deploy-file>` and `<resource-name>` must identify the generated manifest and applied Vulcan resource.

## Related

* [Spark overview](https://v2.dataos.info/references/engine-guide/spark-1.md)
* [Local Spark development](https://v2.dataos.info/references/engine-guide/spark-1/local-development.md)
* [Materialization behavior](https://v2.dataos.info/references/engine-guide/spark-1/materialization-behaviour.md)
* [Spark Lakehouse permissions](https://v2.dataos.info/references/engine-guide/spark-1/permissions-and-grants.md)
* [Troubleshooting](https://v2.dataos.info/references/engine-guide/spark-1/troubleshooting.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
