> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/vulcan/ldk.md).

# LDK setup

Set up the Local Development Kit: Python virtual environment, Vulcan, and engine.

The Local Development Kit (LDK) is the local toolkit you author and test Data Products with. Setting it up means installing Python, creating an isolated environment, installing Vulcan, and pointing it at an engine. Do this once.

{% hint style="info" %}
Vulcan requires **Python 3.10 to 3.13** (`>=3.10,<3.14`). Python 3.9 and earlier, and Python 3.14 and later, are **not supported**.
{% endhint %}

## 1. Check your Python version

Check which Python version you have:

```sh
python --version
```

If it reports 3.10, 3.11, 3.12, or 3.13, move on to the next step. Otherwise, install a supported version from [python.org/downloads](https://www.python.org/downloads/).

{% hint style="info" %}
On some macOS and Linux systems, the command is `python3` instead of `python`. Use whichever command reports the supported version for the rest of this setup.
{% endhint %}

## 2. Create a virtual environment

Always install Vulcan inside an isolated environment so it does not conflict with other Python projects. With a supported version confirmed, create the environment:

```sh
python -m venv .venv
```

Activate it:

{% tabs %}
{% tab title="macOS / Linux" %}

```bash
source .venv/bin/activate
```

{% endtab %}

{% tab title="Windows" %}
**Command Prompt**

```bat
.venv\Scripts\activate.bat
```

**PowerShell**

PowerShell may block the activation script. Run this once for your user:

```powershell
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
```

Then activate the environment:

```powershell
.\.venv\Scripts\Activate.ps1
```

If your environment folder is named `venv`, run `.\venv\Scripts\Activate.ps1`.

{% hint style="info" %}
The execution policy change allows locally created scripts to run. It does not require administrator access.
{% endhint %}
{% endtab %}
{% endtabs %}

Upgrade pip inside the environment:

```sh
pip install --upgrade pip
```

## 3. Install Vulcan

Vulcan ships as a Python wheel. Click on the **Download** button below to directly download the whl file.

{% file src="/files/kOV6VOzYotAGcOq07MNU" %}

Once downloaded, place the Vulcan `.whl` file in your working directory (or use its full path), then install:

```sh
pip install "./vulcan-0.228.1.32-py3-none-any.whl"
```

If you target a specific engine, install the matching extra. Quote the path so your shell does not interpret the brackets:

{% tabs %}
{% tab title="Postgres" %}

```bash
pip install "./vulcan-0.228.1.32-py3-none-any.whl[postgres]"
```

{% endtab %}

{% tab title="Snowflake" %}

```bash
pip install "./vulcan-0.228.1.32-py3-none-any.whl[snowflake]"
```

{% endtab %}

{% tab title="Databricks" %}

```bash
pip install "./vulcan-0.228.1.32-py3-none-any.whl[databricks]"
```

{% endtab %}

{% tab title="Spark" %}

```bash
pip install --use-deprecated=legacy-resolver "./vulcan-0.228.1.32-py3-none-any.whl[spark,postgres]"
```

The local Spark stack needs Python 3.13. `--use-deprecated=legacy-resolver` avoids pip's `ResolutionTooDeep` error on this wheel.
{% endtab %}

{% tab title="Trino" %}

```bash
pip install "./vulcan-0.228.1.32-py3-none-any.whl[trino]"
```

{% endtab %}

{% tab title="SQL Server" %}

```bash
pip install "./vulcan-0.228.1.32-py3-none-any.whl[mssql]"
```

{% endtab %}

{% tab title="Fabric" %}

```bash
pip install "./vulcan-0.228.1.32-py3-none-any.whl[fabric]"
```

{% endtab %}

{% tab title="Redshift" %}

```bash
pip install "./vulcan-0.228.1.32-py3-none-any.whl[redshift]"
```

{% endtab %}

{% tab title="Athena" %}

```bash
pip install "./vulcan-0.228.1.32-py3-none-any.whl[athena]"
```

{% endtab %}
{% endtabs %}

Verify:

```sh
vulcan --version
```

## 4. Initialize the project

```bash
vulcan init
```

The initializer scaffolds the project structure:

<table data-search="false"><thead><tr><th>Folder / file</th><th>What goes here</th></tr></thead><tbody><tr><td><code>config.yaml</code></td><td>Project configuration: connections, model defaults, linting.</td></tr><tr><td><code>usage.yml</code></td><td>Usage metadata: intended use, out-of-scope use, caveats, shown to consumers in the Data Product Hub.</td></tr><tr><td><code>agreement.md</code></td><td>Data-sharing agreement, referenced by <code>config.yaml</code>'s <code>agreement_path</code>. Optional.</td></tr><tr><td><code>models/</code></td><td>SQL and Python model files. Each produces a table or view.</td></tr><tr><td><code>dq/</code></td><td>Data quality checks (<code>kind: dq</code>). Non-blocking; monitor quality over time.</td></tr><tr><td><code>models/semantics/</code></td><td>Semantic models (<code>kind: semantic</code>). Business-friendly wrappers over physical models.</td></tr><tr><td><code>models/metrics/</code></td><td>Metric definitions (<code>kind: metric</code>). Time-series analytical definitions.</td></tr><tr><td><code>seeds/</code></td><td>CSV files loaded as static tables.</td></tr><tr><td><code>audits/</code></td><td>SQL audit files. They run at materialization and block execution if they return rows.</td></tr><tr><td><code>tests/</code></td><td>YAML unit tests. Run with <code>vulcan test</code> before touching the warehouse.</td></tr><tr><td><code>macros/</code></td><td>Reusable SQL snippets and Jinja macros.</td></tr><tr><td><code>plugins/</code></td><td>Auth extension hook (<code>after_authorize</code>) that maps identity provider tags to group names.</td></tr><tr><td><code>policies/</code></td><td>Auth-backed access and masking policies.</td></tr><tr><td><code>.vulcan/</code></td><td>Local state (default DuckDB state database, <code>state.db</code>). Git-ignored by <code>vulcan init</code>.</td></tr></tbody></table>

## 5. Set up your engine

To connect Vulcan locally, use an existing engine instance or spin one up via Docker, then add its connection to `config.yaml`. For the full connection reference per engine, see [Connect engine](https://v2.dataos.info/build/productize/connect-engine).

{% tabs %}
{% tab title="Postgres" %}
Use an existing Postgres instance if you already have one. You need the host, port, database, user, and password. See the [Postgres connection options](https://v2.dataos.info/build/productize/connect-engine/postgres#connection-options) for all supported fields.

Set your password as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export POSTGRES_PASSWORD='your_password'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:POSTGRES_PASSWORD = 'your_password'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: postgres
      host: your_psql_host
      port: 5433
      database: warehouse
      user: vulcan
      password: "{{ env_var('POSTGRES_PASSWORD') }}"
```

**OR**

<details>

<summary><strong>Start Postgres locally with Docker</strong></summary>

Create the network once:

```bash
docker network create vulcan
```

Save this as `docker/docker-compose.warehouse.yml`:

```yaml
volumes:
  warehouse:
    driver: local
networks:
  vulcan:
    external: true
services:
  warehouse:
    image: postgres:17-alpine
    environment:
      POSTGRES_DB: warehouse
      POSTGRES_USER: vulcan
      POSTGRES_PASSWORD: vulcan
      POSTGRES_HOST_AUTH_METHOD: trust
    ports:
      - "5433:5432"
    volumes:
      - warehouse:/var/lib/postgresql/data
    networks:
      - vulcan
```

Start it:

```bash
docker compose -f docker/docker-compose.warehouse.yml up -d
```

</details>

Full reference: [Connect engine → Postgres](https://v2.dataos.info/build/productize/connect-engine/postgres).
{% endtab %}

{% tab title="Snowflake" %}
Use an existing Snowflake account and warehouse. No local Docker service is needed for Snowflake.

Set your password as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export SNOWFLAKE_PASSWORD='your_password'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:SNOWFLAKE_PASSWORD = 'your_password'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: snowflake
      account: your_account
      user: your_user
      password: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
      warehouse: your_warehouse
      database: your_database
      role: your_role

```

Full reference: [Connect engine → Snowflake](https://v2.dataos.info/build/productize/connect-engine/snowflake).
{% endtab %}

{% tab title="Databricks" %}
Use an existing workspace with a SQL warehouse or cluster access. No local Docker service is needed for Databricks.

Set the access token as an environment variable, then:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export DATABRICKS_TOKEN='your_token'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:DATABRICKS_TOKEN = 'your_token'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: databricks
      server_hostname: your-workspace.azuredatabricks.net
      http_path: /sql/1.0/warehouses/your_warehouse_id
      access_token: "{{ env_var('DATABRICKS_TOKEN') }}"
      catalog: your_catalog

```

Full reference: [Connect engine → Databricks](https://v2.dataos.info/build/productize/connect-engine/databricks).
{% endtab %}

{% tab title="Spark" %}
Spark is not a single server you connect to, so local development runs a small Docker stack next to your project: a Spark master and two workers, an Iceberg REST catalog, an object store (local MinIO, or AWS S3, Azure ABFSS, or Google GCS), and a Postgres state store.

Choose where Vulcan runs:

|                   | Local (recommended)                    | Containerized              |
| ----------------- | -------------------------------------- | -------------------------- |
| Host requirements | Docker, Java 17, Python 3.13           | Docker only                |
| Vulcan runs in    | The virtual environment from this page | The `vulcan-cli` container |

{% hint style="warning" %}
For Spark, the virtual environment must use **Python 3.13**, the same minor version as the Spark workers. Any other version fails with `PYTHON_VERSION_MISMATCH`.
{% endhint %}

Follow [Local development](https://v2.dataos.info/references/engine-guide/spark-1/connection-options#local-development) in the Spark engine manual to add the Docker stack (`docker/docker-compose.yml`), choose where your data lives in `.env`, and replace the `gateways:` block in `config.yaml`.

Full reference: [Connect engine → Spark](https://v2.dataos.info/build/productize/connect-engine/spark).
{% endtab %}

{% tab title="Trino" %}
Use an existing Trino cluster with a configured catalog. No local Docker service is needed for Trino. Set the password only if your cluster requires it, then:

For Minerva, DataOS's managed Trino cluster, generate the password from your DataOS user ID, API key, and tenant name: a base64-encoded JSON object with the fields `cluster`, `apikey`, and `tenant`.

**macOS and Linux**

```bash
echo -n '{"cluster":"<cluster>","apikey":"<api-key>","tenant":"<tenant>"}' | base64 | tr -d '\n'
```

**PowerShell**

```powershell
[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes('{"cluster":"<cluster>","apikey":"<api-key>","tenant":"<tenant>"}'))
```

**Command Prompt**

Command Prompt has no built-in base64 encoder. Use `certutil`, then strip its header, footer, and line breaks:

```bat
echo {"cluster":"<cluster>","apikey":"<api-key>","tenant":"<tenant>"}> payload.json
certutil -encode payload.json payload.b64
type payload.b64
```

`certutil -encode` wraps the output in `-----BEGIN CERTIFICATE-----` / `-----END CERTIFICATE-----` lines and inserts line breaks every 64 characters. Open `payload.b64`, remove those two header/footer lines, and join the remaining lines into one string before using it as the password. PowerShell is the simpler option on Windows.

This generates the password for connecting via Minerva, DataOS's managed Trino cluster. For Dedicated Trino or External Trino, see [Connect engine → Trino](https://v2.dataos.info/build/productize/connect-engine/trino) for connection details specific to your cluster type.

Set the generated value as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export TRINO_PASSWORD='your_password'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:TRINO_PASSWORD = 'your_password'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: trino
      host: your_trino_host
      port: 8080
      user: your_user
      catalog: your_catalog
      http_scheme: https
      password: "{{ env_var('TRINO_PASSWORD') }}"

model_defaults:
  dialect: trino
```

Full reference: [Connect engine → Trino](https://v2.dataos.info/build/productize/connect-engine/trino).
{% endtab %}

{% tab title="SQL Server" %}
Use an existing SQL Server instance (2019–2022) or Azure SQL Database. You need the host, port, database, user, and password. See the [SQL Server connection options](https://v2.dataos.info/build/productize/connect-engine/sql-server#connection-options) for all supported fields.

Set your password as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export MSSQL_PASSWORD='your_password'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:MSSQL_PASSWORD = 'your_password'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: mssql
      host: your_mssql_host
      port: 1433
      database: warehouse
      user: your_user
      password: "{{ env_var('MSSQL_PASSWORD') }}"
```

Full reference: [Connect engine → SQL Server](https://v2.dataos.info/build/productize/connect-engine/sql-server).
{% endtab %}

{% tab title="Fabric" %}
Use an existing Fabric workspace with an assigned capacity and a Fabric Warehouse item. No local Docker service is needed for Fabric. You'll need an Entra ID app registration (service principal) with a workspace role, plus the workspace's `tenant_id` and `workspace_id`.

Set the client secret as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export FABRIC_CLIENT_SECRET='your_client_secret'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:FABRIC_CLIENT_SECRET = 'your_client_secret'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: fabric
      host: xxxxx.datawarehouse.fabric.microsoft.com
      driver: pyodbc
      driver_name: "ODBC Driver 18 for SQL Server"
      odbc_properties:
        authentication: ActiveDirectoryServicePrincipal
      tenant_id: your_tenant_id
      workspace_id: your_workspace_id
      user: your_client_id
      password: "{{ env_var('FABRIC_CLIENT_SECRET') }}"
      database: warehouse
```

Full reference: [Connect engine → Microsoft Fabric](https://v2.dataos.info/build/productize/connect-engine/microsoft-fabric).
{% endtab %}

{% tab title="Redshift" %}
Use an existing Redshift cluster or Serverless workgroup. Redshift has no local container, so point local development at a small dev cluster or a low-RPU Serverless workgroup.

Set your password as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export REDSHIFT_PASSWORD='your_password'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:REDSHIFT_PASSWORD = 'your_password'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: redshift
      host: your-cluster.abc123.us-east-1.redshift.amazonaws.com
      port: 5439
      database: your_database
      user: your_user
      password: "{{ env_var('REDSHIFT_PASSWORD') }}"
      ssl: true
      sslmode: require

```

A Postgres `state_connection` is strongly recommended for Redshift, so Vulcan's plan and interval state stays out of the warehouse.

Full reference: [Connect engine → Amazon Redshift](https://v2.dataos.info/build/productize/connect-engine/redshift).
{% endtab %}

{% tab title="Athena" %}
Use an existing Athena workgroup, Glue database, and S3 buckets in the same AWS region. Athena has no local container, so local development points directly at your AWS resources.

Set your AWS credentials as environment variables:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export AWS_ACCESS_KEY_ID='your_access_key_id'
export AWS_SECRET_ACCESS_KEY='your_secret_access_key'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:AWS_ACCESS_KEY_ID = 'your_access_key_id'
$env:AWS_SECRET_ACCESS_KEY = 'your_secret_access_key'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: athena
      aws_access_key_id: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
      aws_secret_access_key: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
      region_name: your_region
      work_group: primary
      s3_staging_dir: s3://your-athena-results/
      schema_name: your_glue_database
      s3_warehouse_location: s3://your-data-bucket/vulcan-warehouse/

```

`s3_warehouse_location` is required for managed table materialization.

Full reference: [Connect engine → Amazon Athena](https://v2.dataos.info/build/productize/connect-engine/athena).
{% endtab %}
{% endtabs %}

## 6. Verify

```bash
vulcan info # check if the connection is successful
```

A successful `vulcan info` confirms the project is ready to configure and build.

## Troubleshooting

<details>

<summary><code>... is not a supported wheel on this platform</code></summary>

Vulcan only supports Python 3.10 to 3.13 (not 3.9 or earlier, not 3.14 or later). Check the version with `python --version`, install a supported version if needed, then recreate the environment:

```sh
deactivate && rm -rf .venv
python -m venv .venv && source .venv/bin/activate
```

</details>

<details>

<summary><code>zsh: no matches found</code></summary>

Your shell is interpreting the `[engine]` brackets. Quote the wheel path: `pip install "./vulcan-...whl[snowflake]"`.

</details>

<details>

<summary>PowerShell blocks <code>Activate.ps1</code></summary>

Allow locally created scripts for your user, then activate the environment:

```powershell
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
.\.venv\Scripts\Activate.ps1
```

If the environment folder is named `venv`, use `.\venv\Scripts\Activate.ps1`.

</details>

<details>

<summary>Dependency conflicts</summary>

Install into a fresh virtual environment, never the system Python. To overwrite an existing install, use `pip install --force-reinstall "./vulcan-0.228.1.32-py3-none-any.whl"`.

</details>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/references/resources/vulcan/ldk.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
