> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/v1/get-started/prerequisites/ldk-setup.md).

# LDK setup

The Local Development Kit (LDK) is the local toolkit you author and test Data Products with. Setting it up means installing Python, creating an isolated environment, installing Vulcan, and pointing it at an engine. Do this once.

{% hint style="info" %}
Vulcan requires **Python 3.10**.
{% endhint %}

## 1. Install Python 3.10

Check first:

```sh
python3.10 --version
```

If it is not present, install it from [python.org/downloads](https://www.python.org/downloads/release/python-3100/).

## 2. Create a virtual environment

Always install Vulcan inside an isolated environment so it does not conflict with other Python projects.

```sh
python3.10 -m venv .venv
```

Activate it:

{% tabs %}
{% tab title="macOS / Linux" %}

```bash
source .venv/bin/activate
```

{% endtab %}

{% tab title="Windows" %}
**Command Prompt**

```bat
.venv\Scripts\activate.bat
```

**PowerShell**

PowerShell may block the activation script. Run this once for your user:

```powershell
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
```

Then activate the environment:

```powershell
.\.venv\Scripts\Activate.ps1
```

If your environment folder is named `venv`, run `.\venv\Scripts\Activate.ps1`.

{% hint style="info" %}
The execution policy change allows locally created scripts to run. It does not require administrator access.
{% endhint %}
{% endtab %}
{% endtabs %}

Upgrade pip inside the environment:

```sh
pip install --upgrade pip
```

## 3. Install Vulcan

Vulcan ships as a Python wheel. Click on the **Download** button below to directly download the whl file.

{% file src="/files/Mwiup3kYil4uA3fL67mN" %}

Once downloaded, place the Vulcan `.whl` file in your working directory (or use its full path), then install:

```sh
pip install "./vulcan-0.228.1.28-py3-none-any.whl"
```

If you target a specific engine, install the matching extra. Quote the path so your shell does not interpret the brackets:

{% tabs %}
{% tab title="Postgres" %}

```bash
pip install "./vulcan-0.228.1.28-py3-none-any.whl[postgres]"
```

{% endtab %}

{% tab title="Snowflake" %}

```bash
pip install "./vulcan-0.228.1.28-py3-none-any.whl[snowflake]"
```

{% endtab %}

{% tab title="Databricks" %}

```bash
pip install "./vulcan-0.228.1.28-py3-none-any.whl[databricks]"
```

{% endtab %}

{% tab title="Spark" %}

```bash
pip install "./vulcan-0.228.1.28-py3-none-any.whl[spark]"
```

{% endtab %}

{% tab title="Trino" %}

```bash
pip install "./vulcan-0.228.1.28-py3-none-any.whl[trino]"
```

{% endtab %}

{% tab title="SQL Server" %}

```bash
pip install "./vulcan-0.228.1.28-py3-none-any.whl[mssql]"
```

{% endtab %}

{% tab title="Fabric" %}

```bash
pip install "./vulcan-0.228.1.28-py3-none-any.whl[fabric]"
```

{% endtab %}
{% endtabs %}

Verify:

```sh
vulcan --version
```

## 4. Initialize the project

```bash
vulcan init
```

The initializer scaffolds the project structure:

<table data-search="false"><thead><tr><th>Folder / file</th><th>What goes here</th></tr></thead><tbody><tr><td><code>config.yaml</code></td><td>Project configuration: connections, model defaults, linting.</td></tr><tr><td><code>usage.yml</code></td><td>Usage metadata: intended use, out-of-scope use, caveats, shown to consumers in the Data Product Hub.</td></tr><tr><td><code>agreement.md</code></td><td>Data-sharing agreement, referenced by <code>config.yaml</code>'s <code>agreement_path</code>. Optional.</td></tr><tr><td><code>models/</code></td><td>SQL and Python model files. Each produces a table or view.</td></tr><tr><td><code>dq/</code></td><td>Data quality rule packs (<code>kind: dq</code>). Non-blocking; monitor quality over time.</td></tr><tr><td><code>models/semantics/</code></td><td>Semantic models (<code>kind: semantic</code>). Business-friendly wrappers over physical models.</td></tr><tr><td><code>models/metrics/</code></td><td>Metric definitions (<code>kind: metric</code>). Time-series analytical definitions.</td></tr><tr><td><code>seeds/</code></td><td>CSV files loaded as static tables.</td></tr><tr><td><code>audits/</code></td><td>SQL audit files. They run at materialization and block execution if they return rows.</td></tr><tr><td><code>tests/</code></td><td>YAML unit tests. Run with <code>vulcan test</code> before touching the warehouse.</td></tr><tr><td><code>macros/</code></td><td>Reusable SQL snippets and Jinja macros.</td></tr><tr><td><code>plugins/</code></td><td>Auth extension hook (<code>after_authorize</code>) that maps identity provider tags to group names.</td></tr><tr><td><code>policies/</code></td><td>Auth-backed access and masking policies.</td></tr><tr><td><code>.vulcan/</code></td><td>Local state (default DuckDB state database, <code>state.db</code>). Git-ignored by <code>vulcan init</code>.</td></tr></tbody></table>

## 5. Set up your engine

To connect Vulcan locally, use an existing engine instance or spin one up via Docker, then add its connection to `config.yaml`. For the full connection reference per engine, see [Connect engine](/build/v1/productize/connect-engine.md).

{% tabs %}
{% tab title="Postgres" %}
Use an existing Postgres instance if you already have one. You need the host, port, database, user, and password. See the [Postgres connection options](/build/v1/productize/connect-engine/postgres.md#connection-options) for all supported fields.

Set your password as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export POSTGRES_PASSWORD='your_password'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:POSTGRES_PASSWORD = 'your_password'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: postgres
      host: your_psql_host
      port: 5433
      database: warehouse
      user: vulcan
      password: "{{ env_var(POSTGRES_PASSWORD) }}"
```

**OR**

<details>

<summary><strong>Start Postgres locally with Docker</strong></summary>

Create the network once:

```bash
docker network create vulcan
```

Save this as `docker/docker-compose.warehouse.yml`:

```yaml
volumes:
  warehouse:
    driver: local
networks:
  vulcan:
    external: true
services:
  warehouse:
    image: postgres:17-alpine
    environment:
      POSTGRES_DB: warehouse
      POSTGRES_USER: vulcan
      POSTGRES_PASSWORD: vulcan
      POSTGRES_HOST_AUTH_METHOD: trust
    ports:
      - "5433:5432"
    volumes:
      - warehouse:/var/lib/postgresql/data
    networks:
      - vulcan
```

Start it:

```bash
docker compose -f docker/docker-compose.warehouse.yml up -d
```

</details>

Full reference: [Connect engine → Postgres](/build/v1/productize/connect-engine/postgres.md).
{% endtab %}

{% tab title="Snowflake" %}
Use an existing Snowflake account and warehouse. No local Docker service is needed for Snowflake.

Set your password as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export SNOWFLAKE_PASSWORD='your_password'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:SNOWFLAKE_PASSWORD = 'your_password'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: snowflake
      account: your_account
      user: your_user
      password: "{{ env_var('SNOWFLAKE_PASSWORD') }}"
      warehouse: your_warehouse
      database: your_database
      role: your_role

```

Full reference: [Connect engine → Snowflake](/build/v1/productize/connect-engine/snowflake.md).
{% endtab %}

{% tab title="Databricks" %}
Use an existing workspace with a SQL warehouse or cluster access. No local Docker service is needed for Databricks.

Set the access token as an environment variable, then:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export DATABRICKS_TOKEN='your_token'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:DATABRICKS_TOKEN = 'your_token'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: databricks
      server_hostname: your-workspace.azuredatabricks.net
      http_path: /sql/1.0/warehouses/your_warehouse_id
      access_token: "{{ env_var('DATABRICKS_TOKEN') }}"
      catalog: your_catalog

```

Full reference: [Connect engine → Databricks](/build/v1/productize/connect-engine/databricks.md).
{% endtab %}

{% tab title="Spark" %}
Use an existing Spark cluster and update `spark.master` in `config.yaml` to point to your cluster.

Update the connection details in `config.yaml` (sample):

{% code overflow="wrap" %}

```yaml
gateways:
  default:
    connection:
      type: spark
      config:
        spark.master: spark://spark-master:7077
        spark.app.name: vulcan
        spark.sql.catalog.local: org.apache.iceberg.spark.SparkCatalog
        spark.sql.catalog.local.type: rest
        spark.sql.catalog.local.uri: http://iceberg-rest:8181
        spark.sql.catalog.local.warehouse: s3://warehouse/
        spark.sql.catalog.local.io-impl: org.apache.iceberg.aws.s3.S3FileIO
        spark.sql.catalog.local.s3.endpoint: http://minio:9000
        spark.sql.catalog.local.s3.path-style-access: "true"
        spark.hadoop.fs.s3a.access.key: admin
        spark.hadoop.fs.s3a.secret.key: password
        spark.hadoop.fs.s3a.endpoint: http://minio:9000
        spark.hadoop.fs.s3a.path.style.access: "true"

model_defaults:
  dialect: spark2
```

{% endcode %}

**OR**

<details>

<summary><strong>Start Spark locally with Docker</strong></summary>

Start a local Spark standalone cluster with MinIO and an Iceberg REST catalog.

This avoids Windows Hadoop or `winutils.exe` issues because the Spark driver runs inside Linux.

Place `vulcan-0.228.1.28-py3-none-any.whl` in your project root, then save this as `docker/docker-compose.spark.yml`:

{% code overflow="wrap" expandable="true" %}

```yaml
services:
  # Spark standalone cluster for running Spark executors in containers.
  spark-master:
    image: tmdcio/vulcan-spark-base:0.228.1.21
    container_name: spark-seeds-minimal-spark-master
    restart: unless-stopped
    command: ["/bin/bash", "-lc", "/opt/spark/sbin/start-master.sh --host 0.0.0.0 --port 7077 --webui-port 8080 && tail -f /opt/spark/logs/*"]
    ports:
      - "7077:7077"
      - "8080:8080"
    networks:
      - spark-seeds-minimal-net

  spark-worker:
    image: tmdcio/vulcan-spark-base:0.228.1.21
    container_name: spark-seeds-minimal-spark-worker
    restart: unless-stopped
    command: ["/bin/bash", "-lc", "/opt/spark/sbin/start-worker.sh spark://spark-master:7077 --webui-port 8081 && tail -f /opt/spark/logs/*"]
    depends_on:
      - spark-master
    ports:
      - "8081:8081"
    networks:
      - spark-seeds-minimal-net

  # MinIO for S3-compatible storage.
  minio:
    image: minio/minio:latest
    container_name: spark-seeds-minimal-minio
    restart: unless-stopped
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      spark-seeds-minimal-net:
        aliases:
          - minio
          - warehouse.minio
    volumes:
      - minio_data:/data
    command: server /data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 5s
      timeout: 5s
      retries: 10

  # MinIO setup - creates warehouse bucket.
  mc:
    image: minio/mc:latest
    container_name: spark-seeds-minimal-mc
    networks:
      - spark-seeds-minimal-net
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
        mc alias set minio http://minio:9000 admin password;
        mc mb --ignore-existing minio/warehouse;
        mc anonymous set public minio/warehouse;
        exit 0;
      "

  # Iceberg REST Catalog.
  iceberg-rest:
    image: tabulario/iceberg-rest:latest
    container_name: spark-seeds-minimal-iceberg-rest
    restart: unless-stopped
    ports:
      - "8181:8181"
    networks:
      - spark-seeds-minimal-net
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000
    depends_on:
      minio:
        condition: service_healthy

networks:
  spark-seeds-minimal-net:
    driver: bridge

volumes:
  minio_data:
```

{% endcode %}

Start the Spark services:

```bash
docker compose -f docker/docker-compose.spark.yml up -d
```

Verify Vulcan through the CLI container:

```bash
docker compose -f docker/docker-compose.spark.yml run --rm vulcan-cli vulcan --version
```

</details>

Full reference: [Connect engine → Spark](/build/v1/productize/connect-engine/spark.md).
{% endtab %}

{% tab title="Trino" %}
Use an existing Trino cluster with a configured catalog. No local Docker service is needed for Trino. Set the password only if your cluster requires it, then:

For Minerva, DataOS's managed Trino cluster, generate the password from your DataOS user ID, API key, and tenant name: a base64-encoded JSON object with the fields `cluster`, `apikey`, and `tenant`.

**macOS and Linux**

```bash
echo -n '{"cluster":"<cluster>","apikey":"<api-key>","tenant":"<tenant>"}' | base64 | tr -d '\n'
```

**PowerShell**

```powershell
[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes('{"cluster":"<cluster>","apikey":"<api-key>","tenant":"<tenant>"}'))
```

**Command Prompt**

Command Prompt has no built-in base64 encoder. Use `certutil`, then strip its header, footer, and line breaks:

```bat
echo {"cluster":"<cluster>","apikey":"<api-key>","tenant":"<tenant>"}> payload.json
certutil -encode payload.json payload.b64
type payload.b64
```

`certutil -encode` wraps the output in `-----BEGIN CERTIFICATE-----` / `-----END CERTIFICATE-----` lines and inserts line breaks every 64 characters. Open `payload.b64`, remove those two header/footer lines, and join the remaining lines into one string before using it as the password. PowerShell is the simpler option on Windows.

This generates the password for connecting via Minerva, DataOS's managed Trino cluster. For Dedicated Trino or External Trino, see [Connect engine → Trino](/build/v1/productize/connect-engine/trino.md) for connection details specific to your cluster type.

Set the generated value as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export TRINO_PASSWORD='your_password'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:TRINO_PASSWORD = 'your_password'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: trino
      host: your_trino_host
      port: 8080
      user: your_user
      catalog: your_catalog
      http_scheme: https
      password: "{{ env_var('TRINO_PASSWORD') }}"

model_defaults:
  dialect: trino
```

Full reference: [Connect engine → Trino](/build/v1/productize/connect-engine/trino.md).
{% endtab %}

{% tab title="SQL Server" %}
Use an existing SQL Server instance (2019–2022) or Azure SQL Database. You need the host, port, database, user, and password. See the [SQL Server connection options](/build/v1/productize/connect-engine/sql-server.md#connection-options) for all supported fields.

Set your password as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export MSSQL_PASSWORD='your_password'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:MSSQL_PASSWORD = 'your_password'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: mssql
      host: your_mssql_host
      port: 1433
      database: warehouse
      user: your_user
      password: "{{ env_var('MSSQL_PASSWORD') }}"
```

Full reference: [Connect engine → SQL Server](/build/v1/productize/connect-engine/sql-server.md).
{% endtab %}

{% tab title="Fabric" %}
Use an existing Fabric workspace with an assigned capacity and a Fabric Warehouse item. No local Docker service is needed for Fabric. You'll need an Entra ID app registration (service principal) with a workspace role, plus the workspace's `tenant_id` and `workspace_id`.

Set the client secret as an environment variable:

{% tabs %}
{% tab title="Mac/Linux" %}

```bash
export FABRIC_CLIENT_SECRET='your_client_secret'
```

{% endtab %}

{% tab title="Windows" %}

```powershell
$env:FABRIC_CLIENT_SECRET = 'your_client_secret'
```

{% endtab %}
{% endtabs %}

Update the connection details in `config.yaml` (sample):

```yaml
gateways:
  default:
    connection:
      type: fabric
      host: xxxxx.datawarehouse.fabric.microsoft.com
      driver: pyodbc
      driver_name: "ODBC Driver 18 for SQL Server"
      odbc_properties:
        authentication: ActiveDirectoryServicePrincipal
      tenant_id: your_tenant_id
      workspace_id: your_workspace_id
      user: your_client_id
      password: "{{ env_var('FABRIC_CLIENT_SECRET') }}"
      database: warehouse
```

Full reference: [Connect engine → Microsoft Fabric](/build/v1/productize/connect-engine/microsoft-fabric.md).
{% endtab %}
{% endtabs %}

## 6. Verify

```bash
vulcan info # check if the connection is successful
```

A successful `vulcan info` confirms the project is ready to configure and build.

## Troubleshooting

<details>

<summary><code>... is not a supported wheel on this platform</code></summary>

Vulcan only supports Python 3.10, so recreate the environment with Python 3.10:

```sh
deactivate && rm -rf .venv
python3.10 -m venv .venv && source .venv/bin/activate
```

</details>

<details>

<summary><code>zsh: no matches found</code></summary>

Your shell is interpreting the `[engine]` brackets. Quote the wheel path: `pip install "./vulcan-...whl[snowflake]"`.

</details>

<details>

<summary>PowerShell blocks <code>Activate.ps1</code></summary>

Allow locally created scripts for your user, then activate the environment:

```powershell
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
.\.venv\Scripts\Activate.ps1
```

If the environment folder is named `venv`, use `.\venv\Scripts\Activate.ps1`.

</details>

<details>

<summary>Dependency conflicts</summary>

Install into a fresh virtual environment, never the system Python. To overwrite an existing install, use `pip install --force-reinstall "./vulcan-<version>-py3-none-any.whl"`.

</details>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/build/v1/get-started/prerequisites/ldk-setup.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.