For the complete documentation index, see llms.txt. This page is also available as Markdown.

Permissions and grants

Provision DataOS, lakehouse, compute, state-store, storage, and Git access for Spark deployments.

This is the authorize stage: the step before you touch a gateway config or write a model. Provision the platform resources, object-storage grants, and DataOS access before you deploy a Vulcan Data Product using Spark.

Platform dependencies

Provision the platform before you write any model code. Spark itself brings no compute, storage, or state store — the items below (compute pool, tenant stack, lakehouse, secrets, and DataOS access) all have to be in place first, or you won't even be able to connect, let alone deploy a model. Work through this list with your SRE team before starting.

Required platform resources

Request these from your SRE team, they cover the Spark runtime, Iceberg storage, state store, and source-code access:

Requirement
Notes

Spark-capable compute pool

Verify with dataos-ctl resource -t compute get -a. Set spec.compute to this pool name.

Tenant-level vulcan-spark stack

Provides the runtime image, serving sidecars, dependency loading, and catalog templating.

DataOS lakehouse resource

Backed by S3 or ABFSS object storage.

Lakehouse depot

Attach as dataos://<depot>?purpose=rw in the Vulcan resource.

External Postgres for Vulcan state

The platform provisions and wires this in production. Do not add it to project config.yaml.

Object-storage credentials

Store these on the lakehouse or depot. Never hard-code them in project code.

Git-sync secret

Lets the Vulcan resource pull model code.

Roles and minimum storage grants

The three roles below separate platform administration, workload execution, and consumer access.

Role
Responsibilities

Admin role

Platform SRE provisions the compute pool, installs the stack, and creates the lakehouse and depot.

Vulcan service role

Configure as runAsUser. Runs Spark jobs and writes Iceberg tables through the lakehouse depot.

Consumer role

Provides read-only access through API endpoints.

The lakehouse depot owns storage credentials. Grant the required cloud access to its backing storage.

Storage
Required access

S3

s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket on the warehouse bucket.

ABFSS

Storage Blob Data Contributor on the container. Store Storage Account Key or SPN credentials on the depot.

DataOS permissions

After platform provisioning is complete, grant the following DataOS permissions:

Permission

What it Allows

Granted By

Tenant Access

Build and manage Data Products within your tenant.

Tenant Admin

roles:id:data-dev (or your tenant's equivalent role tag)

Create and apply Vulcan resources. Verify your role using dataos-ctl user get (check the TAGS column).

Tenant Admin

depot:rw:<lakehouse-depot-name>

Read and write access to the lakehouse depot.

Tenant Admin or Data Admin

depot:r:<lakehouse-depot-name>

Read-only access to the lakehouse depot (for consumers).

Tenant Admin or Data Admin

Can Use (Compute, Depots, Secrets)

Allows you to actually use Compute, Depots, and Secrets. Note: Resource access is deny-by-default. Even if you have a role tag or depot permission, you also need Can Use to use the resource.

Every permission can be granted to a tenant Role now -grant it once and the whole team, including future joiners, inherits it automatically.

git-sync Secret

Allows Vulcan to pull model code from Git using credentials stored in a Secret. Request a git-sync Secret, not raw Git credentials. If you create the Secret yourself, you still need Can Use permission on it.

Tenant Admin, Data Admin, or the Secret owner (with Can Use granted)

Run dataos-ctl get depot to confirm your read access before you start. If the lakehouse depot is missing, request depot:r from your operator.

Python version

Local development requires Python 3.10 to 3.13 (python --version must report 3.10.x to 3.13.x). The Spark runtime inside the Vulcan image manages its own Python version.

Next steps

With access and platform resources ready, configure your gateway in Connect to Spark.

Last updated

Was this helpful?