> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/v1/productize/git-and-deploy.md). # Git & deploy Deploying turns a validated local project into a running, discoverable Data Product. DataOS pulls your project from Git, runs it on a schedule, and serves it through APIs. This page generates the deployment manifest, applies it, and verifies the result. A product isn't complete just because the models run locally. It becomes complete when it's deployed, discoverable, trusted, and queryable by others. ## Before you begin You need a clean local run, plus these deployment building blocks: * A passing [`vulcan plan`](/build/v1/productize/plan-and-run.md). * Your project pushed to a Git branch, with a [Git-sync secret](/build/v1/get-started/prerequisites/repository-setup.md) so DataOS can pull it. * A Depot with read/write access to the target, a valid engine, and a compute resource. * The target engine's own minimum grants provisioned for the Vulcan service account (see the engine's page under [Connect engine](/build/v1/productize/connect-engine.md)). A Depot grant alone doesn't give Vulcan permission to create or write tables on the engine itself. Confirm each of these with: ```bash dataos-ctl resource -t get -a ``` {% hint style="info" %} If `get -a` shows you don't have access to one of these, ask your tenant admin for a `Can Use` grant. This isn't something you can fix yourself. {% endhint %} For the full readiness checklist, see [Operate → Get access and confirm readiness](https://v2.dataos.info/operate/data-product-developer/access-and-readiness). ## 1. Render the deploy manifest Deployment is driven entirely by a `deployment:` block in `config.yaml`: where the repo is, which engine and compute to use, the schedule, and the API surface. You never hand-author or hand-edit the manifest itself; you set these keys and let Vulcan render it. ```yaml # config.yaml deployment: repo: url: https://bitbucket.org/your-org/your-repo base_dir: your-repo/path/to/orders-analytics compute: pacific run_as_user: "johndoetmdcio" engine: postgres # required only when the gateway connection is type: depot ``` {% hint style="info" %} Engine-specific config lives in this same `deployment:` block, not a separate file. `deployment.spark_conf` is only valid when the resolved engine is `spark`, and `deployment.trino` only when it's `trino`; setting either under the wrong engine fails validation. There's no more standalone `trino-deploy.yaml`. `deployment.state_store_connection` and `deployment.object_store_connection` are advanced, optional overrides that also live here. {% endhint %} Then generate the manifest: ```bash vulcan create_deploy_yaml ``` {% hint style="info" %} `create_deploy_yaml` is a **deterministic renderer**, not a starter template. Every value in the output comes from your project or the `deployment:` block above: there's nothing left to fill in by hand. If a value is wrong, fix `config.yaml` and regenerate, rather than editing the YAML output directly. {% endhint %} The rendered file looks like this: ```yaml version: v1alpha type: vulcan name: orders-analytics spec: run_as_user: "johndoetmdcio" compute: pacific engine: postgres repo: url: https://bitbucket.org/your-org/your-repo sync_flags: - '--ref=main' - '--submodules=off' base_dir: your-repo/path/to/orders-analytics secret_id: :git-sync depots: - dataos://postgresDepot?purpose=rw workflow: schedule: crons: ['*/15 * * * *'] end_on: '2027-01-01T00:00:00-00:00' timezone: 'US/Pacific' concurrency_policy: Forbid plan: command: [vulcan] arguments: ['--log-to-stdout', 'plan', '--auto-apply'] run: command: [vulcan] arguments: ['--log-to-stdout', 'run'] api: replicas: 2 ``` Set `end_on` a year or two out. An expired schedule stops silently, with no error. `repo.secret_id` is only needed if the repository is private. `concurrency_policy` always renders as `Forbid`; it isn't something you configure. Skip writing the file at all and pipe straight into apply: ```bash vulcan create_deploy_yaml -o - | dataos-ctl resource apply --in ``` {% hint style="warning" %} **`run_as_user` should be a service identity in production, not you.** The example uses a personal ID to get you running, but a product that runs as a person breaks when that person's access changes or they leave. For anything beyond a first test, run it as an **ApplicationUser**: your tenant admin creates one and grants you `Can Use ApiKey` so you can set it here. See [Operate → Get access and confirm readiness](https://v2.dataos.info/operate/data-product-developer/access-and-readiness). {% endhint %} ## 2. Lint the manifest Before applying for real, render the manifest and validate it with `-l` (lint / dry-run) instead of applying it: ```bash vulcan create_deploy_yaml -o - | dataos-ctl resource apply --in -l ``` The `-l` flag lints the rendered manifest without applying it, so a missing required field, an unsupported `engine`, or an unknown key under `deployment:` gets caught here instead of at DataOS. Fix any errors it reports in `config.yaml` and re-run until it passes clean. ## 3. Apply the resource Once the lint passes clean, run the same render piped into apply, without `-l`: ```bash vulcan create_deploy_yaml -o - | dataos-ctl resource apply --in ``` This deploys the Vulcan resource and starts its runtime components. A deployed product isn't one container: DataOS creates a `plan` component (runs `vulcan plan --auto-apply`), a `run` component (runs `vulcan run` on the schedule), and an `api` component (serves the endpoints). ## 4. Check status and logs ```bash dataos-ctl resource -t vulcan -n orders-analytics get dataos-ctl resource -t vulcan -n orders-analytics logs ``` Pick the log that matches what you're investigating: `run` for model execution, `plan` for planning and migration, `api` for query and endpoint issues. Fetch a specific component and container with `--container-group` and `-c`: ```bash dataos-ctl resource -t vulcan -n orders-analytics logs \ --container-group orders-analytics-run-execute -c main ``` {% hint style="info" %} For Spark gateways, the `run` and `plan` pods act as the Spark driver. Start with the DataOS run logs to see whether the failure happened before or during job submission, then move to the Spark driver and executor logs. {% endhint %} ## Verify the product is live Deployment is done when the product runs, serves, and shows up where consumers look. Confirm three things. **Models materialized.** Connect to the target and confirm the tables or views exist, row counts are sensible, and scheduled models are updating: ```sql SHOW TABLES IN SCHEMA .; SELECT * FROM .. LIMIT 10; ``` **The API responds.** Hit the liveness endpoint: ```bash curl --location 'https:///vulcan/tenants//vulcan/orders-analytics/livez' \ --header 'Authorization: Bearer ' ``` **The product is discoverable.** In DataOS, confirm the product appears in discovery with: * Its metadata, owner, and description * Inputs and outputs * Quality and run signals * Working access paths ## Common issues | Issue | What to check | | ------------------------ | ----------------------------------------------------- | | Repository does not sync | Git URL, branch, `base_dir`, Git-sync secret | | Depot connection fails | Depot name, credentials, permissions | | Plan fails | `config.yaml`, gateway, model defaults, migrations | | Run fails | Model SQL, dependencies, permissions, engine logs | | API fails | API resource allocation, service status, sidecar logs | | Product not visible | Metadata registration, product spec, target tenant | Once models, API, and discovery all check out, the product is live. Capture reusable patterns from the build in [Recipes](/build/v1/recipes/recipes.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/build/v1/productize/git-and-deploy.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.