# DataOS Documentation

## Home





- [Overview](https://v2.dataos.info/readme.md): Learn how to build, govern, and consume Data Products with DataOS.
- [DataOS Philosophy](https://v2.dataos.info/readme-1.md): DataOS is an operating system for building and running Data Products on the data you already have.
- [Data Product Journey](https://v2.dataos.info/data-product-journey-v2.md): From business question to trusted consumption

## Build





- [Introduction](https://v2.dataos.info/build/get-started/readme.md): What building a Data Product involves, the three stages, and where to start.
- [Prerequisites](https://v2.dataos.info/build/get-started/prerequisites.md): Get your environment ready before your first real build.
- [CLI setup](https://v2.dataos.info/build/get-started/prerequisites/cli-setup.md): Set up the DataOS CLI
- [LDK setup](https://v2.dataos.info/build/get-started/prerequisites/ldk-setup.md): Set up the Local Development Kit: Python virtual environment, Vulcan, and engine.
- [Repository setup](https://v2.dataos.info/build/get-started/prerequisites/repository-setup.md): Connect your Git repository so DataOS can pull your project at deploy time.
- [Quickstart](https://v2.dataos.info/build/get-started/quickstart.md): Build a small, working Data Product on your own machine: seeds, a transform, quality gates, a semantic model, and a metric. No DataOS deployment needed.
- [Build journey with AI](https://v2.dataos.info/build/get-started/build-journey-with-ai.md): Build a Data Product end to end by talking to an AI agent that uses builder skills and the Data Product MCP.
- [Understand Your Data](https://v2.dataos.info/build/understand/overview.md): Inspect your sources before you build. Decide from facts, not assumptions.
- [Connect sources](https://v2.dataos.info/build/understand/connect-sources.md): Register an authenticated connection to a source so DataOS can reach it without moving the data.
- [Scan metadata](https://v2.dataos.info/build/understand/scan-metadata.md): Scan a source so its tables, columns, lineage, and usage become discoverable in DataOS, without moving the underlying data.
- [Explore data](https://v2.dataos.info/build/understand/explore-data.md): Inspect the metadata, then the values, before you commit to a source.
- [Datasets](https://v2.dataos.info/build/understand/explore-data/datasets.md): Inspect dataset metadata, lineage, and usage in the catalog.
- [Workbench](https://v2.dataos.info/build/understand/explore-data/workbench.md): Connect to a source, run SQL, and inspect the actual values in Workbench.
- [Move Data](https://v2.dataos.info/build/move/overview.md): Bring raw data into reach when it is not already where your engine can read it.
- [Supported sources and destinations](https://v2.dataos.info/build/move/supported-sources-and-destinations.md): The sources Nilus can move data from, the destinations it can move data to, and which modes each supports.
- [Data movement](https://v2.dataos.info/build/move/mode.md): Pick the data movement type that fits your source and freshness need.
- [Batch](https://v2.dataos.info/build/move/mode/batch.md): Move data on a schedule or on demand with a Nilus batch pipeline.
- [CDC](https://v2.dataos.info/build/move/mode/cdc.md): Capture inserts, updates, and deletes from an operational database in near real-time with Nilus CDC.
- [Streaming](https://v2.dataos.info/build/move/mode/stream.md): Consume and land events continuously from Kafka or NATS with Nilus streaming.
- [Explore moved data](https://v2.dataos.info/build/move/explore-moved-data.md): Confirm the data landed correctly before you build on it.
- [Productize the Data](https://v2.dataos.info/build/productize/overview.md): "Productize" is the P in ELP: define the transform, semantic model, quality contract, and access policy as a single object, then deploy it.
- [Init and configuration](https://v2.dataos.info/build/productize/init-and-configuration.md): Scaffold the project and set config.yaml and usage.yaml.
- [Connect engine](https://v2.dataos.info/build/productize/connect-engine.md): Wire your project to the engine that runs transforms and lands outputs. Config only; the full per-engine manual is in References.
- [Snowflake](https://v2.dataos.info/build/productize/connect-engine/snowflake.md): Configure Snowflake as the engine for your Data Product.
- [Databricks](https://v2.dataos.info/build/productize/connect-engine/databricks.md): Configure Databricks as the engine for your Data Product.
- [Spark](https://v2.dataos.info/build/productize/connect-engine/spark.md): Configure Apache Spark as the engine for your Data Product.
- [Trino](https://v2.dataos.info/build/productize/connect-engine/trino.md): Connect Vulcan to Trino for federated SQL across catalogs and data lakes.
- [Dedicated Trino](https://v2.dataos.info/build/productize/connect-engine/trino/dedicated-trino.md): Ship a Trino cluster as part of the Data Product. Vulcan provisions and owns one cluster per product.
- [Minerva](https://v2.dataos.info/build/productize/connect-engine/trino/minerva.md): Connect Vulcan to DataOS Minerva, a shared Trino-compatible cluster managed by DataOS.
- [External Trino](https://v2.dataos.info/build/productize/connect-engine/trino/external-trino.md): Connect Vulcan to your own Trino-compatible endpoint, such as Starburst or self-hosted Trino.
- [Postgres](https://v2.dataos.info/build/productize/connect-engine/postgres.md): Configure PostgreSQL as the engine for your Data Product.
- [SQL Server](https://v2.dataos.info/build/productize/connect-engine/sql-server.md): Configure Microsoft SQL Server as the engine for your Data Product.
- [Microsoft Fabric](https://v2.dataos.info/build/productize/connect-engine/microsoft-fabric.md): Configure Microsoft Fabric Warehouse as the engine for your Data Product.
- [Amazon Redshift](https://v2.dataos.info/build/productize/connect-engine/redshift.md): Configure Amazon Redshift as the engine for your Data Product.
- [Amazon Athena](https://v2.dataos.info/build/productize/connect-engine/athena.md): Configure Amazon Athena as the engine for your Data Product.
- [Assets](https://v2.dataos.info/build/productize/assets.md): The components you create for a Data Product, and the order to build them.
- [Inputs](https://v2.dataos.info/build/productize/assets/inputs.md): Declare the upstream tables your models read from, as external models.
- [Seeds](https://v2.dataos.info/build/productize/assets/seeds.md): Load small static CSV reference data as warehouse tables with SEED models.
- [Data model](https://v2.dataos.info/build/productize/assets/data-model.md): Write the transforms that turn raw inputs into reliable output tables.
- [Semantic model](https://v2.dataos.info/build/productize/assets/semantic-model.md): Map physical columns to business measures, dimensions, segments, and joins so people and AI can query consistent definitions.
- [Metrics](https://v2.dataos.info/build/productize/assets/metrics.md): Combine a measure with a time column into a queryable time-series metric for dashboards and APIs.
- [Data quality](https://v2.dataos.info/build/productize/data-quality.md): Add DQ checks that monitor quality over time and profile columns.
- [Assertions](https://v2.dataos.info/build/productize/assertions.md): Block bad data at materialization. Audits define the rule; assertions attach it to a model and stop the run if it fails.
- [Unit Test](https://v2.dataos.info/build/productize/unit-test.md): Unit tests verify a model's SQL logic against mock fixtures, independent of any warehouse, before assertions or a real run ever see it.
- [Governance](https://v2.dataos.info/build/productize/governance.md): Control how the product runs and who can see which rows and columns. Access policy is created with the product, not bolted on.
- [Plan & run](https://v2.dataos.info/build/productize/plan-and-run.md): Validate the whole project locally with vulcan info, test, evaluate, and plan before anything ships.
- [Git & deploy](https://v2.dataos.info/build/productize/git-and-deploy.md): Deploy the Vulcan resource to DataOS, and verify the product is live and discoverable.
- [Monitor & delete](https://v2.dataos.info/build/productize/monitor-and-delete.md): Monitor a deployed Vulcan Data Product with the DataOS CLI, read its runtime container groups, and delete the deployed resource when it's no longer needed.
- [Recipes](https://v2.dataos.info/build/recipes.md): Proven patterns for common build scenarios. Copy, adapt, apply.
- [Reuse a Data Product](https://v2.dataos.info/build/recipes/reuse-data-products.md): Build a downstream product that reads another product's published outputs, instead of rebuilding the same logic from scratch.
- [Build a Python model for AI/ML workflows](https://v2.dataos.info/build/recipes/build-a-python-model.md): Write a Vulcan Python model that returns a DataFrame instead of SQL, for machine learning, forecasting, or logic that doesn't fit cleanly in SQL.
- [Cluster data with a Spark ML Python model](https://v2.dataos.info/build/recipes/spark-ml-python-model.md): Write a Vulcan Python model that clusters data with Spark MLlib when Spark is available, and falls back to scikit-learn when it isn't.
- [Wait for data to be ready with signals](https://v2.dataos.info/build/recipes/wait-for-data-with-signals.md): Make a scheduled Vulcan model wait for late-arriving source data instead of running on a fixed clock and computing an incomplete result.
- [Route plan and run failures to Slack or Teams](https://v2.dataos.info/build/recipes/route-failures-to-slack-or-teams.md): Once a data product is published, wire Vulcan's notification\_targets so a failed plan or run pages Slack or Teams automatically, instead of waiting for someone to notice it in the logs.
- [Track dimension history with SCD Type 2](https://v2.dataos.info/build/recipes/track-dimension-history-with-scd-type-2.md): Convert an overwrite-style dimension model to SCD\_TYPE\_2 so changes are versioned with valid\_from/valid\_to instead of lost on the next run.
- [Mask sensitive columns in a semantic model by role](https://v2.dataos.info/build/recipes/mask-sensitive-columns-by-role.md): Mask sensitive columns in a physical model per caller role using an access policy, column\_mask\_expressions, and an auth extension that maps DataOS roles to policy groups: no forked pipelines or models

## Consume





- [Consumer journey](https://v2.dataos.info/consume/get-started/readme.md): Learn how a consumer goes from finding a Data Product to using it through BI tools, APIs, or AI workflows.
- [Before you begin](https://v2.dataos.info/consume/get-started/before-you-begin.md): Learn what you need to consume a Data Product: tenant access, a deployed product, and an API token for external tools.
- [Discover Data Products](https://v2.dataos.info/consume/discover/overview.md): Learn how to find the right Data Product, perspective, or metric before you dig into details.
- [Search the Hub](https://v2.dataos.info/consume/discover/search-the-hub.md): Learn to search, filter, preview, and open Data Products, perspectives, and metrics in the Hub.
- [Filters](https://v2.dataos.info/consume/discover/search-the-hub/filters.md): Narrow the Hub by domain, type, quality, last-run status, tags, and members.
- [Perspectives](https://v2.dataos.info/consume/discover/search-the-hub/perspectives.md): Learn to browse, filter, and preview the saved, governed queries available in a tenant.
- [Metrics](https://v2.dataos.info/consume/discover/search-the-hub/metrics.md): Browse, filter, and preview the computed business measures available in a tenant.
- [Evaluate a Data Product](https://v2.dataos.info/consume/evaluate/overview.md): Review a Data Product's context, assets, and trust signals before you query or consume it.
- [Explore assets](https://v2.dataos.info/consume/evaluate/assets.md): Explore what a Data Product is built from, what it publishes, and how data flows through it.
- [Assets overview](https://v2.dataos.info/consume/evaluate/assets/overview.md): Trace data from inputs to outputs, semantic models, and metrics in the asset graph.
- [Seeds](https://v2.dataos.info/consume/evaluate/assets/seeds.md): Static reference datasets used to enrich a Data Product's outputs.
- [Inputs](https://v2.dataos.info/consume/evaluate/assets/inputs.md): Inspect the source datasets, raw fields, and lineage a Data Product reads from.
- [Outputs](https://v2.dataos.info/consume/evaluate/assets/outputs.md): Review the curated datasets a Data Product publishes for downstream use.
- [Semantics](https://v2.dataos.info/consume/evaluate/assets/semantics.md): Review the dimensions, measures, and relationships exposed for business queries.
- [Metrics](https://v2.dataos.info/consume/evaluate/assets/metrics.md): Review the governed business measures a Data Product exposes and the models behind them.
- [Query in Studio](https://v2.dataos.info/consume/evaluate/query-in-studio.md): Query a Data Product's semantic models in Studio to confirm it exposes the right data.
- [Save as Perspective](https://v2.dataos.info/consume/evaluate/query-in-studio/save-as-perspective.md): Save a Studio query as a perspective to reuse, share, and serve through a Data API.
- [Trust and freshness](https://v2.dataos.info/consume/evaluate/trust-and-freshness.md): Check a Data Product's quality, freshness, and AI-readiness before relying on it.
- [Track activity](https://v2.dataos.info/consume/evaluate/track-activity.md): Review a Data Product's runs and version history to judge execution health and recent changes.
- [Activate a Data Product](https://v2.dataos.info/consume/activate/overview.md): Choose the path to consume a Data Product: BI tools, SQL clients, APIs, or AI.
- [Querying Data Products](https://v2.dataos.info/consume/activate/querying-data-products.md): How querying a Data Product works, and where each query path leads.
- [BI tools](https://v2.dataos.info/consume/activate/bi-tools.md): Use Data Products in BI and analytics tools such as Power BI and Tableau Desktop.
- [Tableau Desktop](https://v2.dataos.info/consume/activate/bi-tools/tableau.md): Connect Tableau Desktop to turn your dimensions and measures into charts, dashboards, and drill-downs - no SQL required.
- [Tableau Cloud](https://v2.dataos.info/consume/activate/bi-tools/tableau-cloud.md): This guide covers connecting Tableau Cloud to Data Products, publishing semantic models, and creating workbooks.
- [Power BI Desktop](https://v2.dataos.info/consume/activate/bi-tools/desktop.md): Connect Power BI Desktop to turn your dimensions and measures into charts, dashboards, and drill-downs - no SQL required.
- [Power BI Service](https://v2.dataos.info/consume/activate/bi-tools/service.md): Publish Power BI reports to Power BI Service and connect them to DataOS through an on-premises gateway.
- [APIs](https://v2.dataos.info/consume/activate/apis.md): Connect applications and automation to Data Products with GraphQL, semantic query, metadata, activity, and quality APIs.
- [Collection](https://v2.dataos.info/consume/activate/apis/collection.md): Import the Postman collection for a Data Product and test API requests fast.
- [SDK](https://v2.dataos.info/consume/activate/apis/sdk.md): Build applications and automations on top of Data Product APIs with typed SDKs.
- [Python](https://v2.dataos.info/consume/activate/apis/sdk/python-sdk.md): Install and use the Python SDK for Data Product APIs.
- [TypeScript](https://v2.dataos.info/consume/activate/apis/sdk/typescript-sdk.md): Install and use the TypeScript SDK for Data Product APIs.
- [Data application](https://v2.dataos.info/consume/activate/apis/data-application.md): Register and run an application built on top of a Data Product's APIs.
- [Deploy](https://v2.dataos.info/consume/activate/apis/data-application/deploy.md): Deploy a containerized Data Product application in DataOS using an app Resource.
- [Link](https://v2.dataos.info/consume/activate/apis/data-application/link.md): Surface an externally hosted Data Product application using a link Resource.
- [Database clients](https://v2.dataos.info/consume/activate/database-clients.md): Connect a desktop SQL client to a Data Product and query its semantic models through the MySQL wire protocol.
- [MySQL](https://v2.dataos.info/consume/activate/database-clients/mysql.md): Connect a command-line MySQL client to a Data Product and query its semantic models.
- [DBeaver](https://v2.dataos.info/consume/activate/database-clients/dbeaver.md): Connect DBeaver to a Data Product and query semantic models over the MySQL protocol.
- [Reusability](https://v2.dataos.info/consume/activate/reusability.md): Reuse a published Data Product as the input for a new one instead of rebuilding it.
- [Consume with AI](https://v2.dataos.info/consume/consume-with-ai/overview.md): Use AI assistants and agents with Data Product MCP to discover, evaluate, and query governed data.
- [Connect a client](https://v2.dataos.info/consume/consume-with-ai/connect-a-client.md): Connect an MCP-compatible AI client to Data Product MCP.
- [Claude](https://v2.dataos.info/consume/consume-with-ai/connect-a-client/claude.md): Configure Claude Code or Claude Desktop to connect with Data Product MCP.
- [Cursor](https://v2.dataos.info/consume/consume-with-ai/connect-a-client/cursor.md): Connect Cursor to Data Product MCP for AI-assisted discovery and queries.
- [Copilot in VS Code](https://v2.dataos.info/consume/consume-with-ai/connect-a-client/copilot-vs-code.md): Configure GitHub Copilot in VS Code to use Data Product MCP.
- [ChatGPT Desktop (Codex)](https://v2.dataos.info/consume/consume-with-ai/connect-a-client/codex-desktop.md): Configure ChatGPT Desktop (Codex) to use Data Product MCP.
- [Custom](https://v2.dataos.info/consume/consume-with-ai/connect-a-client/custom.md): Connect any MCP-compatible LLM tool to Data Product MCP.
- [Connect an agentic framework](https://v2.dataos.info/consume/consume-with-ai/connect-an-agentic-framework.md): Connect agentic frameworks to Data Product MCP for governed tool access.
- [LangChain](https://v2.dataos.info/consume/consume-with-ai/connect-an-agentic-framework/langchain.md): Connect LangChain workflows to Data Product MCP for governed tools.
- [Vercel AI SDK](https://v2.dataos.info/consume/consume-with-ai/connect-an-agentic-framework/vercel-ai-sdk.md): Connect Vercel AI SDK applications to Data Product MCP tools.
- [AgentScope](https://v2.dataos.info/consume/consume-with-ai/connect-an-agentic-framework/agentscope.md): Connect AgentScope applications to Data Product MCP with an HTTP client.
- [Custom](https://v2.dataos.info/consume/consume-with-ai/connect-an-agentic-framework/custom.md): Connect any agentic framework to Data Product MCP with the generic configuration.
- [Discover in natural language](https://v2.dataos.info/consume/consume-with-ai/discover-in-natural-language.md): Find the right Data Product by asking about topics, domains, metrics, or ownership in natural language.
- [Understand in natural language](https://v2.dataos.info/consume/consume-with-ai/understand-in-natural-language.md): Inspect a Data Product's schema, lineage, profile, quality, and runs before you rely on it, through conversation.
- [Recipes](https://v2.dataos.info/consume/recipes.md): Step-by-step, end-to-end walkthroughs for common consumer tasks.
- [Discovering the right Data Product in natural language](https://v2.dataos.info/consume/recipes/find-a-data-product.md): Go from a business question to a vetted Data Product: discover candidates, check quality and runs, confirm ownership.
- [Answering KPI questions in natural language](https://v2.dataos.info/consume/recipes/answer-a-kpi-question.md): Turn a quarterly KPI question into a governed, cited answer: inspect the schema, run the query, verify the metric and source.
- [Deploy a data application on a Perspective](https://v2.dataos.info/consume/recipes/deploy-a-data-application-on-a-perspective.md): Turn a saved query into a self-service app: save it as a Perspective, deploy a Data Application against its Data API, confirm it inherits governed access.
- [Build applications with Semantic GraphQL](https://v2.dataos.info/consume/recipes/query-semantic-model-with-graphql.md): Query specific measures and dimensions through the Semantic GraphQL endpoint and get a structured response back in one request.
- [Retrieve large result sets with pagination](https://v2.dataos.info/consume/recipes/retrieve-large-result-sets-with-pagination.md): Page through Semantic REST result sets larger than 50,000 records using limit, offset, and a deterministic order.
- [Generate native warehouse SQL](https://v2.dataos.info/consume/recipes/transpile-semantic-queries-to-native-warehouse-sql.md): Use transpile\_only=true to inspect the native warehouse SQL generated from a Semantic SQL or Semantic REST query, without executing it.

## Operate





- [Introduction](https://v2.dataos.info/operate/readme.md): How DataOS runs as one governed platform, and the five roles that operate and maintain it.
- [Architecture](https://v2.dataos.info/operate/architecture.md): DataOS follows the Control Plane–Data Plane architecture pattern, where the DataOS Instance acts as the Control Plane.
- [Authentication & Authorization](https://v2.dataos.info/operate/authentication-and-authorization.md): Connect DataOS to your identity provider for authentication, and understand how authorization is evaluated after sign-in.
- [Access control](https://v2.dataos.info/operate/access-control.md): How DataOS controls access through access levels, roles, permissions, explicit grants, resource dependencies, and runtime identity.
- [Roles & Permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions.md): How DataOS decides who can do what: roles, the explicit-grant model, and running resources as another user.
- [Resource-level permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions.md): This page includes permissions specific to each resource within DataOS
- [Compute: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/compute-access-control.md): Who can create a Compute Resource, and how access to it is managed.
- [Depot: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/depot-access-control.md): Who can create a Depot, and how access to it and its data is managed.
- [Lakehouse: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/lakehouse-access-control.md): Who can create a Lakehouse, what permissions it depends on, and how access is managed once it exists.
- [Minerva Cluster: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/minerva-cluster-access-control.md): Who can create a Minerva Cluster, what permissions it depends on, and how access is managed once it exists.
- [Nilus: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/nilus-access-control.md): Who can create a Nilus resource, what it depends on, and how access is managed once it exists.
- [Secret: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/secret-access-control.md): Who can create a Secret, and how access to it is managed.
- [Service: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/service-access-control.md): Who can create a Service, what permissions it depends on, and how access is managed once it exists.
- [Vulcan: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/vulcan-access-control.md): Who can create a Vulcan resource, what it depends on, and how access is managed once it exists.
- [Worker: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/worker-access-control.md): Who can create a Worker, what it depends on, and how access is managed once it exists.
- [Workflow: permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/workflow-access-control.md): Who can create a Workflow, what it depends on, and how access is managed once it exists.
- [RunAsUser permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/runasuser-permissions.md): Run a resource under another user's identity without changing ownership or submitter identity.
- [Create an application user](https://v2.dataos.info/operate/access-control/roles-and-permissions/runasuser-permissions/create-an-application-user.md): Create a non-personal application identity in DataOS.
- [Grant permissions](https://v2.dataos.info/operate/access-control/roles-and-permissions/runasuser-permissions/grant-permissions.md): Grant an application user the permissions it needs, then grant developers RunAsUser to act under its identity.
- [Observability](https://v2.dataos.info/operate/observability-model.md): The layers of monitoring in DataOS, and which signals each role watches.
- [DataOS Instance Status](https://v2.dataos.info/operate/instance-status.md): Check the status of DataOS Instance services from the Instance Status page.
- [Tenant readiness](https://v2.dataos.info/operate/tenant-readiness.md): The end-to-end sequence that makes a Tenant ready to use, and which role owns each step.
- [CLI command references](https://v2.dataos.info/operate/cli.md): The dataos-ctl commands Operators and Tenant Admins use to provision, run, and govern a Tenant from the terminal.
- [DataOS Operator overview](https://v2.dataos.info/operate/dataos-operator/operator.md): End-to-end Operator workflow from infrastructure deployment to Tenant readiness.
- [Provision Data Plane](https://v2.dataos.info/operate/dataos-operator/provision-data-plane.md): Provision the Data Plane, the execution layer in the customer cloud: prerequisites, the sequence, and the per-cloud runbooks.
- [AWS](https://v2.dataos.info/operate/dataos-operator/provision-data-plane/aws.md): Provision AWS Data Plane
- [AWS (UI)](https://v2.dataos.info/operate/dataos-operator/provision-data-plane/aws-ui.md): Provision an AWS BYOC Data Plane from the DataOS UI
- [Azure](https://v2.dataos.info/operate/dataos-operator/provision-data-plane/azure.md): Provision Azure Data Plane
- [Azure (UI)](https://v2.dataos.info/operate/dataos-operator/provision-data-plane/azure-ui.md): Provision an Azure BYOC Data Plane from the DataOS UI
- [Design Tenants](https://v2.dataos.info/operate/dataos-operator/design-tenants.md): How to slice Tenants before you create them: the common blueprints and a framework for choosing.
- [Choosing between an Instance and a Tenant](https://v2.dataos.info/operate/dataos-operator/choosing-between-instance-and-tenant.md): Decide whether a new requirement needs a separate DataOS Instance or a Tenant in an existing one.
- [Create & set up Tenant](https://v2.dataos.info/operate/dataos-operator/create-tenants.md): Create a Tenant, attach the Data Plane and Compute, install Stacks, and make it ready for a team to deploy Data Products.
- [Deploy Stacks](https://v2.dataos.info/operate/dataos-operator/create-tenants/deploy-stacks.md): Deploy the Lakehouse MRuntime Stack, the Data Product MCP server, and its OAuth broker for a Tenant.
- [Hand off Tenant](https://v2.dataos.info/operate/dataos-operator/create-tenants/hand-off-tenant.md): Confirm the Tenant is ready, assign a Tenant Admin, and hand day-to-day operation over to them.
- [Manage access](https://v2.dataos.info/operate/dataos-operator/governance.md): What the Operator governs: assigning Tenant Admins, keeping Tenant roles consistent, and auditing high-privilege access.
- [Monitoring](https://v2.dataos.info/operate/dataos-operator/monitoring.md): Watch the Instance Status page and Data Plane health from the Operator's seat.
- [Tenant Admin overview](https://v2.dataos.info/operate/tenant-admin/tenant-admin.md): The role that runs a Tenant day to day: resources, access, governance, and health.
- [Create Compute](https://v2.dataos.info/operate/tenant-admin/create-compute.md): Provision additional Compute (node pools) for Tenant Data Product workloads.
- [Add Depots & Secrets](https://v2.dataos.info/operate/tenant-admin/add-data-sources-and-secrets.md): Connect the Tenant to external systems with Secrets and Depots.
- [Manage access](https://v2.dataos.info/operate/tenant-admin/governance.md): Grant developers access to Compute and data sources, onboard users, and set up the ApplicationUser and runAsUser mechanics.
- [Monitoring](https://v2.dataos.info/operate/tenant-admin/monitoring.md): Watch the Tenant's Compute and Resource health, and how to check it.
- [Data Admin overview](https://v2.dataos.info/operate/data-admin/data-admin.md): The role that owns and governs the Lakehouse.
- [Manage Lakehouse](https://v2.dataos.info/operate/data-admin/manage-lakehouse.md): Provision the Lakehouse and manage its namespaces, schemas, tables, and views.
- [Manage Lakehouse data access](https://v2.dataos.info/operate/data-admin/access-control.md): Govern who can read and write Lakehouse data.
- [Monitoring](https://v2.dataos.info/operate/data-admin/monitoring.md): Watch Lakehouse health and usage.
- [Data Developer overview](https://v2.dataos.info/operate/data-developer/data-product-developer.md): The role that builds and operates Data Products inside a Tenant.
- [Get access](https://v2.dataos.info/operate/data-developer/access-and-readiness.md): The roles and grants a Data Developer needs, and the checklist that confirms the Tenant can run what you build.
- [Deploy and promote across environments](https://v2.dataos.info/operate/data-developer/deploy-and-promote.md): The operational path from a dev Tenant to production.
- [Manage Data Product access](https://v2.dataos.info/operate/data-developer/manage-data-product-access.md): The access policies your product enforces, and who you let consume it.
- [Monitor Data Product](https://v2.dataos.info/operate/data-developer/monitor-data-product.md): Keep a live product healthy: runs, failures, versioning, and cost.
- [CLI command reference: Data Developer](https://v2.dataos.info/operate/data-developer/cli-dev-cheatsheet.md): One-page dataos-ctl command reference for the Data Developer persona.
- [Data Consumer overview](https://v2.dataos.info/operate/data-consumer/data-product-consumer.md): The read-only role that consumes approved Data Products.
- [Get access](https://v2.dataos.info/operate/data-consumer/get-access.md): Request access to a Tenant and the products you need, and understand read-only scope.
- [Consumption paths](https://v2.dataos.info/operate/data-consumer/modes-of-consumption.md): The paths for consuming a product, and how to choose one.
- [Governance enforced on you](https://v2.dataos.info/operate/data-consumer/governance-enforced.md): Why you see what you see, and how to ask for more.
- [Trust and monitor](https://v2.dataos.info/operate/data-consumer/trust-and-monitor.md): The signals to check before you rely on a Data Product.
- [Raise issues and track changes](https://v2.dataos.info/operate/data-consumer/support-and-changes.md): Reach the owner, raise issues, and stay ahead of breaking changes.
- [Overview](https://v2.dataos.info/operate/data-steward/overview.md): How Data Stewards update business glossary descriptions.
- [Recipes](https://v2.dataos.info/operate/recipes.md): Step-by-step, end-to-end walkthroughs for common Operator/Tenant Admin tasks.
- [Provide Resource permissions to Developers](https://v2.dataos.info/operate/recipes/how-to-give-access.md): Grant a Data Developer the Can Use permissions their Data Product needs: Compute, a Snowflake Depot, its linked Secret, and a Cluster.
- [Set a Data Product's runtime identity](https://v2.dataos.info/operate/recipes/how-to-set-a-data-products-runtime-identity.md): Give a deployed Data Product a stable runtime identity. An ApplicationUser referenced through runAsUser reaches its Depots, Secrets, Compute, and Cluster without depending on one person's access.

## References





- [DataOS Reference](https://v2.dataos.info/references/readme.md): The full reference layer of DataOS: per-engine manuals, complete DataOS resource syntax, and interface specifications.
- [Engine Reference Guide](https://v2.dataos.info/references/engine-guide/engine-guide.md): The full per-engine technical reference for building and running a Data Product on each supported Vulcan engine in DataOS.
- [Snowflake](https://v2.dataos.info/references/engine-guide/snowflake.md): The full technical reference for building and running a Vulcan Data Product on Snowflake, covering connection, grants, materialization, operations, performance, and troubleshooting.
- [Permissions and grants](https://v2.dataos.info/references/engine-guide/snowflake/permissions-and-grants.md): Configure required Snowflake roles, grants, and DataOS access for Vulcan deployments.
- [Connect to Snowflake](https://v2.dataos.info/references/engine-guide/snowflake/connect-to-snowflake.md): Connect Vulcan to Snowflake using direct authentication or a DataOS Depot.
- [Materialization behavior](https://v2.dataos.info/references/engine-guide/snowflake/materialization-behavior.md): Understand how Vulcan model kinds map to native Snowflake operations.
- [Limits & Performance](https://v2.dataos.info/references/engine-guide/snowflake/limits-and-performance.md): Size Snowflake compute, control concurrency, manage costs, and plan for performance.
- [Troubleshooting](https://v2.dataos.info/references/engine-guide/snowflake/troubleshooting.md): Diagnose common Snowflake and Vulcan failures, then recover production workloads.
- [Recipes](https://v2.dataos.info/references/engine-guide/snowflake/recipes.md): Implement common Snowflake patterns with production-ready Vulcan recipes.
- [Dynamic Tables and Access Policies](https://v2.dataos.info/references/engine-guide/snowflake/recipes/build-a-snowflake-powered-data-product.md): Create a Snowflake Data Product with a Dynamic Table, column masking, and row access policies using Vulcan.
- [Use Snowpark Python Models](https://v2.dataos.info/references/engine-guide/snowflake/recipes/use-snowpark-python-models.md): Build and run an RFM customer-segmentation model with Snowpark Python on Snowflake.
- [Databricks](https://v2.dataos.info/references/engine-guide/databricks.md): The full technical reference for building and running a Vulcan Data Product on Databricks, covering connection, Unity Catalog grants, materialization, state store, operations, and troubleshooting.
- [Permissions and grants](https://v2.dataos.info/references/engine-guide/databricks/permissions-and-grants.md): Know about Unity Catalog grants and DataOS access for Databricks deployments.
- [Connect to Databricks](https://v2.dataos.info/references/engine-guide/databricks/connect-to-databricks.md): Configure direct and depot connections, external state storage, local development, and production deployment.
- [Materialization behavior](https://v2.dataos.info/references/engine-guide/databricks/materialization-behavior.md): Understand how Vulcan model kinds map to Delta Lake operations on Databricks.
- [Limits & Performance](https://v2.dataos.info/references/engine-guide/databricks/limits-and-performance.md): Configure Databricks warehouse sizing, concurrency, cost guardrails, and performance baselines for reliable production workloads.
- [Troubleshooting](https://v2.dataos.info/references/engine-guide/databricks/troubleshooting-databricks.md): Diagnose Databricks connection, warehouse, and state-store failures.
- [Postgres](https://v2.dataos.info/references/engine-guide/postgres.md): Choose, configure, build, serve, and operate PostgreSQL-backed Vulcan Data Products.
- [Permissions and grants](https://v2.dataos.info/references/engine-guide/postgres/permissions-and-grants.md): Provision PostgreSQL and DataOS access before configuring or deploying a Data Product.
- [Connect to Postgres](https://v2.dataos.info/references/engine-guide/postgres/connect-to-postgres.md): Configure PostgreSQL gateways, develop locally with Docker, and prepare production deployment.
- [Materialization behavior](https://v2.dataos.info/references/engine-guide/postgres/materialization-behavior.md): Choose PostgreSQL model strategies and apply engine-specific model behavior.
- [Limits & performance](https://v2.dataos.info/references/engine-guide/postgres/limits-and-performance.md): Size, schedule, and tune PostgreSQL workloads for reliable production operations.
- [Troubleshooting](https://v2.dataos.info/references/engine-guide/postgres/troubleshooting.md): Diagnose PostgreSQL connection, permissions, model, and runtime failures.
- [Recipes](https://v2.dataos.info/references/engine-guide/postgres/recipes.md): Implement common PostgreSQL patterns with Vulcan.
- [Full-Text Search and Row-Level Security](https://v2.dataos.info/references/engine-guide/postgres/recipes/full-text-search-and-row-level-security.md): Build PostgreSQL search and access controls with full-text search and row-level security.
- [Spark](https://v2.dataos.info/references/engine-guide/spark-1.md): Technical guide to building and running a Vulcan Data Product on Apache Spark with Iceberg: connection setup, materialization, tuning, and troubleshooting.
- [Permissions and grants](https://v2.dataos.info/references/engine-guide/spark-1/permissions-and-grants.md): Provision DataOS, lakehouse, compute, state-store, storage, and Git access for Spark deployments.
- [Connect to Spark](https://v2.dataos.info/references/engine-guide/spark-1/connection-options.md): Configure the Vulcan gateway connection to Spark and load the JVM and   Python dependencies the driver and executors need.
- [Spark lakehouse limitations](https://v2.dataos.info/references/engine-guide/spark-1/spark-lakehouse-limitations.md): Constraints for Vulcan Spark and Iceberg lakehouse workloads.
- [Materialization behaviour](https://v2.dataos.info/references/engine-guide/spark-1/materialization-behaviour.md): Understand how Vulcan model kinds map to native Spark/Iceberg operations,   and how the semantic layer and business metrics build on top of them.
- [Limits & Performance](https://v2.dataos.info/references/engine-guide/spark-1/limits-and-performance.md): Size Spark workloads, tune execution, and set performance and cost guardrails.
- [Troubleshooting](https://v2.dataos.info/references/engine-guide/spark-1/troubleshooting.md): Diagnose Spark, Iceberg, dependency, deployment, and runtime failures.
- [Recipes](https://v2.dataos.info/references/engine-guide/spark-1/recipes.md): Implement common Spark patterns with production-ready Vulcan recipes.
- [Deployment recipes](https://v2.dataos.info/references/engine-guide/spark-1/recipes/deployment-recipes.md): Ready-to-copy sparkConf and resource sizing for common Spark deployment   shapes: daily incremental, medium lakehouse mart, large backfill, and   MERGE-heavy CDC.
- [Trino](https://v2.dataos.info/references/engine-guide/trino-1.md): The full technical reference for running a Vulcan Data Product on Trino, with three deployment shapes (dedicated managed Trino cluster, shared Minerva, and bring-your-own external Trino).
- [Dedicated Trino](https://v2.dataos.info/references/engine-guide/trino-1/dedicated-cluster.md): The full reference for the dedicated managed Trino cluster (vulcan-dg-trino), one cluster per Data Product, covering the deploy manifest, catalog priority, JVM tuning, and failure modes.
- [Cluster sizing and operations](https://v2.dataos.info/references/engine-guide/trino-1/dedicated-cluster/dedicated-cluster-sizing-and-operations.md): Configure fixed worker capacity, write targets, fault handling, and JVM memory for dedicated Trino.
- [Minerva](https://v2.dataos.info/references/engine-guide/trino-1/minerva-1.md): The full reference for connecting Vulcan to a shared, DataOS-managed Minerva (Trino) cluster, covering the secret and password generation, projection into config, and the connection setup.
- [External Trino](https://v2.dataos.info/references/engine-guide/trino-1/external-trino.md): The full reference for connecting Vulcan to a bring-your-own Trino, Starburst, or self-hosted cluster that exists outside DataOS, covering connection options, authentication, and catalog notes.
- [Starburst](https://v2.dataos.info/references/engine-guide/trino-1/external-trino/starburst.md): Build and run a Vulcan Data Product on Starburst Galaxy using a Trino-compatible external cluster, covering connection, deployment, materialization, operations, and troubleshooting.
- [Capabilities and limitations](https://v2.dataos.info/references/engine-guide/trino-1/capabilities-and-limitations.md): Confirm Trino catalog compatibility, model support, and view security before deployment.
- [SQL Server](https://v2.dataos.info/references/engine-guide/sql-server.md): The full technical reference for building and running a Vulcan Data Product on Microsoft SQL Server, covering connection, grants, materialization, local Docker setup, and troubleshooting.
- [Permissions and grants](https://v2.dataos.info/references/engine-guide/sql-server/permissions-and-grants.md): Provision SQL Server and DataOS access before configuring or deploying a Data Product.
- [Connect to SQL Server](https://v2.dataos.info/references/engine-guide/sql-server/connect-to-sql-server.md): Configure SQL Server gateways, develop locally with Docker, and prepare production deployment.
- [Materialization behavior](https://v2.dataos.info/references/engine-guide/sql-server/materialization-behavior.md): Choose SQL Server model strategies and apply engine-specific model behavior.
- [Limits & performance](https://v2.dataos.info/references/engine-guide/sql-server/limits-and-performance.md): Size, schedule, and tune SQL Server workloads for reliable production operations.
- [Troubleshooting](https://v2.dataos.info/references/engine-guide/sql-server/troubleshooting.md): Diagnose SQL Server connection, permissions, model, and runtime failures.
- [Microsoft Fabric](https://v2.dataos.info/references/engine-guide/microsoft-fabric.md): Technical reference for building and running Vulcan Data Products on Microsoft Fabric Warehouse, covering connection, authentication, materialization, and troubleshooting.
- [Permissions and grants](https://v2.dataos.info/references/engine-guide/microsoft-fabric/permissions-and-grants.md): Provision Fabric and DataOS access before configuring or deploying a Data Product.
- [Connect to Microsoft Fabric](https://v2.dataos.info/references/engine-guide/microsoft-fabric/connect-to-microsoft-fabric.md): Configure a Fabric gateway, develop against a dev warehouse, and prepare production deployment.
- [Materialization behavior](https://v2.dataos.info/references/engine-guide/microsoft-fabric/materialization-behavior.md): Choose Fabric model kinds and understand their native Warehouse behavior.
- [Limits & performance](https://v2.dataos.info/references/engine-guide/microsoft-fabric/limits-and-performance.md): Set Fabric capacity, concurrency, and workload guardrails for reliable operations.
- [Troubleshooting](https://v2.dataos.info/references/engine-guide/microsoft-fabric/troubleshooting.md): Diagnose Fabric connection, warehouse, materialization, and endpoint failures.
- [Redshift](https://v2.dataos.info/references/engine-guide/redshift.md): The full technical reference for building and running a Vulcan Data Product on Amazon Redshift, covering connection, grants, materialization, operations, performance, and troubleshooting.
- [Permissions and Grants](https://v2.dataos.info/references/engine-guide/redshift/permissions-and-grants.md): Set up Redshift roles, grants, and DataOS access for Vulcan.
- [Connect to Redshift](https://v2.dataos.info/references/engine-guide/redshift/connect-to-redshift.md): Configure Redshift connections for local development and DataOS Depot deployments.
- [Materialization Behavior](https://v2.dataos.info/references/engine-guide/redshift/materialization-behavior.md): Choose materialization strategies for Redshift models and semantic layers.
- [Limits and Performance](https://v2.dataos.info/references/engine-guide/redshift/limits-and-performance.md): Set Redshift resource, concurrency, and performance guardrails.
- [Troubleshooting](https://v2.dataos.info/references/engine-guide/redshift/troubleshooting.md): Diagnose and recover from common Redshift deployment and runtime failures.
- [Recipes](https://v2.dataos.info/references/engine-guide/redshift/recipes.md): Implement common Redshift patterns with production-ready Vulcan recipes.
- [Deployment Recipes](https://v2.dataos.info/references/engine-guide/redshift/recipes/deployment-recipes.md): Deploy Vulcan projects to Redshift using DataOS resources.
- [Athena](https://v2.dataos.info/references/engine-guide/athena.md): The full technical reference for building and running a Vulcan Data Product on Amazon Athena, covering connection, IAM permissions, materialization, operations, performance, and troubleshooting.
- [Permissions and grants](https://v2.dataos.info/references/engine-guide/athena/permissions-and-grants.md): Configure required Athena IAM roles, permissions, and DataOS access for Vulcan deployments.
- [Connect to Athena](https://v2.dataos.info/references/engine-guide/athena/connect-to-athena.md): Configure Athena gateways, local development, and production deployment for Vulcan.
- [Materialization behavior](https://v2.dataos.info/references/engine-guide/athena/materialization-behavior.md): Understand how Vulcan model kinds map to native Athena operations.
- [Limits and performance](https://v2.dataos.info/references/engine-guide/athena/limits-and-performance.md): Size Athena workloads, control concurrency, manage costs, and plan for performance.
- [Troubleshooting](https://v2.dataos.info/references/engine-guide/athena/troubleshooting.md): Diagnose common Athena and Vulcan failures, then recover production workloads.
- [Recipes](https://v2.dataos.info/references/engine-guide/athena/recipes.md): Implement common Athena patterns with production-ready Vulcan recipes.
- [Deployment recipes](https://v2.dataos.info/references/engine-guide/athena/recipes/deployment-recipes.md): Implement common Athena patterns with production-ready Vulcan recipes.
- [Understand DataOS Resources](https://v2.dataos.info/references/resources/overview.md): Understand what DataOS Resources are, how they are structured, and how to deploy and manage them.
- [Vulcan](https://v2.dataos.info/references/resources/vulcan.md): Understand what Vulcan is, how it builds governed data products, and how it runs transformation, quality, and semantics on your existing engine.
- [LDK setup](https://v2.dataos.info/references/resources/vulcan/ldk.md): Set up the Local Development Kit: Python virtual environment, Vulcan, and engine.
- [Concepts](https://v2.dataos.info/references/resources/vulcan/concepts.md): How Vulcan is put together, how a data product moves from setup to a governed asset, and how \`vulcan plan\` and \`vulcan run\` decide what to execute.
- [Architecture](https://v2.dataos.info/references/resources/vulcan/concepts/architecture.md): How Vulcan is structured: the fingerprint-plan-apply flow, the model materialization strategies, and the extensible engine adapter model.
- [Data Product lifecycle](https://v2.dataos.info/references/resources/vulcan/concepts/data-product-lifecycle.md): The full path from local setup to a production data product: project initialization, model development, testing, semantics, planning and running, data quality, API access, and monitoring.
- [State management](https://v2.dataos.info/references/resources/vulcan/concepts/state-management.md): How Vulcan remembers what it has already done: the PostgreSQL state store, the object store cache, and the two Secrets that connect them.
- [Run and plan](https://v2.dataos.info/references/resources/vulcan/concepts/run-and-plan.md): Understand what \`vulcan plan\` and \`vulcan run\` do before you use them.
- [Plan without a virtual layer](https://v2.dataos.info/references/resources/vulcan/concepts/run-and-plan/plan-guide.md): Understand what \`vulcan plan\` does without a virtual layer, using direct materialization instead of virtual-layer promotion.
- [Plan with a virtual layer](https://v2.dataos.info/references/resources/vulcan/concepts/run-and-plan/plan-with-vde.md): What happens when \`vulcan plan\` runs with a virtual layer: snapshot fingerprints, the context diff, breaking vs. non-breaking changes, backfill, restatement, and virtual-layer promotion.
- [Run and scheduling](https://v2.dataos.info/references/resources/vulcan/concepts/run-and-plan/run-and-scheduling.md): Understand what \`vulcan run\` does to refresh already-applied data: missing intervals, schedules, signals, and run activity.
- [Write-Audit-Publish](https://v2.dataos.info/references/resources/vulcan/concepts/write-audit-publish.md): How Write-Audit-Publish keeps bad data out of production: write to the model, run blocking assertions, publish only if they pass.
- [Query result caching](https://v2.dataos.info/references/resources/vulcan/concepts/query-result-caching.md): How Vulcan reuses results of identical SQL, semantic, and metric queries while the underlying data is current.
- [Configurations](https://v2.dataos.info/references/resources/vulcan/configurations.md): Configure a Vulcan project with config.yaml, usage.yaml, and agreement.md: project settings, gateways, model defaults, the semantic layer, variables, execution hooks, the linter, notifications, deploy
- [Usage guidance](https://v2.dataos.info/references/resources/vulcan/configurations/usage.md): What usage.yaml is for, how it differs from config.yaml, and a full key-by-key reference: what each key documents, when to add it, and when to leave the file minimal or omit it entirely.
- [Agreement](https://v2.dataos.info/references/resources/vulcan/configurations/agreement.md): What agreement.md is for, how it differs from config.yaml and usage.yaml, and how Vulcan resolves, reads, and surfaces it to consumers.
- [Variables](https://v2.dataos.info/references/resources/vulcan/configurations/variables.md): Configure Vulcan variables and environment overrides: .env files, the env\_var() function, VULCAN\_\_ configuration overrides, and dynamic values like the user() function.
- [Model defaults](https://v2.dataos.info/references/resources/vulcan/configurations/model-defaults.md): Configure model\_defaults in config.yaml: the required dialect, owner enforcement against config.users, identifier normalization strategy, and gateway-specific overrides.
- [Execution hooks](https://v2.dataos.info/references/resources/vulcan/configurations/execution-hooks.md): Configure before\_all and after\_all execution hooks to run SQL, macros, or statement files at the start and end of vulcan plan and vulcan run, with use cases and comparison to model-level pre/post stat
- [Linter](https://v2.dataos.info/references/resources/vulcan/configurations/linter.md): Configure the Vulcan linter: built-in and custom rules, Data Product-level rules, enabling rules via config.yaml, excluding models, and warn vs. error rule violation behavior.
- [Notifications](https://v2.dataos.info/references/resources/vulcan/configurations/notifications.md): Configure Vulcan notification targets and event types: Teams webhook, Slack webhook and API, and email/SMTP, plus per-user targets.
- [Models](https://v2.dataos.info/references/resources/vulcan/models.md): How Vulcan models fit together: physical data models, the semantic layer on top of them, and business metrics on top of that.
- [Data Models](https://v2.dataos.info/references/resources/vulcan/models/data-models.md): What a Vulcan data model is: the MODEL block, the SELECT query, naming and typing conventions, comment registration, and the day-to-day workflow for adding, editing, and deleting models.
- [Properties](https://v2.dataos.info/references/resources/vulcan/models/data-models/properties.md): General, storage, engine, and behavior properties for the \`MODEL\` block, plus kind-specific incremental properties and model naming.
- [Types](https://v2.dataos.info/references/resources/vulcan/models/data-models/types.md): SQL, Python, and external model types, and how to choose between them.
- [SQL models](https://v2.dataos.info/references/resources/vulcan/models/data-models/types/sql.md): SQL-based model definition, blueprinting, automatic dependencies, transpilation, and casting conventions.
- [Python models](https://v2.dataos.info/references/resources/vulcan/models/data-models/types/python.md): The \`@model\` decorator, execution context, dependencies, blueprinting, and DataFrame examples for Python models.
- [External models](https://v2.dataos.info/references/resources/vulcan/models/data-models/types/external-models.md): How \`EXTERNAL\` models describe schemas for tables Vulcan does not manage, and how to create and validate them.
- [Kinds](https://v2.dataos.info/references/resources/vulcan/models/data-models/model-kinds.md): INCREMENTAL, FULL, VIEW, EMBEDDED, SEED, and SCD Type 2 model kinds, and their materialization strategies by engine.
- [Statements](https://v2.dataos.info/references/resources/vulcan/models/data-models/statements.md): Pre-statements, post-statements, and on-virtual-update statements, and how to set defaults for them at the project level.
- [Semantic models](https://v2.dataos.info/references/resources/vulcan/models/semantic-models.md): Define business-friendly semantic models on top of Vulcan data models.
- [Dimensions](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/dimensions.md): Define semantic model dimensions, formatting, granularities, semantic behavior, and AI context.
- [Measures](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/measures.md): Define semantic model aggregations, measure types, filters, semantic behavior, and rolling windows.
- [Segments](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/segments.md): Define reusable semantic model filters for common business subsets.
- [Joins](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/joins.md): Define relationships between semantic models for cross-model analysis.
- [Rollups](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/rollups.md): Define pre-aggregation specs that build rollup tables for a semantic model.
- [AI context](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/ai-context.md): Add AI and LLM guidance to semantic models, dimensions, measures, segments, joins, and granularities.
- [Business metrics](https://v2.dataos.info/references/resources/vulcan/models/business-metrics.md): Business metric YAML structure: required and optional properties, dimensions, segments, cross-model metrics, time granularity, forbidden legacy keys, reserved names, and validation rules.
- [Quality](https://v2.dataos.info/references/resources/vulcan/quality.md): Quality overview: how assertions, DQ checks, and unit tests work together to keep models trustworthy, and when to use each.
- [Assertions](https://v2.dataos.info/references/resources/vulcan/quality/assertions.md): Audits and assertions: built-in and user-defined rules that block bad rows.
- [Data quality](https://v2.dataos.info/references/resources/vulcan/quality/data-quality.md): DQ checks (\`kind: dq\`): expectations, dimensions, and profiling.
- [Unit tests](https://v2.dataos.info/references/resources/vulcan/quality/tests.md): Unit tests: YAML fixtures for inputs and outputs, testing incremental models and CTEs, and running tests with the CLI.
- [Policies](https://v2.dataos.info/references/resources/vulcan/policies.md): Technical reference for Vulcan access policies, row filters, column masking, auth plugins, and physical-model-only policy inheritance.
- [Plugins and auth](https://v2.dataos.info/references/resources/vulcan/plugins-and-auth.md): The plugins/ folder auth extension: configuring the after\_authorize hook, AuthExtensionContext and SecurityContext, the resolve\_user\_groups example, and how policies use resolved groups and claims.
- [Roles and permissions](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions.md): Minimum engine privileges required to run a Vulcan domain resource, engine by engine: PostgreSQL, Snowflake, Databricks, Spark, Trino, Redshift, and Athena.
- [PostgreSQL](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/postgres-minimum-permissions.md): Minimum PostgreSQL grants for a Vulcan deployment: source, target, and state schema privileges, by secret\_name tier.
- [Snowflake](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/snowflake-minimum-permissions.md): Minimum Snowflake grants for a Vulcan deployment: source, target, and state schema privileges, by secret\_name tier.
- [Databricks](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/databricks-minimum-permissions.md): Minimum Databricks Unity Catalog grants for a Vulcan deployment: source, target, and state schema privileges, by secret\_name tier.
- [Spark](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/spark-minimum-permissions.md): Minimum DataOS Lakehouse and object-storage permissions for running Vulcan on Spark, across both the DataOS and cloud IAM credential layers.
- [External Trino](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/external-trino-minimum-permissions.md): Minimum Trino SQL or connector access-control privileges for running Vulcan against a client-managed Trino cluster, by secret\_name tier.
- [Dedicated Trino](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/dedicated-trino-minimum-permissions.md): Permission model for a DataOS-managed Dedicated Trino deployment, where access control lives on each depot's underlying data source instead of Trino SQL grants.
- [Advanced features](https://v2.dataos.info/references/resources/vulcan/advanced-features.md): Advanced features that extend Vulcan beyond standard model definitions: macros, signals, custom materializations, and importing Snowflake semantic views.
- [Macros](https://v2.dataos.info/references/resources/vulcan/advanced-features/macros.md): Macros make SQL dynamic with variables and functions. Overview of Vulcan macros, Jinja macros, and the predefined variables both systems share.
- [Built-in](https://v2.dataos.info/references/resources/vulcan/advanced-features/macros/built-in.md): Reference for Vulcan's native macro system: user-defined variables, built-in macro operators for SQL clauses and control flow, and Python macro functions.
- [Jinja](https://v2.dataos.info/references/resources/vulcan/advanced-features/macros/jinja.md): Use Jinja templating in Vulcan SQL models: predefined and user-defined variables, for loops and if statements, and user-defined macro functions.
- [Variables](https://v2.dataos.info/references/resources/vulcan/advanced-features/macros/variables.md): Reference for Vulcan's predefined macro variables: temporal variables like @execution\_ds, runtime variables like @runtime\_stage and @this\_model, and before\_all/after\_all variables.
- [Signals](https://v2.dataos.info/references/resources/vulcan/advanced-features/signals.md): Signals add custom readiness checks before a scheduled model runs, so a model waits for late-arriving or upstream data before it processes an interval.
- [Custom materializations](https://v2.dataos.info/references/resources/vulcan/advanced-features/custom-materializations.md): Write a custom materialization in Python when built-in model kinds don't fit, using the CustomMaterialization and CustomKind classes, then package and share it across projects.
- [Import Snowflake semantic views](https://v2.dataos.info/references/resources/vulcan/advanced-features/import-snowflake-semantic-views.md): Import a Snowflake semantic view into a Vulcan project: configure the Snowflake gateway, run import\_semantic\_view and create\_external\_models, fix inputs.yaml, then plan and query the result over REST.
- [CLI commands](https://v2.dataos.info/references/resources/vulcan/cli.md): Every Vulcan CLI command, its options, and example output: planning and running models, auditing data quality, diffing environments, and managing state.
- [Deployment steps](https://v2.dataos.info/references/resources/vulcan/deployment.md): Deploy a Vulcan Data Product into a DataOS environment: prerequisites, the deployment: section in config.yaml, linting, deploying, and checking status.
- [Monitor & delete](https://v2.dataos.info/references/resources/vulcan/monitor-and-delete.md): Monitor a deployed Vulcan Data Product with the DataOS CLI: read its status and runtime, find the failing container group, read its logs, rerun after a fix, and delete the deployed Resource when it's
- [Troubleshooting](https://v2.dataos.info/references/resources/vulcan/troubleshooting.md): Every validation and request error Vulcan returns, what triggered it, and how to fix it.
- [Nilus](https://v2.dataos.info/references/resources/nilus.md): Nilus is DataOS's unified data movement framework. It moves data from databases, SaaS APIs, event streams, and data warehouses into analytical destinations.
- [Quick start](https://v2.dataos.info/references/resources/nilus/quick-start-guides.md): Decision table for picking a pipeline mode, minimal batch/CDC config snippets, and a pre-production checklist for a first Nilus pipeline.
- [Sources overview](https://v2.dataos.info/references/resources/nilus/sources.md): Browse Nilus source connectors by pipeline types.
- [Core concepts](https://v2.dataos.info/references/resources/nilus/concepts.md): Core concepts that underpin Nilus, architecture, schema evolution, and data masking.
- [Architecture](https://v2.dataos.info/references/resources/nilus/concepts/architecture-and-mechanism.md): How Nilus is structured: the extract-normalize-load flow, the four pipeline types (batch, CDC, stream, metadata), and the extensible connector model.
- [Schema evolution](https://v2.dataos.info/references/resources/nilus/concepts/schema-evolution.md): Learn how Nilus handles schema changes over time across batch and CDC pipelines.
- [Data masking](https://v2.dataos.info/references/resources/nilus/concepts/understanding-data-masking.md): Learn how Nilus masks sensitive values during ingestion with configurable masking rules.
- [Secrets and projections](https://v2.dataos.info/references/resources/nilus/concepts/secrets-and-projections.md): How Nilus resolves credentials at runtime: automatic depot inference for dataos:// addresses, explicit secret projection for direct connector URIs, and repo-auth for custom source sync.
- [Roles and permissions](https://v2.dataos.info/references/resources/nilus/roles-and-permissions.md): How Nilus's privilege requirements map to pipeline mode, not to a single fixed role, plus the minimum grants for each supported source and destination.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/postgresql-minimum-permissions.md): Minimum PostgreSQL grants for each Nilus pipeline mode: batch read and write, CDC via logical replication, and metadata ingestion.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/mongodb-minimum-permissions.md): Minimum MongoDB roles for each Nilus pipeline mode: batch read and write, change-stream CDC, and metadata ingestion.
- [MS SQL Server](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/ms-sql-server-minimum-permissions.md): Minimum SQL Server grants for each Nilus pipeline mode: batch read and write, native CDC, and metadata ingestion.
- [Snowflake](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/snowflake-minimum-permissions.md): Minimum Snowflake grants for each Nilus pipeline mode: batch read and write, and metadata ingestion.
- [Databricks](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/databricks-minimum-permissions.md): Minimum Databricks Unity Catalog grants for each Nilus pipeline mode: batch read and write, and metadata ingestion.
- [Kafka](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/kafka-minimum-permissions.md): Minimum Kafka ACLs for Nilus stream pipelines, across all four supported security protocols. Kafka is supported as a source only.
- [Salesforce](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/salesforce-minimum-permissions.md): Minimum Salesforce org permissions for Nilus batch pipelines: object and field-level read access, and the incremental strategy used per object.
- [Stripe](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/stripe-minimum-permissions.md): Minimum Stripe API key permissions for Nilus batch pipelines: a Restricted Key scoped to read-only access on the ingested resource categories.
- [Batch data movement](https://v2.dataos.info/references/resources/nilus/batch.md): Conceptual overview of Nilus batch data movement: when to use it, how periodic extraction works, and how it differs from CDC and stream modes.
- [Batch pipeline config](https://v2.dataos.info/references/resources/nilus/batch/pipeline-config.md): Complete field reference for Nilus batch pipelines: spec structure, all source.options and sink.options, supported values, and annotated examples.
- [Batch sample configs](https://v2.dataos.info/references/resources/nilus/batch/sample-configs.md): Ready-to-use batch pipeline configurations covering full snapshots, incremental loads, and merge patterns across common source types.
- [Batch sources](https://v2.dataos.info/references/resources/nilus/batch/batch-sources.md): Index of Nilus batch source connectors by category (database vs. SaaS API), linking to each connector's requirements, options, and behavior.
- [AWS Redshift](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/aws-redshift.md): AWS Redshift batch source. Reads tables or custom SQL via the PostgreSQL-compatible wire protocol; the connector address uses the redshift:// scheme and rewrites it internally.
- [Azure Synapse](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/azure-synapse.md): Connector notes for Azure Synapse as a batch source: why there is no dedicated synapse:// scheme, connecting via the MS SQL Server path, and dedicated-pool caveats.
- [Databricks](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/databricks.md): Databricks batch source. Reads tables and views from a Databricks SQL warehouse with optional incremental extraction; supports PAT and OAuth M2M authentication.
- [Delta Lake](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/delta-lake.md): Configuration reference for the Delta Lake batch source: the DuckDB-based read mechanism, supported storage backends, source options, and troubleshooting.
- [Google Analytics](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/google-analytics.md): Configuration reference for the Google Analytics 4 batch source: service-account setup, supported report resources, source/sink options, and sampling limits.
- [Google BigQuery](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/google-bigquery.md): Google BigQuery batch source. Reads tables, views, or custom SQL with optional incremental extraction; uses a service-account credential and optionally a DataOS depot for connection management.
- [Google Sheets](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/google-sheets.md): Google Sheets batch source. Loads a single sheet tab or A1 range per pipeline run; snapshot-oriented and best suited for analyst-maintained reference tables.
- [HubSpot](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/hubspot.md): HubSpot batch source. Extracts CRM objects (contacts, deals, companies, tickets, products, custom objects) as full-refresh snapshots via the Private App API.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/mongodb.md): MongoDB batch source. Reads documents with optional incremental ingestion; supports aggregation-pipeline queries by passing a pipeline array as the source\_table value.
- [MS SQL Server](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/ms-sql-server.md): MS SQL Server batch source. Reads tables or custom SQL via ODBC; supports SQL Authentication and Azure AD access-token authentication.
- [MySQL](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/mysql.md): MySQL batch source. Reads tables or custom SQL with optional incremental ingestion; the connector uses the pymysql driver and rewrites the URI internally.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/postgresql.md): Configuration reference for the PostgreSQL batch source: connectivity and grants, source options, sample configs, and troubleshooting for connection and incremental-load failures.
- [Salesforce](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/salesforce.md): Salesforce batch source. Extracts standard and custom CRM objects; selected high-change objects (accounts, opportunities) support automatic incremental loading via Salesforce modification timestamps.
- [Snowflake](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/snowflake.md): Configuration reference for the Snowflake batch source: warehouse and role setup, password vs. key-pair authentication, source options, and cost-control guidance.
- [Stripe](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/stripe.md): Stripe batch source. Extracts payment objects (charges, customers, subscriptions, invoices, balance transactions) in async, sync, or sync-incremental loading modes.
- [Custom sources](https://v2.dataos.info/references/resources/nilus/batch/custom-sources.md): Index of custom-source paths for systems without a first-party connector: a hand-authored SQL query versus a fully custom Python source implementation.
- [Building a custom source](https://v2.dataos.info/references/resources/nilus/batch/custom-sources/creating-custom-source.md): Step-by-step reference for building a custom Nilus source: Git repo sync configuration, the implementation contract, source options, and troubleshooting.
- [Custom Query (SQL Sources)](https://v2.dataos.info/references/resources/nilus/batch/custom-sources/custom-query-sql-sources.md): Configuration reference for Custom Query batch sources: writing a query:-prefixed SQL source, incremental-column requirements, and common pitfalls.
- [Change data capture](https://v2.dataos.info/references/resources/nilus/cdc.md): Conceptual overview of Nilus Change Data Capture: how it reads database logs, handles initial snapshots, propagates row-level changes, and manages state.
- [CDC pipeline config](https://v2.dataos.info/references/resources/nilus/cdc/pipeline-config.md): Complete field reference for Nilus CDC pipelines: spec structure, all source.options, source.cdc settings, sink.options, and annotated examples.
- [CDC sample configs](https://v2.dataos.info/references/resources/nilus/cdc/sample-configs.md): Ready-to-use CDC pipeline configurations for PostgreSQL, MongoDB, MS SQL Server, MySQL, and IBM DB2 sources.
- [CDC sources](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources.md): Index of Nilus CDC source connectors (IBM DB2, MongoDB, MS SQL Server, MySQL, PostgreSQL), linking to per-connector setup, options, and troubleshooting.
- [IBM DB2](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/ibm-db2.md): Setup and operations reference for the IBM DB2 CDC source: ASN capture-service requirements, snapshot/stream mechanics, CDC options, and troubleshooting.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/mongodb.md): MongoDB CDC source. Captures document-level changes via replica-set change streams; covers oplog sizing, depot setup, snapshot modes, and strategy options.
- [MS SQL Server](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/ms-sql-server.md): MS SQL Server CDC source. Reads row-level changes from SQL Server CDC change tables; covers enabling CDC per table, permissions, and sink configuration.
- [MySQL](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/mysql.md): MySQL CDC source. Reads row-level changes from the binary log (binlog); supports GTID and non-GTID replication modes and multiple authentication plugins.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/postgresql.md): Setup and operations reference for the PostgreSQL CDC source: logical replication setup, slot and publication management, WAL retention, and FAQs.
- [Stream pipelines](https://v2.dataos.info/references/resources/nilus/stream.md): Conceptual overview of Nilus stream pipelines: connecting to Kafka or NATS+JetStream, consuming records in micro-batches, and writing them to a destination.
- [Stream pipeline config](https://v2.dataos.info/references/resources/nilus/stream/pipeline-config.md): Complete field reference for Nilus Stream pipelines: spec structure, all source.options, source.cdc settings, sink.options, and annotated examples.
- [Stream sample configs](https://v2.dataos.info/references/resources/nilus/stream/sample-configs.md): Ready-to-use stream pipeline configurations for Kafka (plaintext, SASL/SSL, Confluent Cloud) and NATS+JetStream sources.
- [Stream sources](https://v2.dataos.info/references/resources/nilus/stream/stream-sources.md): Index of Nilus stream source connectors (Kafka, NATS) with a note on typical sink strategy for event-shaped records.
- [Kafka](https://v2.dataos.info/references/resources/nilus/stream/stream-sources/kafka.md): Kafka stream source. Consumes topics in micro-batches as a long-running consumer; supports plaintext, SASL/SSL, Confluent Cloud, and mTLS configurations.
- [NATS](https://v2.dataos.info/references/resources/nilus/stream/stream-sources/nats.md): Configuration reference for the NATS+JetStream stream source: authentication modes, ephemeral vs. durable consumers, source/sink options, and troubleshooting.
- [Metadata pipelines](https://v2.dataos.info/references/resources/nilus/metadata-pipelines.md): Conceptual overview of how Nilus metadata pipelines work: shallow vs. deep extraction modes, the stages each runs (inventory, lineage, profiling, classification, usage), and when to use each.
- [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md): This page explains the Nilus metadata pipeline config shape, a single workflow that extracts catalog metadata, column profiles, classification tags, query lineage, and query usage from a...
- [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md): Ready-to-use metadata pipeline configurations for Snowflake, Databricks, and DataOS Lakehouse (Iceberg) sources.
- [Metadata sources](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources.md): Index of Nilus metadata-capable sources, their supported shallow/deep stages, and how to scope extraction with database/schema/table filters.
- [Databricks](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/databricks-metadata.md): Configuration reference for the Databricks metadata source: supported stages, source options, required Unity Catalog/Hive Metastore grants, and troubleshooting.
- [DataOS Lakehouse](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/dataos-lakehouse-metadata.md): Configuration reference for the DataOS Lakehouse metadata-only source: what inventory it brings in, source options, required depot permissions, and troubleshooting.
- [Microsoft Fabric Lakehouse](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric-lakehouse.md): Configure Nilus metadata pipelines for Microsoft Fabric Lakehouse SQL analytics endpoints.
- [Microsoft Fabric Warehouse](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric.md): Configuration reference for the Microsoft Fabric Warehouse metadata source: supported stages, source options, required permissions, and troubleshooting.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/mongodb.md): Configuration reference for the MongoDB metadata source: supported stages, source options, required roles, and troubleshooting.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/postgresql.md): Configuration reference for the PostgreSQL metadata source: supported stages, source options, required grants, and troubleshooting.
- [Snowflake](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/snowflake-metadata.md): Configuration reference for the Snowflake metadata source: supported stages, account-usage permission requirements, source options, and troubleshooting.
- [Destinations](https://v2.dataos.info/references/resources/nilus/destinations.md): Know about Nilus destination connectors, their supported pipeline modes, and incremental strategies.
- [Cloud warehouses](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses.md): Configure BigQuery, Databricks, Redshift, and Snowflake destinations for Nilus pipelines.
- [BigQuery](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses/bigquery.md): BigQuery destination. Writes Nilus batch, CDC, and stream output into BigQuery datasets using a service-account credential; supports GCS staging and parquet loading.
- [Databricks](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses/databricks.md): Databricks destination. Writes Nilus batch, CDC, and stream output into Unity Catalog tables via a SQL warehouse; supports PAT and OAuth M2M authentication.
- [Redshift](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses/redshift.md): Configuration reference for the Redshift destination: connectivity and grants, sink options, distribution/sort-key guidance for merge, and troubleshooting.
- [Snowflake](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses/snowflake.md): Configuration reference for the Snowflake destination: warehouse and role requirements, password vs. key-pair authentication, sink options, and troubleshooting.
- [Databases](https://v2.dataos.info/references/resources/nilus/destinations/databases.md): Configure MongoDB, MS SQL Server, and PostgreSQL destinations for Nilus pipelines.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/destinations/databases/mongodb.md): MongoDB destination. Writes Nilus batch, CDC, and stream output into collections; supports replace and merge strategies only. Append is explicitly rejected.
- [MS SQL Server](https://v2.dataos.info/references/resources/nilus/destinations/databases/ms-sql-server.md): Configuration reference for the MS SQL Server destination: ODBC driver and TLS setup, SQL vs. Azure AD authentication, sink options, and troubleshooting.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/destinations/databases/postgresql.md): Configuration reference for the PostgreSQL destination: connectivity and grants, sink options, partitioned-table writes, and troubleshooting.
- [DataOS Lakehouse](https://v2.dataos.info/references/resources/nilus/destinations/dataos-lakehouse.md): The DataOS Lakehouse destination writes Apache Iceberg-backed datasets to object storage. Supported for AWS S3, Azure ADLS Gen2, and GCP GCS backends.
- [AWS-backed](https://v2.dataos.info/references/resources/nilus/destinations/dataos-lakehouse/aws-backed.md): AWS-backed DataOS Lakehouse destination. Writes Nilus batch, CDC, and stream output as Apache Iceberg datasets on Amazon S3; covers depot setup, secrets, and Iceberg sink options.
- [Azure-backed](https://v2.dataos.info/references/resources/nilus/destinations/dataos-lakehouse/azure-backed.md): Azure-backed DataOS Lakehouse destination. Writes Nilus batch, CDC, and stream output as Apache Iceberg datasets on ADLS Gen2 (ABFSS); covers depot setup, secrets, and Iceberg sink options.
- [GCP-backed](https://v2.dataos.info/references/resources/nilus/destinations/dataos-lakehouse/gcp-backed.md): GCP-backed DataOS Lakehouse destination. Writes Nilus batch, CDC, and stream output as Apache Iceberg datasets on Google Cloud Storage; covers depot setup, secrets, and Iceberg sink options.
- [Microsoft Fabric Lakehouse](https://v2.dataos.info/references/resources/nilus/destinations/microsoft-fabric-lakehouse.md): Write Delta tables from Nilus batch pipelines to Microsoft Fabric Lakehouses in OneLake.
- [Observability](https://v2.dataos.info/references/resources/nilus/observability.md): Index of Nilus observability surfaces (logs, API, metrics, Grafana) and the core signals to check for pipeline health.
- [API endpoints](https://v2.dataos.info/references/resources/nilus/observability/api-endpoints.md): Reference for Nilus Manager's read-only HTTP API: base URLs, authentication, and endpoints for run history, CDC offsets, and schema history.
- [Grafana dashboards](https://v2.dataos.info/references/resources/nilus/observability/grafana-dashboard.md): Guide to the two Nilus Grafana dashboards (Pipelines, Fleet Overview): what each answers, how to read the panels, and setup requirements.
- [Prometheus metrics](https://v2.dataos.info/references/resources/nilus/observability/prometheus-metrics.md): Reference for Nilus Prometheus metrics: which metrics answer whether a pipeline is running, slow, processing records, or healthy.
- [Pipeline optimization](https://v2.dataos.info/references/resources/nilus/pipeline-optimization.md): Symptom-to-page routing table for Nilus pipeline optimization, plus the measurement loop and common tuning knobs shared across guides.
- [Optimizing for time](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-time.md): Diagnostic guide for slow pipelines: bottleneck identification, tuning order, and the Iceberg file-size and worker-concurrency knobs that affect runtime.
- [Optimizing for resource](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-resource.md): Diagnostic guide for high CPU, memory, or destination-cost usage: symptom table, tuning order, and Iceberg-load memory/volume guidance.
- [Optimize sink datasets](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets.md): Full knob taxonomy for sink tuning (correctness, shape, throughput, sampling), recommended config profiles, a sizing playbook, and common anti-patterns.
- [Shape knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-shape-knobs.md): Settings that control the physical layout of the destination dataset: partitioning, clustering, and nested-field flattening depth.
- [Sampling knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-sampling-knobs.md): Settings that cap extraction volume per run: sql\_limit and yield\_limit for controlled sampling and validation runs.
- [Throughput knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-throughput-knobs.md): Settings that control pipeline speed: page size, extract parallelism, loader file size, and worker counts.
- [Correctness knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-correctness-knobs.md): Settings that control what data lands in the destination: incremental strategy, primary key, type hints, interval bounds, and column masking.
- [Tuning large Lakehouse (Iceberg) loads](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-lakehouse-iceberg-loads.md): Benchmark-backed configurations for large Iceberg loads to DataOS Lakehouse. Covers wide and narrow table shapes, file-size decisions, persistent volume setup, and ongoing table maintenance.
- [Destination gotchas and troubleshooting](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-destination-gotchas-and-troubleshooting.md): Destination-specific quirks and a troubleshooting matrix covering OOM errors, slow loads, duplicate rows, file bloat, and validation failures.
- [Optimizing CDC](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-cdc.md): Tune pod memory, Debezium queue depth, and JVM heap for CDC pipelines.
- [Troubleshooting](https://v2.dataos.info/references/resources/nilus/troubleshooting.md): Global triage checklist for Nilus pipeline failures: first-response steps, a symptom-to-page map, and evidence to collect before escalating.
- [Checking logs](https://v2.dataos.info/references/resources/nilus/troubleshooting/checking-logs.md): How to read Nilus runtime logs: what fields to capture, log-level configuration, common log patterns, and the recommended reading order.
- [Common errors](https://v2.dataos.info/references/resources/nilus/troubleshooting/common-errors.md): Catalog of common Nilus error messages and symptoms across pipeline modes, each paired with its likely cause and first recovery step.
- [Snowflake key-pair authentication](https://v2.dataos.info/references/resources/nilus/troubleshooting/snowflake-key-pair-authentication.md): Step-by-step guide to Snowflake key-pair authentication for Nilus: generating and registering RSA keys, depot configuration, and key rotation.
- [PostgreSQL partitioned tables for CDC](https://v2.dataos.info/references/resources/nilus/troubleshooting/postgresql-cdc-partitioned-tables.md): How to configure Nilus CDC for PostgreSQL declarative partitioning: including parent and child tables in the capture list, publication coverage, and a worked config example.
- [MongoDB CDC error 286](https://v2.dataos.info/references/resources/nilus/troubleshooting/mongodb-cdc-error-286.md): Error 286 indicates that a MongoDB change stream (which Nilus relies on for CDC) attempted to resume from a point that is no longer present in the replica‑set oplog (local.oplog.rs). When this...
- [MongoDB CDC BufferingChangeStreamCursor](https://v2.dataos.info/references/resources/nilus/troubleshooting/mongodb-cdc-bufferingchangestreamcursor.md): This warning comes from the Nilus MongoDB CDC reader. It means the runtime's in-memory buffer is full and cannot accept more events until the downstream consumer catches up.
- [Projections](https://v2.dataos.info/references/resources/projections.md): Use this guide to understand and build projections in DataOS for runtime dynamic configurations.
- [How projections work](https://v2.dataos.info/references/resources/projections/how-projections-work.md): Understand how DataOS renders, stores, and mounts projection output at runtime.
- [Best practices](https://v2.dataos.info/references/resources/projections/best-practices.md): Design secure, reusable, and maintainable projections for DataOS workloads.
- [Troubleshooting](https://v2.dataos.info/references/resources/projections/troubleshooting.md): Resolve common projection template, context, and file-mount issues.
- [Secret](https://v2.dataos.info/references/resources/secret.md): A Secret is a DataOS Resource designed for securely storing sensitive information at the DataOS Tenant-level
- [Data sources](https://v2.dataos.info/references/resources/secret/data-sources.md): Create tenant Secrets for supported data source credentials.
- [Azure Blob File System Secure (ABFSS)](https://v2.dataos.info/references/resources/secret/data-sources/azure-blob-file-system-secure-abfss.md): Create a Secret for securing ABFSS storage credentials.
- [Bigquery](https://v2.dataos.info/references/resources/secret/data-sources/bigquery.md): Create a Secret for securing BigQuery credentials.
- [Databricks](https://v2.dataos.info/references/resources/secret/data-sources/databricks.md): Create a Secret for securing Databricks credentials.
- [Google Cloud Storage (GCS)](https://v2.dataos.info/references/resources/secret/data-sources/google-cloud-storage-gcs.md): Create a Secret for securing Google Cloud Storage credentials.
- [Java Database Connectivity (JDBC)](https://v2.dataos.info/references/resources/secret/data-sources/java-database-connectivity-jdbc.md): Create a Secret for securing JDBC connection credentials.
- [Kafka](https://v2.dataos.info/references/resources/secret/data-sources/kafka.md): Create a Secret for securing Kafka credentials across multiple security modes.
- [Microsoft SQL Server (MSSQL) or Azure SQL](https://v2.dataos.info/references/resources/secret/data-sources/microsoft-sql-server-mssql-or-azure-sql.md): Create a Secret for securing Microsoft SQL Server or Azure SQL credentials.
- [MongoDB](https://v2.dataos.info/references/resources/secret/data-sources/mongodb.md): Create a Secret for securing MongoDB credentials.
- [MySQL](https://v2.dataos.info/references/resources/secret/data-sources/mysql.md): Create a Secret for securing MySQL credentials.
- [Opensearch](https://v2.dataos.info/references/resources/secret/data-sources/opensearch.md): Create a Secret for securing OpenSearch credentials.
- [Oracle](https://v2.dataos.info/references/resources/secret/data-sources/oracle.md): Create a Secret for securing Oracle database credentials.
- [PostgreSQL](https://v2.dataos.info/references/resources/secret/data-sources/postgresql.md): Create a Secret for securing PostgreSQL credentials.
- [Redshift](https://v2.dataos.info/references/resources/secret/data-sources/redshift.md): Create a Secret for securing Amazon Redshift credentials.
- [Simple Storage Service (Amazon S3)](https://v2.dataos.info/references/resources/secret/data-sources/simple-storage-service-amazon-s3.md): Create a Secret for securing Amazon S3 credentials.
- [Snowflake](https://v2.dataos.info/references/resources/secret/data-sources/snowflake.md): Steps to create Secret to store Snowflake credentials
- [RSA Key-pair](https://v2.dataos.info/references/resources/secret/data-sources/snowflake/rsa-key-pair.md): Steps to Generate and Register a Key Pair in Snowflake
- [Repositories](https://v2.dataos.info/references/resources/secret/repositories.md): Create tenant Secrets for supported repository credentials.
- [AWS CodeCommit](https://v2.dataos.info/references/resources/secret/repositories/aws-codecommit.md): Create a tenant Secret for securing AWS CodeCommit credentials
- [Bitbucket](https://v2.dataos.info/references/resources/secret/repositories/bitbucket.md): Create a Secret for securing Bitbucket repository credentials
- [Git](https://v2.dataos.info/references/resources/secret/repositories/git.md): Create a Secret for securing Git repository credentials
- [Container registries](https://v2.dataos.info/references/resources/secret/container-registries.md): Create a Secret for securing container registry (Docker) credentials used to pull private images
- [Consume](https://v2.dataos.info/references/resources/secret/consume.md): Consume Secret in other DataOS Resources.
- [Manifest configurations](https://v2.dataos.info/references/resources/secret/manifest-configuration.md): Following attributes/fields are declared for each Secret Resource object in its YAML manifest file. Some of these attributes need to be mandatorily declared, while others are optional.
- [Depot](https://v2.dataos.info/references/resources/depot.md): Depot is a Resource used to connect different data sources to DataOS.
- [Data sources](https://v2.dataos.info/references/resources/depot/supported-sources.md): Choose the source-specific guide for creating a Depot.
- [Amazon Redshift](https://v2.dataos.info/references/resources/depot/supported-sources/redshift.md): Create a Depot resource for Amazon Redshift connections.
- [Amazon S3](https://v2.dataos.info/references/resources/depot/supported-sources/s3.md): Create a Depot resource for Amazon S3 buckets.
- [Azure Blob File System Secure (ABFSS)](https://v2.dataos.info/references/resources/depot/supported-sources/abfss.md): Create a Depot resource for ABFSS object storage.
- [Databricks](https://v2.dataos.info/references/resources/depot/supported-sources/databricks.md): Create a Depot resource for Databricks workspaces.
- [Elasticsearch](https://v2.dataos.info/references/resources/depot/supported-sources/elasticsearch.md): Create a Depot resource for Elasticsearch clusters.
- [Eventhub](https://v2.dataos.info/references/resources/depot/supported-sources/eventhub.md): Create a Depot resource for Azure Event Hubs.
- [Google BigQuery](https://v2.dataos.info/references/resources/depot/supported-sources/bigquery.md): Create a Depot resource for Google BigQuery datasets.
- [Google Cloud Storage (GCS)](https://v2.dataos.info/references/resources/depot/supported-sources/gcs.md): Create a Depot resource for Google Cloud Storage buckets.
- [Java Database Connectivity (JDBC)](https://v2.dataos.info/references/resources/depot/supported-sources/jdbc.md): Create a Depot resource using a JDBC-compatible connection.
- [Kafka](https://v2.dataos.info/references/resources/depot/supported-sources/kafka.md): Create a Depot resource for Kafka topics and brokers across multiple security modes.
- [Lakehouse (Iceberg on ABFSS, GCS, or S3)](https://v2.dataos.info/references/resources/depot/supported-sources/lakehouse.md): Create a Depot resource for a Lakehouse backed by ABFSS, GCS, or Amazon S3.
- [Microsoft SQL Server (MSSQL) or Azure SQL](https://v2.dataos.info/references/resources/depot/supported-sources/mssql.md): Create a Depot resource for Microsoft SQL Server or Azure SQL.
- [MongoDB](https://v2.dataos.info/references/resources/depot/supported-sources/mongo.md): Create a Depot resource for MongoDB databases and collections.
- [MySQL](https://v2.dataos.info/references/resources/depot/supported-sources/mysql.md): Create a Depot resource for MySQL databases.
- [Opensearch](https://v2.dataos.info/references/resources/depot/supported-sources/opensearch.md): Create a Depot resource for OpenSearch clusters.
- [Oracle](https://v2.dataos.info/references/resources/depot/supported-sources/oracle.md): Create a Depot resource for Oracle databases.
- [PostgreSQL](https://v2.dataos.info/references/resources/depot/supported-sources/postgresql.md): Create a Depot resource for PostgreSQL databases.
- [Snowflake](https://v2.dataos.info/references/resources/depot/supported-sources/snowflake.md): Create a Depot resource for Snowflake databases and schemas.
- [Supported table format](https://v2.dataos.info/references/resources/depot/supported-table-format.md): Supported table formats that can be used with Depot.
- [Iceberg](https://v2.dataos.info/references/resources/depot/supported-table-format/iceberg.md): Using Iceberg table format in object storage Depots
- [Delta](https://v2.dataos.info/references/resources/depot/supported-table-format/delta.md): Using Delta table format in object storage Depots
- [Using Depots](https://v2.dataos.info/references/resources/depot/consume.md): Consume data from Depot using DataOS interfaces and Stacks.
- [Data Movement (Nilus)](https://v2.dataos.info/references/resources/depot/consume/nilus.md): Move data between Depot-connected source and target systems with Nilus batch and CDC ingestion pipelines.
- [Data Modeling and Transformation (Vulcan)](https://v2.dataos.info/references/resources/depot/consume/vulcan.md): Build semantic models, transform Depot-connected datasets, and deliver governed analytics with Vulcan.
- [Querying (Minerva)](https://v2.dataos.info/references/resources/depot/consume/querying.md): Query Depot-connected data interactively through Minerva's Trino-compatible SQL endpoint.
- [Add a custom Depot type](https://v2.dataos.info/references/resources/depot/create-depot-type.md): Define a custom Depot type so DataOS can recognize a new source system, validate its configuration, and resolve Uniform Data Links for it.
- [Manifest configurations](https://v2.dataos.info/references/resources/depot/configurations.md): Reference Depot manifest fields and source-specific configuration options.
- [Lakehouse](https://v2.dataos.info/references/resources/lakehouse.md): What a Lakehouse is, how it's built, and how to create and manage one.
- [Architecture](https://v2.dataos.info/references/resources/lakehouse/architecture.md): Understand the storage, catalog, orchestration, and compute components of a Lakehouse.
- [Create a Lakehouse](https://v2.dataos.info/references/resources/lakehouse/create-a-lakehouse.md): Prerequisites, permissions, and steps to create, apply, and verify a Lakehouse Resource on S3, ABFSS, or GCS.
- [Storage-specific configuration](https://v2.dataos.info/references/resources/lakehouse/supported-sources.md): Choose the storage-specific guide for creating a Lakehouse.
- [Lakehouse on ABFSS](https://v2.dataos.info/references/resources/lakehouse/supported-sources/lakehouse-on-abfss-1.md): Create a Lakehouse Resource on an Azure Blob File System Secure (ABFSS) data source.
- [Lakehouse on S3](https://v2.dataos.info/references/resources/lakehouse/supported-sources/lakehouse-on-s3-1.md): Create a Lakehouse Resource on an Amazon Simple Storage Service (S3) data source.
- [Lakehouse on GCS](https://v2.dataos.info/references/resources/lakehouse/supported-sources/lakehouse-on-gcs-1.md): Create a Lakehouse Resource on a Google Cloud Storage (GCS) data source.
- [Create a Lakehouse Depot](https://v2.dataos.info/references/resources/lakehouse/create-a-lakehouse-depot.md): Connect an existing Lakehouse Resource with a Lakehouse Depot.
- [Manifest configurations](https://v2.dataos.info/references/resources/lakehouse/configurations.md): Reference for the Lakehouse manifest structure and every supported configuration field.
- [Command reference](https://v2.dataos.info/references/resources/lakehouse/command-reference.md): Lakehouse resource management commands
- [Minerva](https://v2.dataos.info/references/resources/minerva.md): Minerva and its role in the runtime layer.
- [Access control (WIP)](https://v2.dataos.info/references/resources/minerva/access-control.md): Roles and permissions for creating, managing, and querying Minerva.
- [Create a Minerva resource](https://v2.dataos.info/references/resources/minerva/create-a-minerva-resource.md): Create and verify a Minerva resource that exposes configured depots through a Trino-compatible endpoint.
- [Troubleshoot and monitor](https://v2.dataos.info/references/resources/minerva/troubleshoot-and-monitor.md): Diagnose Minerva resource failures and monitor runtime health.
- [Performance considerations](https://v2.dataos.info/references/resources/minerva/limitations.md): Understand where Minerva fits best, where it slows down, and when to use curated tables or Spark-based processing instead.
- [Bundle](https://v2.dataos.info/references/resources/bundle.md): Explains bundle resource and how they package workload assets.
- [Core concepts](https://v2.dataos.info/references/resources/bundle/core-concepts.md): Core concepts specific to bundle resource.
- [Field reference](https://v2.dataos.info/references/resources/bundle/field-reference.md): Reference for Bundle manifest fields and nested configuration.
- [Volume](https://v2.dataos.info/references/resources/volume.md): Explains the Volume resource and the two ways to attach persistent storage to a workload.
- [Service](https://v2.dataos.info/references/resources/service.md): Explains services for long-running and exposed workloads.
- [Core concepts](https://v2.dataos.info/references/resources/service/core-concepts.md): This section includes core concepts specific to a service resource
- [Example](https://v2.dataos.info/references/resources/service/example.md): End-to-end Service examples for common deployment patterns.
- [Field reference](https://v2.dataos.info/references/resources/service/field-reference.md): Reference for Service manifest fields and nested runtime configuration.
- [Worker](https://v2.dataos.info/references/resources/worker.md): Explains workers for background and task-based execution.
- [Core concepts](https://v2.dataos.info/references/resources/worker/core-concepts.md): Core concepts specific to Worker resources.
- [Example](https://v2.dataos.info/references/resources/worker/example.md): End-to-end Worker examples for common deployment patterns.
- [Field reference](https://v2.dataos.info/references/resources/worker/field-reference.md): Reference for Worker manifest fields and nested runtime configuration.
- [Workflow](https://v2.dataos.info/references/resources/workflow.md): Explains workflows for orchestrated, multi-step execution.
- [Core concepts](https://v2.dataos.info/references/resources/workflow/core-concepts.md): Core concepts specific to Workflow resources.
- [Create a Workflow](https://v2.dataos.info/references/resources/workflow/create-a-workflow.md): Define, apply, and verify a Workflow resource.
- [Field reference](https://v2.dataos.info/references/resources/workflow/field-reference.md): Reference for Workflow manifest fields and DAG step configuration.
- [Example DAG patterns](https://v2.dataos.info/references/resources/workflow/example.md): End-to-end Workflow examples for common orchestration patterns.
- [Understand Interfaces](https://v2.dataos.info/references/interfaces/overview.md): The ways to query, connect to, and operate Data Products through APIs, BI tools, database clients, AI assistants, and the CLI.
- [APIs](https://v2.dataos.info/references/interfaces/apis.md): Choose between the Data Product APIs and the Core APIs for building on DataOS.
- [Data Product APIs](https://v2.dataos.info/references/interfaces/apis/data-product-apis.md): Explore the Data Product API surface for governed discovery, queries, reusable results, and operational insights.
- [Getting started](https://v2.dataos.info/references/interfaces/apis/data-product-apis/getting-started.md): A practical introduction to Data Product API access, requests, and error recovery.
- [Authentication & authorization](https://v2.dataos.info/references/interfaces/apis/data-product-apis/getting-started/authentication.md): Secure Data Product API calls with DataOS tokens, identity checks, and access controls.
- [API calls](https://v2.dataos.info/references/interfaces/apis/data-product-apis/getting-started/api-calls.md): Core HTTP conventions for calling a Data Product API, including URLs, pagination, and Postman.
- [Error codes](https://v2.dataos.info/references/interfaces/apis/data-product-apis/getting-started/error-codes.md): Reference for identifying API failures and choosing the right recovery action.
- [Querying Data Products](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products.md): Guidance for matching Data Product query interfaces to integration and analytics needs.
- [REST](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/rest.md): Reference for expressing semantic analytics through structured JSON requests.
- [Semantic SQL](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/semantic-sql.md): A SQL-oriented interface for analyzing governed business definitions.
- [GraphQL](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/graphql.md): A typed API for selecting Data Product fields and receiving nested results.
- [Metrics](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/metrics.md): A focused interface for reading approved KPI time series.
- [Perspectives](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/perspectives.md): Turn a successful query into a reusable, governed data asset.
- [Advanced](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/advanced.md): Deeper guidance for selecting, executing, and extending Data Product queries.
- [Choosing a query method](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/advanced/choosing-a-query-method.md): Compare the available query interfaces and select the right fit for your integration.
- [Query lifecycle](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/advanced/query-lifecycle.md): Learn how asynchronous requests progress from submission to a final result or error.
- [Measure and dimension behavior](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/advanced/measure-and-dimension-behavior.md): Clarify the semantic rules that shape calculations, grouping, filtering, and relationships.
- [Cross-model queries](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/advanced/cross-model-queries.md): Combine related business entities in a single governed analytical request.
- [Source SQL](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/advanced/source-sql.md): Use the physical data layer only for questions beyond the semantic model.
- [Statement lifecycle](https://v2.dataos.info/references/interfaces/apis/data-product-apis/querying-data-products/advanced/statement-lifecycle.md): Detailed sequence diagrams walking through the statement lifecycle, step by step.
- [SDKs](https://v2.dataos.info/references/interfaces/apis/data-product-apis/sdks.md): Client libraries for building typed Data Product API integrations.
- [Python SDK](https://v2.dataos.info/references/interfaces/apis/data-product-apis/sdks/python-sdk.md): A typed Python client for integrating applications with Data Product APIs.
- [TypeScript SDK](https://v2.dataos.info/references/interfaces/apis/data-product-apis/sdks/typescript-sdk.md): A fully typed TypeScript client for integrating applications with Data Product APIs.
- [API reference](https://v2.dataos.info/references/interfaces/apis/data-product-apis/api-reference.md): Endpoint specifications for integrating with the Data Product API.
- [Discover](https://v2.dataos.info/references/interfaces/apis/data-product-apis/api-reference/discover.md): Find the API contracts that expose a Data Product's metadata and semantic model.
- [Query](https://v2.dataos.info/references/interfaces/apis/data-product-apis/api-reference/query.md): API contracts for running analytical requests and working with their outputs.
- [GraphQL](https://v2.dataos.info/references/interfaces/apis/data-product-apis/api-reference/graphql.md): Technical contract for GraphQL schema access and execution.
- [Perspectives](https://v2.dataos.info/references/interfaces/apis/data-product-apis/api-reference/perspectives.md): Technical contract for managing saved, reusable query results.
- [Engagement](https://v2.dataos.info/references/interfaces/apis/data-product-apis/api-reference/engagement.md): Technical contract for working with consumer engagement data.
- [Quality & Activity](https://v2.dataos.info/references/interfaces/apis/data-product-apis/api-reference/quality-and-activity.md): Technical contract for quality information and operational activity.
- [Health](https://v2.dataos.info/references/interfaces/apis/data-product-apis/api-reference/health.md): Technical contract for checking service availability and runtime health.
- [Core APIs](https://v2.dataos.info/references/interfaces/apis/core-apis.md): Reference for platform-api: tenant apps and links, log search, workload metrics, and user tokens.
- [Authentication](https://v2.dataos.info/references/interfaces/apis/core-apis/authentication.md): Authenticate platform-api requests with a Bearer token or apikey header, plus optional tracing headers.
- [API reference](https://v2.dataos.info/references/interfaces/apis/core-apis/api-reference.md): Endpoint index for platform-api: tenant apps and links, log search, metrics, and user tokens.
- [Tenant apps & links](https://v2.dataos.info/references/interfaces/apis/core-apis/api-reference/tenant-resources.md): List the apps and links registered for a tenant, with optional filtering by product.
- [Log search](https://v2.dataos.info/references/interfaces/apis/core-apis/api-reference/logs-search.md): Search logs for a tenant resource with the v1 (grouped by pod) or v2 (flat list) endpoint.
- [Observability metrics](https://v2.dataos.info/references/interfaces/apis/core-apis/api-reference/observability-metrics.md): Query workload time-series metrics from Instance Prometheus with PromQL through platform-api.
- [User tokens](https://v2.dataos.info/references/interfaces/apis/core-apis/api-reference/user-tokens.md): Create, list, retrieve, and delete user API tokens through platform-api to automate token management.
- [CLI](https://v2.dataos.info/references/interfaces/command-line-interface.md): Use the CLI to manage DataOS Resources.
- [Resource commands](https://v2.dataos.info/references/interfaces/command-line-interface/resource.md): Reference for dataos-ctl resource commands.
- [Lakehouse commands](https://v2.dataos.info/references/interfaces/command-line-interface/lakehouse.md): Reference for dataos-ctl lakehouse commands.
- [User commands](https://v2.dataos.info/references/interfaces/command-line-interface/user.md): Reference for dataos-ctl user commands.
- [Apps](https://v2.dataos.info/references/interfaces/apps.md): Build applications on a Data Product, then deploy them in DataOS or link an externally hosted app.
- [Deploy a data application](https://v2.dataos.info/references/interfaces/apps/deploy-a-data-application.md): Run a data application from a container image in DataOS using an app Resource.
- [Link an external application](https://v2.dataos.info/references/interfaces/apps/link-an-external-application.md): Register an externally hosted application as a link Resource so consumers can find it in DataOS.
- [BI tools](https://v2.dataos.info/references/interfaces/bi-tools.md): Connect Tableau, Power BI Desktop, or Power BI Service to a Data Product's semantic layer.
- [Tableau Desktop](https://v2.dataos.info/references/interfaces/bi-tools/tableau.md): Download the Tableau .tds project, then review supported types, limits, and troubleshooting.
- [Tableau Cloud](https://v2.dataos.info/references/interfaces/bi-tools/tableau-cloud.md): This page covers everything beyond the initial setup.
- [Power BI Desktop](https://v2.dataos.info/references/interfaces/bi-tools/power-bi-desktop.md): Download the Power BI .pbip project, then review supported types, limits, and troubleshooting.
- [Power BI Service](https://v2.dataos.info/references/interfaces/bi-tools/power-bi-service.md): How Power BI Service connects to DataOS through a gateway, plus limitations and troubleshooting.
- [Database clients](https://v2.dataos.info/references/interfaces/db-clients.md): Query a Data Product's semantic models with plain SQL from any MySQL-compatible client.
- [AI activation](https://v2.dataos.info/references/interfaces/ai-activation.md): Explains AI activation in DataOS and how Data Product MCP connects governed Data Products to AI assistants.
- [MCP components](https://v2.dataos.info/references/interfaces/ai-activation/mcp-components.md): Understand the MCP host, client, server, tools, resources, and prompts used in AI activation.
- [MCP layers](https://v2.dataos.info/references/interfaces/ai-activation/mcp-layers.md): Understand the data and transport layers used by Data Product MCP.
- [Data Product MCP lifecycle](https://v2.dataos.info/references/interfaces/ai-activation/data-product-mcp-lifecycle.md): See how a Data Product MCP session initializes, operates, and ends.
- [Request flow](https://v2.dataos.info/references/interfaces/ai-activation/request-flow.md): Follow how a user prompt becomes a governed Data Product MCP response.
- [Authentication and authorization](https://v2.dataos.info/references/interfaces/ai-activation/authentication-and-authorization.md): See how Data Product MCP enforces DataOS identity, access, and policy controls.
- [Consumer journey](https://v2.dataos.info/references/interfaces/ai-activation/consumer-journey.md): See how AI activation helps users find, trust, query, and act on governed Data Products.
- [Builder journey](https://v2.dataos.info/references/interfaces/ai-activation/builder-journey.md): See how AI activation supports discovery, design, planning, build, and validation of Data Products.
- [Trust model](https://v2.dataos.info/references/interfaces/ai-activation/trust-model.md): Understand the governance and assistant trust boundaries in AI activation.
- [Supported connectors](https://v2.dataos.info/references/supported-connectors.md): This document lists the data sources supported by DataOS.
- [Glossary](https://v2.dataos.info/references/glossary.md): Use this glossary for core terms used across DataOS documentation
- [Key concepts](https://v2.dataos.info/references/key-concepts.md): Includes key concepts specific to DataOS
- [API tokens](https://v2.dataos.info/references/key-concepts/api-tokens.md): What a DataOS API token is, how it's created and revoked, and how its scope and expiry work.

## Release Notes

- [September 2026](https://v2.dataos.info/releasenotes/september-2026.md): More control over platform operations, broader engine support, and tighter governance over how data gets shared.
- [August 2026](https://v2.dataos.info/releasenotes/august-2026.md): More ways to land data, more confidence in every AI answer.
- [July 2026](https://v2.dataos.info/releasenotes/july-2026.md): Tighter contracts on the way in, more ways to query on the way out.
- [June 2026](https://v2.dataos.info/releasenotes/june-2026.md): A major DataOS release to build governed Data Products on any engine, then serve them to people, tools, and AI.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
This site has an agentic ask interface you may use to query the documentation dynamically by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/index.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.