> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources.md).

# DataOS Resources

- [Overview](https://v2.dataos.info/references/resources/overview.md): Understand what DataOS Resources are, how they are structured, and how to deploy and manage them.
- [Vulcan](https://v2.dataos.info/references/resources/vulcan.md): Understand what Vulcan is, how it builds governed data products, and how it runs transformation, quality, and semantics on your existing engine.
- [LDK setup](https://v2.dataos.info/references/resources/vulcan/ldk.md): Set up the Local Development Kit: Python virtual environment, Vulcan, and engine.
- [Concepts](https://v2.dataos.info/references/resources/vulcan/concepts.md): How Vulcan is put together, how a data product moves from setup to a governed asset, and how \`vulcan plan\` and \`vulcan run\` decide what to execute.
- [Architecture](https://v2.dataos.info/references/resources/vulcan/concepts/architecture.md): How Vulcan is structured: the fingerprint-plan-apply flow, the model materialization strategies, and the extensible engine adapter model.
- [Data Product lifecycle](https://v2.dataos.info/references/resources/vulcan/concepts/data-product-lifecycle.md): The full path from local setup to a production data product: project initialization, model development, testing, semantics, planning and running, data quality, API access, and monitoring.
- [Run and plan](https://v2.dataos.info/references/resources/vulcan/concepts/run-and-plan.md): Understand what \`vulcan plan\` and \`vulcan run\` do before you use them.
- [Plan without a virtual layer](https://v2.dataos.info/references/resources/vulcan/concepts/run-and-plan/plan-guide.md): Understand what \`vulcan plan\` does without a virtual layer, using direct materialization instead of virtual-layer promotion.
- [Plan with a virtual layer](https://v2.dataos.info/references/resources/vulcan/concepts/run-and-plan/plan-with-vde.md): What happens when \`vulcan plan\` runs with a virtual layer: snapshot fingerprints, the context diff, breaking vs. non-breaking changes, backfill, restatement, and virtual-layer promotion.
- [Run and scheduling](https://v2.dataos.info/references/resources/vulcan/concepts/run-and-plan/run-and-scheduling.md): Understand what \`vulcan run\` does to refresh already-applied data: missing intervals, schedules, signals, and run activity.
- [Configurations](https://v2.dataos.info/references/resources/vulcan/configurations.md): Configure a Vulcan project with config.yaml, usage.yaml, and agreement.yaml: project settings, gateways, model defaults, the semantic layer, variables, execution hooks, the linter, notifications, depl
- [Usage guidance](https://v2.dataos.info/references/resources/vulcan/configurations/usage.md): What usage.yaml is for, how it differs from config.yaml, and a full key-by-key reference: what each key documents, when to add it, and when to leave the file minimal or omit it entirely.
- [Agreement](https://v2.dataos.info/references/resources/vulcan/configurations/agreement.md): What agreement.yaml is for, how it differs from config.yaml and usage.yaml, and a key-by-key reference: what each key documents, when to add it, and when to leave the file minimal or omit it entirely.
- [Variables](https://v2.dataos.info/references/resources/vulcan/configurations/variables.md): Configure Vulcan variables and environment overrides: .env files, the env\_var() function, VULCAN\_\_ configuration overrides, and dynamic values like the user() function.
- [Model defaults](https://v2.dataos.info/references/resources/vulcan/configurations/model-defaults.md): Configure model\_defaults in config.yaml: the required dialect, owner enforcement against config.users, identifier normalization strategy, and gateway-specific overrides.
- [Execution hooks](https://v2.dataos.info/references/resources/vulcan/configurations/execution-hooks.md): Configure before\_all and after\_all execution hooks to run SQL, macros, or statement files at the start and end of vulcan plan and vulcan run, with use cases and comparison to model-level pre/post stat
- [Linter](https://v2.dataos.info/references/resources/vulcan/configurations/linter.md): Configure the Vulcan linter: built-in and custom rules, Data Product-level rules, enabling rules via config.yaml, excluding models, and warn vs. error rule violation behavior.
- [Notifications](https://v2.dataos.info/references/resources/vulcan/configurations/notifications.md): Configure Vulcan notification targets and event types: Teams webhook, Slack webhook and API, and email/SMTP, plus per-user targets.
- [Models](https://v2.dataos.info/references/resources/vulcan/models.md): How Vulcan models fit together: physical data models, the semantic layer on top of them, and business metrics on top of that.
- [Data Models](https://v2.dataos.info/references/resources/vulcan/models/data-models.md): What a Vulcan data model is: the MODEL block, the SELECT query, naming and typing conventions, comment registration, and the day-to-day workflow for adding, editing, and deleting models.
- [Properties](https://v2.dataos.info/references/resources/vulcan/models/data-models/properties.md): General, storage, engine, and behavior properties for the \`MODEL\` block, plus kind-specific incremental properties and model naming.
- [Types](https://v2.dataos.info/references/resources/vulcan/models/data-models/types.md): SQL, Python, and external model types, and how to choose between them.
- [SQL models](https://v2.dataos.info/references/resources/vulcan/models/data-models/types/sql.md): SQL-based model definition, blueprinting, automatic dependencies, transpilation, and casting conventions.
- [Python models](https://v2.dataos.info/references/resources/vulcan/models/data-models/types/python.md): The \`@model\` decorator, execution context, dependencies, blueprinting, and DataFrame examples for Python models.
- [External models](https://v2.dataos.info/references/resources/vulcan/models/data-models/types/external-models.md): How \`EXTERNAL\` models describe schemas for tables Vulcan does not manage, and how to create and validate them.
- [Kinds](https://v2.dataos.info/references/resources/vulcan/models/data-models/model-kinds.md): INCREMENTAL, FULL, VIEW, EMBEDDED, SEED, and SCD Type 2 model kinds, and their materialization strategies by engine.
- [Statements](https://v2.dataos.info/references/resources/vulcan/models/data-models/statements.md): Pre-statements, post-statements, and on-virtual-update statements, and how to set defaults for them at the project level.
- [Semantic models](https://v2.dataos.info/references/resources/vulcan/models/semantic-models.md): Define business-friendly semantic models on top of Vulcan data models.
- [Dimensions](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/dimensions.md): Define semantic model dimensions, formatting, granularities, semantic behavior, and AI context.
- [Measures](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/measures.md): Define semantic model aggregations, measure types, filters, semantic behavior, and rolling windows.
- [Segments](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/segments.md): Define reusable semantic model filters for common business subsets.
- [Joins](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/joins.md): Define relationships between semantic models for cross-model analysis.
- [Rollups](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/rollups.md): Define pre-aggregation specs that build rollup tables for a semantic model.
- [AI context](https://v2.dataos.info/references/resources/vulcan/models/semantic-models/ai-context.md): Add AI and LLM guidance to semantic models, dimensions, measures, segments, joins, and granularities.
- [Business metrics](https://v2.dataos.info/references/resources/vulcan/models/business-metrics.md): Business metric YAML structure: required and optional properties, dimensions, segments, cross-model metrics, time granularity, forbidden legacy keys, reserved names, and validation rules.
- [Quality](https://v2.dataos.info/references/resources/vulcan/quality.md): Quality overview: how assertions, DQ checks, and unit tests work together to keep models trustworthy, and when to use each.
- [Assertions](https://v2.dataos.info/references/resources/vulcan/quality/assertions.md): Audits and assertions: built-in and user-defined rules that block bad rows.
- [Data quality](https://v2.dataos.info/references/resources/vulcan/quality/data-quality.md): DQ checks (\`kind: dq\`): rules, dimensions, and profiling.
- [Unit tests](https://v2.dataos.info/references/resources/vulcan/quality/tests.md): Unit tests: YAML fixtures for inputs and outputs, testing incremental models and CTEs, and running tests with the CLI.
- [Policies](https://v2.dataos.info/references/resources/vulcan/policies.md): Technical reference for Vulcan access policies, row filters, column masking, auth plugins, and physical-model-only policy inheritance.
- [Plugins and auth](https://v2.dataos.info/references/resources/vulcan/plugins-and-auth.md): The plugins/ folder auth extension: configuring the after\_authorize hook, AuthExtensionContext and SecurityContext, the resolve\_user\_groups example, and how policies use resolved groups and claims.
- [Roles and permissions](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions.md): Minimum engine privileges required to run a Vulcan domain resource, engine by engine: PostgreSQL, Snowflake, Databricks, Spark, and Trino.
- [PostgreSQL](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/postgres-minimum-permissions.md): Minimum PostgreSQL grants for a Vulcan deployment: source, target, and state schema privileges, by secret\_name tier.
- [Snowflake](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/snowflake-minimum-permissions.md): Minimum Snowflake grants for a Vulcan deployment: source, target, and state schema privileges, by secret\_name tier.
- [Databricks](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/databricks-minimum-permissions.md): Minimum Databricks Unity Catalog grants for a Vulcan deployment: source, target, and state schema privileges, by secret\_name tier.
- [Spark](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/spark-minimum-permissions.md): Minimum DataOS Lakehouse and object-storage permissions for running Vulcan on Spark, across both the DataOS and cloud IAM credential layers.
- [External Trino](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/external-trino-minimum-permissions.md): Minimum Trino SQL or connector access-control privileges for running Vulcan against a client-managed Trino cluster, by secret\_name tier.
- [Dedicated Trino](https://v2.dataos.info/references/resources/vulcan/roles-and-permissions/dedicated-trino-minimum-permissions.md): Permission model for a DataOS-managed Dedicated Trino deployment, where access control lives on each depot's underlying data source instead of Trino SQL grants.
- [Advanced features](https://v2.dataos.info/references/resources/vulcan/advanced-features.md): Advanced features that extend Vulcan beyond standard model definitions: macros, signals, custom materializations, and importing Snowflake semantic views.
- [Macros](https://v2.dataos.info/references/resources/vulcan/advanced-features/macros.md): Macros make SQL dynamic with variables and functions. Overview of Vulcan macros, Jinja macros, and the predefined variables both systems share.
- [Built-in](https://v2.dataos.info/references/resources/vulcan/advanced-features/macros/built-in.md): Reference for Vulcan's native macro system: user-defined variables, built-in macro operators for SQL clauses and control flow, and Python macro functions.
- [Jinja](https://v2.dataos.info/references/resources/vulcan/advanced-features/macros/jinja.md): Use Jinja templating in Vulcan SQL models: predefined and user-defined variables, for loops and if statements, and user-defined macro functions.
- [Variables](https://v2.dataos.info/references/resources/vulcan/advanced-features/macros/variables.md): Reference for Vulcan's predefined macro variables: temporal variables like @execution\_ds, runtime variables like @runtime\_stage and @this\_model, and before\_all/after\_all variables.
- [Signals](https://v2.dataos.info/references/resources/vulcan/advanced-features/signals.md): Signals add custom readiness checks before a scheduled model runs, so a model waits for late-arriving or upstream data before it processes an interval.
- [Custom materializations](https://v2.dataos.info/references/resources/vulcan/advanced-features/custom-materializations.md): Write a custom materialization in Python when built-in model kinds don't fit, using the CustomMaterialization and CustomKind classes, then package and share it across projects.
- [Import Snowflake semantic views](https://v2.dataos.info/references/resources/vulcan/advanced-features/import-snowflake-semantic-views.md): Import a Snowflake semantic view into a Vulcan project: configure the Snowflake gateway, run import\_semantic\_view and create\_external\_models, fix inputs.yaml, then plan and query the result over REST.
- [CLI commands](https://v2.dataos.info/references/resources/vulcan/cli.md): Every Vulcan CLI command, its options, and example output: planning and running models, auditing data quality, diffing environments, and managing state.
- [Deployment steps](https://v2.dataos.info/references/resources/vulcan/deployment.md): Deploy a Vulcan Data Product into a DataOS environment: prerequisites, the deployment: section in config.yaml, linting, deploying, and checking status.
- [Troubleshooting](https://v2.dataos.info/references/resources/vulcan/troubleshooting.md): Every validation and request error Vulcan returns, what triggered it, and how to fix it.
- [Nilus](https://v2.dataos.info/references/resources/nilus.md): Nilus is DataOS's unified data movement framework. It moves data from databases, SaaS APIs, event streams, and data warehouses into analytical destinations.
- [Quick start](https://v2.dataos.info/references/resources/nilus/quick-start-guides.md): Decision table for picking a pipeline mode, minimal batch/CDC config snippets, and a pre-production checklist for a first Nilus pipeline.
- [Sources overview](https://v2.dataos.info/references/resources/nilus/sources.md): Browse Nilus source connectors by pipeline types.
- [Core concepts](https://v2.dataos.info/references/resources/nilus/concepts.md): Core concepts that underpin Nilus, architecture, schema evolution, and data masking.
- [Architecture](https://v2.dataos.info/references/resources/nilus/concepts/architecture-and-mechanism.md): How Nilus is structured: the extract-normalize-load flow, the four pipeline types (batch, CDC, stream, metadata), and the extensible connector model.
- [Schema evolution](https://v2.dataos.info/references/resources/nilus/concepts/schema-evolution.md): Learn how Nilus handles schema changes over time across batch and CDC pipelines.
- [Data masking](https://v2.dataos.info/references/resources/nilus/concepts/understanding-data-masking.md): Learn how Nilus masks sensitive values during ingestion with configurable masking rules.
- [Secrets and projections](https://v2.dataos.info/references/resources/nilus/concepts/secrets-and-projections.md): How Nilus resolves credentials at runtime: automatic depot inference for dataos:// addresses, explicit secret projection for direct connector URIs, and repo-auth for custom source sync.
- [Roles and permissions](https://v2.dataos.info/references/resources/nilus/roles-and-permissions.md): How Nilus's privilege requirements map to pipeline mode, not to a single fixed role, plus the minimum grants for each supported source and destination.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/postgresql-minimum-permissions.md): Minimum PostgreSQL grants for each Nilus pipeline mode: batch read and write, CDC via logical replication, and Hera metadata ingestion.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/mongodb-minimum-permissions.md): Minimum MongoDB roles for each Nilus pipeline mode: batch read and write, change-stream CDC, and Hera metadata ingestion.
- [MS SQL Server](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/ms-sql-server-minimum-permissions.md): Minimum SQL Server grants for each Nilus pipeline mode: batch read and write, native CDC, and Hera metadata ingestion.
- [Snowflake](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/snowflake-minimum-permissions.md): Minimum Snowflake grants for each Nilus pipeline mode: batch read and write, and Hera metadata ingestion.
- [Databricks](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/databricks-minimum-permissions.md): Minimum Databricks Unity Catalog grants for each Nilus pipeline mode: batch read and write, and Hera metadata ingestion.
- [Kafka](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/kafka-minimum-permissions.md): Minimum Kafka ACLs for Nilus stream pipelines, across all four supported security protocols. Kafka is supported as a source only.
- [Salesforce](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/salesforce-minimum-permissions.md): Minimum Salesforce org permissions for Nilus batch pipelines: object and field-level read access, and the incremental strategy used per object.
- [Stripe](https://v2.dataos.info/references/resources/nilus/roles-and-permissions/stripe-minimum-permissions.md): Minimum Stripe API key permissions for Nilus batch pipelines: a Restricted Key scoped to read-only access on the ingested resource categories.
- [Batch data movement](https://v2.dataos.info/references/resources/nilus/batch.md): Conceptual overview of Nilus batch data movement: when to use it, how periodic extraction works, and how it differs from CDC and stream modes.
- [Batch pipeline config](https://v2.dataos.info/references/resources/nilus/batch/pipeline-config.md): Complete field reference for Nilus batch pipelines: spec structure, all source.options and sink.options, supported values, and annotated examples.
- [Batch sample configs](https://v2.dataos.info/references/resources/nilus/batch/sample-configs.md): Ready-to-use batch pipeline configurations covering full snapshots, incremental loads, and merge patterns across common source types.
- [Batch sources](https://v2.dataos.info/references/resources/nilus/batch/batch-sources.md): Index of Nilus batch source connectors by category (database vs. SaaS API), linking to each connector's requirements, options, and behavior.
- [AWS Redshift](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/aws-redshift.md): AWS Redshift batch source. Reads tables or custom SQL via the PostgreSQL-compatible wire protocol; the connector address uses the redshift:// scheme and rewrites it internally.
- [Azure Synapse](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/azure-synapse.md): Connector notes for Azure Synapse as a batch source: why there is no dedicated synapse:// scheme, connecting via the MS SQL Server path, and dedicated-pool caveats.
- [ClickHouse](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/clickhouse.md): ClickHouse batch source. Reads tables or custom SQL via the native ClickHouse protocol; TLS is enabled by default and the connector rewrites the URI internally before connecting.
- [Databricks](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/databricks.md): Databricks batch source. Reads tables and views from a Databricks SQL warehouse with optional incremental extraction; supports PAT and OAuth M2M authentication.
- [Delta Lake](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/delta-lake.md): Configuration reference for the Delta Lake batch source: the DuckDB-based read mechanism, supported storage backends, source options, and troubleshooting.
- [Google Analytics](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/google-analytics.md): Configuration reference for the Google Analytics 4 batch source: service-account setup, supported report resources, source/sink options, and sampling limits.
- [Google BigQuery](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/google-bigquery.md): Google BigQuery batch source. Reads tables, views, or custom SQL with optional incremental extraction; uses a service-account credential and optionally a DataOS depot for connection management.
- [Google Sheets](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/google-sheets.md): Google Sheets batch source. Loads a single sheet tab or A1 range per pipeline run; snapshot-oriented and best suited for analyst-maintained reference tables.
- [HubSpot](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/hubspot.md): HubSpot batch source. Extracts CRM objects (contacts, deals, companies, tickets, products, custom objects) as full-refresh snapshots via the Private App API.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/mongodb.md): MongoDB batch source. Reads documents with optional incremental ingestion; supports aggregation-pipeline queries by passing a pipeline array as the source\_table value.
- [MS SQL Server](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/ms-sql-server.md): MS SQL Server batch source. Reads tables or custom SQL via ODBC; supports SQL Authentication and Azure AD access-token authentication.
- [MySQL](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/mysql.md): MySQL batch source. Reads tables or custom SQL with optional incremental ingestion; the connector uses the pymysql driver and rewrites the URI internally.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/postgresql.md): Configuration reference for the PostgreSQL batch source: connectivity and grants, source options, sample configs, and troubleshooting for connection and incremental-load failures.
- [Salesforce](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/salesforce.md): Salesforce batch source. Extracts standard and custom CRM objects; selected high-change objects (accounts, opportunities) support automatic incremental loading via Salesforce modification timestamps.
- [Snowflake](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/snowflake.md): Configuration reference for the Snowflake batch source: warehouse and role setup, password vs. key-pair authentication, source options, and cost-control guidance.
- [Stripe](https://v2.dataos.info/references/resources/nilus/batch/batch-sources/stripe.md): Stripe batch source. Extracts payment objects (charges, customers, subscriptions, invoices, balance transactions) in async, sync, or sync-incremental loading modes.
- [Custom sources](https://v2.dataos.info/references/resources/nilus/batch/custom-sources.md): Index of custom-source paths for systems without a first-party connector: a hand-authored SQL query versus a fully custom Python source implementation.
- [Building a custom source](https://v2.dataos.info/references/resources/nilus/batch/custom-sources/creating-custom-source.md): Step-by-step reference for building a custom Nilus source: Git repo sync configuration, the implementation contract, source options, and troubleshooting.
- [Custom query (SQL)](https://v2.dataos.info/references/resources/nilus/batch/custom-sources/custom-query-sql-sources.md): Configuration reference for Custom Query batch sources: writing a query:-prefixed SQL source, incremental-column requirements, and common pitfalls.
- [Change data capture](https://v2.dataos.info/references/resources/nilus/cdc.md): Conceptual overview of Nilus Change Data Capture: how it reads database logs, handles initial snapshots, propagates row-level changes, and manages state.
- [CDC pipeline config](https://v2.dataos.info/references/resources/nilus/cdc/pipeline-config.md): Complete field reference for Nilus CDC pipelines: spec structure, all source.options, source.cdc settings, sink.options, and annotated examples.
- [CDC sample configs](https://v2.dataos.info/references/resources/nilus/cdc/sample-configs.md): Ready-to-use CDC pipeline configurations for PostgreSQL, MongoDB, MS SQL Server, MySQL, and IBM DB2 sources.
- [CDC sources](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources.md): Index of Nilus CDC source connectors (IBM DB2, MongoDB, MS SQL Server, MySQL, PostgreSQL), linking to per-connector setup, options, and troubleshooting.
- [IBM DB2](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/ibm-db2.md): Setup and operations reference for the IBM DB2 CDC source: ASN capture-service requirements, snapshot/stream mechanics, CDC options, and troubleshooting.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/mongodb.md): MongoDB CDC source. Captures document-level changes via replica-set change streams; covers oplog sizing, depot setup, snapshot modes, and strategy options.
- [MS SQL Server](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/ms-sql-server.md): MS SQL Server CDC source. Reads row-level changes from SQL Server CDC change tables; covers enabling CDC per table, permissions, and sink configuration.
- [MySQL](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/mysql.md): MySQL CDC source. Reads row-level changes from the binary log (binlog); supports GTID and non-GTID replication modes and multiple authentication plugins.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/cdc/cdc-sources/postgresql.md): Setup and operations reference for the PostgreSQL CDC source: logical replication setup, slot and publication management, WAL retention, and FAQs.
- [Stream pipelines](https://v2.dataos.info/references/resources/nilus/stream.md): Nilus stream pipelines connect to Kafka or NATS+JetStream and consume records in micro-batches and write them ti the destination.
- [Stream pipeline config](https://v2.dataos.info/references/resources/nilus/stream/pipeline-config.md): Complete field reference for Nilus Stream pipelines: spec structure, all source.options, source.cdc settings, sink.options, and annotated examples.
- [Stream sample configs](https://v2.dataos.info/references/resources/nilus/stream/sample-configs.md): Ready-to-use stream pipeline configurations for Kafka (plaintext, SASL/SSL, Confluent Cloud) and NATS+JetStream sources.
- [Stream sources](https://v2.dataos.info/references/resources/nilus/stream/stream-sources.md): Index of Nilus stream source connectors (Kafka, NATS) with a note on typical sink strategy for event-shaped records.
- [Kafka](https://v2.dataos.info/references/resources/nilus/stream/stream-sources/kafka.md): Kafka stream source. Consumes topics in micro-batches as a long-running consumer; supports plaintext, SASL/SSL, Confluent Cloud, and mTLS configurations.
- [NATS](https://v2.dataos.info/references/resources/nilus/stream/stream-sources/nats.md): Configuration reference for the NATS+JetStream stream source: authentication modes, ephemeral vs. durable consumers, source/sink options, and troubleshooting.
- [Metadata pipelines](https://v2.dataos.info/references/resources/nilus/metadata-pipelines.md): How Nilus metadata pipelines work: shallow vs. deep extraction modes, the stages each runs (inventory, lineage, profiling, classification, usage), and when to use each.
- [Metadata pipeline config](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/pipeline-config.md): This page explains the Nilus metadata pipeline config shape, a single workflow that extracts catalog metadata, column profiles, classification tags, query lineage, and query usage from a...
- [Metadata sample configs](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/sample-configs.md): Ready-to-use metadata pipeline configurations for Snowflake, Databricks, and DataOS Lakehouse (Iceberg) sources.
- [Metadata sources](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources.md): Index of Nilus metadata-capable sources, their supported shallow/deep stages, and how to scope extraction with database/schema/table filters.
- [Databricks](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/databricks-metadata.md): Configuration reference for the Databricks metadata source: supported stages, source options, required Unity Catalog/Hive Metastore grants, and troubleshooting.
- [DataOS Lakehouse](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/dataos-lakehouse-metadata.md): Configuration reference for the DataOS Lakehouse metadata-only source: what inventory it brings in, source options, required depot permissions, and troubleshooting.
- [Microsoft Fabric](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/microsoft-fabric.md): Configuration reference for the Microsoft Fabric Warehouse metadata source: supported stages, source options, required permissions, and troubleshooting.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/mongodb.md): Configuration reference for the MongoDB metadata source: supported stages, source options, required roles, and troubleshooting.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/postgresql.md): Configuration reference for the PostgreSQL metadata source: supported stages, source options, required grants, and troubleshooting.
- [Snowflake](https://v2.dataos.info/references/resources/nilus/metadata-pipelines/metadata-sources/snowflake-metadata.md): Configuration reference for the Snowflake metadata source: supported stages, account-usage permission requirements, source options, and troubleshooting.
- [Destinations](https://v2.dataos.info/references/resources/nilus/destinations.md): Index of Nilus destination connectors with supported pipeline modes and incremental strategies per destination, plus guidance on choosing a strategy.
- [Cloud warehouses](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses.md): Index of Nilus cloud warehouse destinations (BigQuery, Databricks, Redshift, Snowflake) linking to each connector's authentication and sink options.
- [BigQuery](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses/bigquery.md): BigQuery destination. Writes Nilus batch, CDC, and stream output into BigQuery datasets using a service-account credential; supports GCS staging and parquet loading.
- [Databricks](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses/databricks.md): Databricks destination. Writes Nilus batch, CDC, and stream output into Unity Catalog tables via a SQL warehouse; supports PAT and OAuth M2M authentication.
- [Redshift](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses/redshift.md): Configuration reference for the Redshift destination: connectivity and grants, sink options, distribution/sort-key guidance for merge, and troubleshooting.
- [Snowflake](https://v2.dataos.info/references/resources/nilus/destinations/cloud-warehouses/snowflake.md): Configuration reference for the Snowflake destination: warehouse and role requirements, password vs. key-pair authentication, sink options, and troubleshooting.
- [Databases](https://v2.dataos.info/references/resources/nilus/destinations/databases.md): Index of Nilus database destinations (PostgreSQL, MS SQL Server, MongoDB) linking to each connector's supported incremental strategies and sink options.
- [MongoDB](https://v2.dataos.info/references/resources/nilus/destinations/databases/mongodb.md): MongoDB destination. Writes Nilus batch, CDC, and stream output into collections; supports replace and merge strategies only. Append is explicitly rejected.
- [MS SQL Server](https://v2.dataos.info/references/resources/nilus/destinations/databases/ms-sql-server.md): Configuration reference for the MS SQL Server destination: ODBC driver and TLS setup, SQL vs. Azure AD authentication, sink options, and troubleshooting.
- [PostgreSQL](https://v2.dataos.info/references/resources/nilus/destinations/databases/postgresql.md): Configuration reference for the PostgreSQL destination: connectivity and grants, sink options, partitioned-table writes, and troubleshooting.
- [DataOS Lakehouse](https://v2.dataos.info/references/resources/nilus/destinations/dataos-lakehouse.md): The DataOS Lakehouse destination writes Apache Iceberg-backed datasets to object storage. Supported for AWS S3, Azure ADLS Gen2, and GCP GCS backends.
- [AWS-backed](https://v2.dataos.info/references/resources/nilus/destinations/dataos-lakehouse/aws-backed.md): AWS-backed DataOS Lakehouse destination. Writes Nilus batch, CDC, and stream output as Apache Iceberg datasets on Amazon S3; covers depot setup, secrets, and Iceberg sink options.
- [Azure-backed](https://v2.dataos.info/references/resources/nilus/destinations/dataos-lakehouse/azure-backed.md): Azure-backed DataOS Lakehouse destination. Writes Nilus batch, CDC, and stream output as Apache Iceberg datasets on ADLS Gen2 (ABFSS); covers depot setup, secrets, and Iceberg sink options.
- [GCP-backed](https://v2.dataos.info/references/resources/nilus/destinations/dataos-lakehouse/gcp-backed.md): GCP-backed DataOS Lakehouse destination. Writes Nilus batch, CDC, and stream output as Apache Iceberg datasets on Google Cloud Storage; covers depot setup, secrets, and Iceberg sink options.
- [Observability](https://v2.dataos.info/references/resources/nilus/observability.md): Index of Nilus observability surfaces (logs, API, metrics, Grafana) and the core signals to check for pipeline health.
- [API endpoints](https://v2.dataos.info/references/resources/nilus/observability/api-endpoints.md): Reference for Nilus Manager's read-only HTTP API: base URLs, authentication, and endpoints for run history, CDC offsets, and schema history.
- [Grafana dashboards](https://v2.dataos.info/references/resources/nilus/observability/grafana-dashboard.md): Guide to the two Nilus Grafana dashboards (Pipelines, Fleet Overview): what each answers, how to read the panels, and setup requirements.
- [Prometheus metrics](https://v2.dataos.info/references/resources/nilus/observability/prometheus-metrics.md): Reference for Nilus Prometheus metrics: which metrics answer whether a pipeline is running, slow, processing records, or healthy.
- [Pipeline optimization](https://v2.dataos.info/references/resources/nilus/pipeline-optimization.md): Symptom-to-page routing table for Nilus pipeline optimization, plus the measurement loop and common tuning knobs shared across guides.
- [Optimizing for time](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-time.md): Diagnostic guide for slow pipelines: bottleneck identification, tuning order, and the Iceberg file-size and worker-concurrency knobs that affect runtime.
- [Optimizing for resource](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimizing-for-resource.md): Diagnostic guide for high CPU, memory, or destination-cost usage: symptom table, tuning order, and Iceberg-load memory/volume guidance.
- [Optimize sink datasets](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets.md): Full knob taxonomy for sink tuning (correctness, shape, throughput, sampling), recommended config profiles, a sizing playbook, and common anti-patterns.
- [Shape knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-shape-knobs.md): Settings that control the physical layout of the destination dataset: partitioning, clustering, and nested-field flattening depth.
- [Sampling knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-sampling-knobs.md): Settings that cap extraction volume per run: sql\_limit and yield\_limit for controlled sampling and validation runs.
- [Throughput knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-throughput-knobs.md): Settings that control pipeline speed: page size, extract parallelism, loader file size, and worker counts.
- [Correctness knobs](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-correctness-knobs.md): Settings that control what data lands in the destination: incremental strategy, primary key, type hints, interval bounds, and column masking.
- [Tuning large Lakehouse (Iceberg) loads](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-lakehouse-iceberg-loads.md): Benchmark-backed configurations for large Iceberg loads to DataOS Lakehouse. Covers wide and narrow table shapes, file-size decisions, persistent volume setup, and ongoing table maintenance.
- [Destination gotchas and troubleshooting](https://v2.dataos.info/references/resources/nilus/pipeline-optimization/optimize-sink-datasets/optimize-destination-gotchas-and-troubleshooting.md): Destination-specific quirks and a troubleshooting matrix covering OOM errors, slow loads, duplicate rows, file bloat, and validation failures.
- [Troubleshooting](https://v2.dataos.info/references/resources/nilus/troubleshooting.md): Global triage checklist for Nilus pipeline failures: first-response steps, a symptom-to-page map, and evidence to collect before escalating.
- [Checking logs](https://v2.dataos.info/references/resources/nilus/troubleshooting/checking-logs.md): How to read Nilus runtime logs: what fields to capture, log-level configuration, common log patterns, and the recommended reading order.
- [Common errors](https://v2.dataos.info/references/resources/nilus/troubleshooting/common-errors.md): Catalog of common Nilus error messages and symptoms across pipeline modes, each paired with its likely cause and first recovery step.
- [Snowflake key-pair authentication](https://v2.dataos.info/references/resources/nilus/troubleshooting/snowflake-key-pair-authentication.md): Step-by-step guide to Snowflake key-pair authentication for Nilus: generating and registering RSA keys, depot configuration, and key rotation.
- [PostgreSQL partitioned tables for CDC](https://v2.dataos.info/references/resources/nilus/troubleshooting/postgresql-cdc-partitioned-tables.md): How to configure Nilus CDC for PostgreSQL declarative partitioning: including parent and child tables in the capture list, publication coverage, and a worked config example.
- [MongoDB CDC error 286](https://v2.dataos.info/references/resources/nilus/troubleshooting/mongodb-cdc-error-286.md): Error 286 indicates that a MongoDB change stream (which Nilus relies on for CDC) attempted to resume from a point that is no longer present in the replica‑set oplog (local.oplog.rs). When this...
- [MongoDB CDC BufferingChangeStreamCursor](https://v2.dataos.info/references/resources/nilus/troubleshooting/mongodb-cdc-bufferingchangestreamcursor.md): This warning comes from the Nilus MongoDB CDC reader. It means the runtime's in-memory buffer is full and cannot accept more events until the downstream consumer catches up.
- [Projections](https://v2.dataos.info/references/resources/projections.md): Use this guide to understand and build projections in DataOS for runtime dynamic configurations.
- [How projections work](https://v2.dataos.info/references/resources/projections/how-projections-work.md): Understand how DataOS renders, stores, and mounts projection output at runtime.
- [Best practices](https://v2.dataos.info/references/resources/projections/best-practices.md): Design secure, reusable, and maintainable projections for DataOS workloads.
- [Troubleshooting](https://v2.dataos.info/references/resources/projections/troubleshooting.md): Resolve common projection template, context, and file-mount issues.
- [Secret](https://v2.dataos.info/references/resources/secret.md): A Secret is a DataOS Resource designed for securely storing sensitive information at the DataOS Tenant-level
- [Data sources](https://v2.dataos.info/references/resources/secret/data-sources.md): Create tenant Secrets for supported data source credentials.
- [Azure Blob File System Secure (ABFSS)](https://v2.dataos.info/references/resources/secret/data-sources/azure-blob-file-system-secure-abfss.md): Create a Secret for securing ABFSS storage credentials.
- [Bigquery](https://v2.dataos.info/references/resources/secret/data-sources/bigquery.md): Create a Secret for securing BigQuery credentials.
- [Databricks](https://v2.dataos.info/references/resources/secret/data-sources/databricks.md): Create a Secret for securing Databricks credentials.
- [Google Cloud Storage (GCS)](https://v2.dataos.info/references/resources/secret/data-sources/google-cloud-storage-gcs.md): Create a Secret for securing Google Cloud Storage credentials.
- [Java Database Connectivity (JDBC)](https://v2.dataos.info/references/resources/secret/data-sources/java-database-connectivity-jdbc.md): Create a Secret for securing JDBC connection credentials.
- [Kafka](https://v2.dataos.info/references/resources/secret/data-sources/kafka.md): Create a Secret for securing Kafka credentials across multiple security modes.
- [Microsoft SQL Server (MSSQL) or Azure SQL](https://v2.dataos.info/references/resources/secret/data-sources/microsoft-sql-server-mssql-or-azure-sql.md): Create a Secret for securing Microsoft SQL Server or Azure SQL credentials.
- [MongoDB](https://v2.dataos.info/references/resources/secret/data-sources/mongodb.md): Create a Secret for securing MongoDB credentials.
- [MySQL](https://v2.dataos.info/references/resources/secret/data-sources/mysql.md): Create a Secret for securing MySQL credentials.
- [Opensearch](https://v2.dataos.info/references/resources/secret/data-sources/opensearch.md): Create a Secret for securing OpenSearch credentials.
- [Oracle](https://v2.dataos.info/references/resources/secret/data-sources/oracle.md): Create a Secret for securing Oracle database credentials.
- [PostgreSQL](https://v2.dataos.info/references/resources/secret/data-sources/postgresql.md): Create a Secret for securing PostgreSQL credentials.
- [Redshift](https://v2.dataos.info/references/resources/secret/data-sources/redshift.md): Create a Secret for securing Amazon Redshift credentials.
- [Simple Storage Service (Amazon S3)](https://v2.dataos.info/references/resources/secret/data-sources/simple-storage-service-amazon-s3.md): Create a Secret for securing Amazon S3 credentials.
- [Snowflake](https://v2.dataos.info/references/resources/secret/data-sources/snowflake.md): Steps to create Secret to store Snowflake credentials
- [RSA Key-pair](https://v2.dataos.info/references/resources/secret/data-sources/snowflake/rsa-key-pair.md): Steps to Generate and Register a Key Pair in Snowflake
- [Repositories](https://v2.dataos.info/references/resources/secret/repositories.md): Create tenant Secrets for supported repository credentials.
- [AWS CodeCommit](https://v2.dataos.info/references/resources/secret/repositories/aws-codecommit.md): Create a tenant Secret for securing AWS CodeCommit credentials
- [Bitbucket](https://v2.dataos.info/references/resources/secret/repositories/bitbucket.md): Create a Secret for securing Bitbucket repository credentials
- [Git](https://v2.dataos.info/references/resources/secret/repositories/git.md): Create a Secret for securing Git repository credentials
- [Container registries](https://v2.dataos.info/references/resources/secret/container-registries.md): Create a Secret for securing container registry (Docker) credentials used to pull private images
- [Consume](https://v2.dataos.info/references/resources/secret/consume.md): Consume Secret in other DataOS Resources.
- [Manifest configurations](https://v2.dataos.info/references/resources/secret/manifest-configuration.md): Following attributes/fields are declared for each Secret Resource object in its YAML manifest file. Some of these attributes need to be mandatorily declared, while others are optional.
- [Depot](https://v2.dataos.info/references/resources/depot.md): Depot is a Resource used to connect different data sources to DataOS.
- [Create depot](https://v2.dataos.info/references/resources/depot/supported-sources.md): Choose the source-specific guide for creating a Depot.
- [Amazon Redshift](https://v2.dataos.info/references/resources/depot/supported-sources/redshift.md): Create a Depot resource for Amazon Redshift connections.
- [Amazon S3](https://v2.dataos.info/references/resources/depot/supported-sources/s3.md): Create a Depot resource for Amazon S3 buckets.
- [Azure Blob File System Secure (ABFSS)](https://v2.dataos.info/references/resources/depot/supported-sources/abfss.md): Create a Depot resource for ABFSS object storage.
- [Databricks](https://v2.dataos.info/references/resources/depot/supported-sources/databricks.md): Create a Depot resource for Databricks workspaces.
- [Elasticsearch](https://v2.dataos.info/references/resources/depot/supported-sources/elasticsearch.md): Create a Depot resource for Elasticsearch clusters.
- [Eventhub](https://v2.dataos.info/references/resources/depot/supported-sources/eventhub.md): Create a Depot resource for Azure Event Hubs.
- [Google BigQuery](https://v2.dataos.info/references/resources/depot/supported-sources/bigquery.md): Create a Depot resource for Google BigQuery datasets.
- [Google Cloud Storage (GCS)](https://v2.dataos.info/references/resources/depot/supported-sources/gcs.md): Create a Depot resource for Google Cloud Storage buckets.
- [Java Database Connectivity (JDBC)](https://v2.dataos.info/references/resources/depot/supported-sources/jdbc.md): Create a Depot resource using a JDBC-compatible connection.
- [Kafka](https://v2.dataos.info/references/resources/depot/supported-sources/kafka.md): Create a Depot resource for Kafka topics and brokers across multiple security modes.
- [Lakehouse (Iceberg on ABFSS, GCS, or S3)](https://v2.dataos.info/references/resources/depot/supported-sources/lakehouse.md): Create a Depot resource for a Lakehouse backed by ABFSS, GCS, or Amazon S3.
- [Microsoft SQL Server (MSSQL) or Azure SQL](https://v2.dataos.info/references/resources/depot/supported-sources/mssql.md): Create a Depot resource for Microsoft SQL Server or Azure SQL.
- [MongoDB](https://v2.dataos.info/references/resources/depot/supported-sources/mongo.md): Create a Depot resource for MongoDB databases and collections.
- [MySQL](https://v2.dataos.info/references/resources/depot/supported-sources/mysql.md): Create a Depot resource for MySQL databases.
- [Opensearch](https://v2.dataos.info/references/resources/depot/supported-sources/opensearch.md): Create a Depot resource for OpenSearch clusters.
- [Oracle](https://v2.dataos.info/references/resources/depot/supported-sources/oracle.md): Create a Depot resource for Oracle databases.
- [PostgreSQL](https://v2.dataos.info/references/resources/depot/supported-sources/postgresql.md): Create a Depot resource for PostgreSQL databases.
- [Snowflake](https://v2.dataos.info/references/resources/depot/supported-sources/snowflake.md): Create a Depot resource for Snowflake databases and schemas.
- [Manifest configurations](https://v2.dataos.info/references/resources/depot/configurations.md): Reference Depot manifest fields and source-specific configuration options.
- [Supported connectors](https://v2.dataos.info/references/resources/depot/supported-connectors.md): This document lists the data sources supported by DataOS.
- [Add a depot type](https://v2.dataos.info/references/resources/depot/create-depot-type.md): Define a custom depot type so DataOS can recognize a new source system, validate its configuration, and resolve Uniform Data Links for it.
- [Consume](https://v2.dataos.info/references/resources/depot/consume.md): Consume data from Depot using DataOS interfaces and Stacks.
- [Data Ingestion (Nilus)](https://v2.dataos.info/references/resources/depot/consume/nilus.md): Consume Depot data through Nilus for data ingestion patterns.
- [Data Modeling and Transformation (Vulcan)](https://v2.dataos.info/references/resources/depot/consume/vulcan.md): Consume Depot data through Vulcan for semantic modeling and analytics.
- [Querying (Minerva)](https://v2.dataos.info/references/resources/depot/consume/querying.md): Consume Depot data through Minerva for interactive SQL querying.
- [Supported table format](https://v2.dataos.info/references/resources/depot/supported-table-format.md): Supported table formats that can be used with Depot.
- [Iceberg](https://v2.dataos.info/references/resources/depot/supported-table-format/iceberg.md): Using Iceberg table format in object storage Depots
- [Delta](https://v2.dataos.info/references/resources/depot/supported-table-format/delta.md): Using Delta table format in object storage Depots
- [Lakehouse](https://v2.dataos.info/references/resources/lakehouse.md): What a Lakehouse is, how it's built, and how to create and manage one.
- [Architecture](https://v2.dataos.info/references/resources/lakehouse/architecture.md): Understand the storage, catalog, orchestration, and compute components of a Lakehouse.
- [Create a Lakehouse](https://v2.dataos.info/references/resources/lakehouse/create-a-lakehouse.md)
- [Lakehouse manifest reference](https://v2.dataos.info/references/resources/lakehouse/configurations.md): Reference for the Lakehouse manifest structure and every supported configuration field.
- [Storage-specific configuration](https://v2.dataos.info/references/resources/lakehouse/supported-sources.md): Choose the storage-specific guide for creating a Lakehouse.
- [Lakehouse on ABFSS](https://v2.dataos.info/references/resources/lakehouse/supported-sources/lakehouse-on-abfss-1.md): Create a Lakehouse Resource on an Azure Blob File System Secure (ABFSS) data source.
- [Lakehouse on S3](https://v2.dataos.info/references/resources/lakehouse/supported-sources/lakehouse-on-s3-1.md): Create a Lakehouse Resource on an Amazon Simple Storage Service (S3) data source.
- [Lakehouse on GCS](https://v2.dataos.info/references/resources/lakehouse/supported-sources/lakehouse-on-gcs-1.md): Create a Lakehouse Resource on a Google Cloud Storage (GCS) data source.
- [Create a Lakehouse Depot](https://v2.dataos.info/references/resources/lakehouse/create-a-lakehouse-depot.md): Connect an existing Lakehouse Resource with a Lakehouse Depot.
- [Access control](https://v2.dataos.info/references/resources/lakehouse/lakehouse-access-control.md)
- [Command reference](https://v2.dataos.info/references/resources/lakehouse/command-reference.md): Lakehouse resource management commands
- [Minerva](https://v2.dataos.info/references/resources/minerva.md): Minerva and its role in the runtime layer.
- [Access control (WIP)](https://v2.dataos.info/references/resources/minerva/access-control.md): Roles and permissions for creating, managing, and querying Minerva.
- [Create a Minerva resource](https://v2.dataos.info/references/resources/minerva/create-a-minerva-resource.md): Create and verify a Minerva resource that exposes configured depots through a Trino-compatible endpoint.
- [Troubleshoot and monitor](https://v2.dataos.info/references/resources/minerva/troubleshoot-and-monitor.md): Diagnose Minerva resource failures and monitor runtime health.
- [Performance considerations](https://v2.dataos.info/references/resources/minerva/limitations.md): Understand where Minerva fits best, where it slows down, and when to use curated tables or Spark-based processing instead.
- [Bundle](https://v2.dataos.info/references/resources/bundle.md): Explains bundle resource and how they package workload assets.
- [Core concepts](https://v2.dataos.info/references/resources/bundle/core-concepts.md): Core concepts specific to bundle resource.
- [Field reference](https://v2.dataos.info/references/resources/bundle/field-reference.md): Reference for Bundle manifest fields and nested configuration.
- [Volume](https://v2.dataos.info/references/resources/volume.md): Explains the Volume resource and the two ways to attach persistent storage to a workload.
- [Service](https://v2.dataos.info/references/resources/service.md): Explains services for long-running and exposed workloads.
- [Core concepts](https://v2.dataos.info/references/resources/service/core-concepts.md): This section includes core concepts specific to a service resource
- [Example](https://v2.dataos.info/references/resources/service/example.md): End-to-end Service examples for common deployment patterns.
- [Field reference](https://v2.dataos.info/references/resources/service/field-reference.md): Reference for Service manifest fields and nested runtime configuration.
- [Worker](https://v2.dataos.info/references/resources/worker.md): Explains workers for background and task-based execution.
- [Core concepts](https://v2.dataos.info/references/resources/worker/core-concepts.md): Core concepts specific to Worker resources.
- [Example](https://v2.dataos.info/references/resources/worker/example.md): End-to-end Worker examples for common deployment patterns.
- [Field reference](https://v2.dataos.info/references/resources/worker/field-reference.md): Reference for Worker manifest fields and nested runtime configuration.
- [Workflow](https://v2.dataos.info/references/resources/workflow.md): Explains workflows for orchestrated, multi-step execution.
- [Core concepts](https://v2.dataos.info/references/resources/workflow/core-concepts.md): Core concepts specific to Workflow resources.
- [Example](https://v2.dataos.info/references/resources/workflow/example.md): End-to-end Workflow examples for common orchestration patterns.
- [Field reference](https://v2.dataos.info/references/resources/workflow/field-reference.md): Reference for Workflow manifest fields and DAG step configuration.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/resources.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.