> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/resources/depot.md). # Depot Depot is a Resource used to connect different data sources to DataOS. Depot in DataOS is a Resource used to connect different data sources to DataOS by abstracting the complexities associated with the underlying source system (including protocols, credentials, and connection schemas). It enables users to establish connections and retrieve data from various data sources, such as file systems (e.g., AWS S3, Google GCS, Azure Blob Storage), data lake systems, database systems (e.g., Redshift, SnowflakeDB, BigQuery, Postgres), and event systems (e.g., Kafka), without moving the data. **What a UDL is, and why it matters** Once a source system is registered as a Depot, the Depot Service assigns it a unique address called a Uniform Data Link (UDL). Without a UDL, a Data Product would need to know the source system's native connection details and credentials directly — a different access pattern for every database, file store, or stream it touches. The UDL removes that friction: it gives a Data Product a single, consistent address for a dataset that works the same way across reads, writes, and transformations, regardless of what kind of system sits behind it, and it means credentials never need to be re-entered or hardcoded each time the dataset is accessed. In effect, the UDL is the handle through which everything else in DataOS — queries, pipelines, Data Products — refers to a piece of data, wherever it actually lives. {% hint style="warning" icon="hand-back-point-right" %} Depot Service is the DataOS Service that manages the Depot Resource. It introspects Depots and their underlying storage engines. After a Depot is created, you can read dataset details from it: constraints, partitions, indexing, and other metadata. {% endhint %} {% hint style="success" icon="circle-info" %} **Depot in the Data Product lifecycle** A Depot is the DataOS Resource that connects to a data source and exposes it for use, without copying the data. It hides differences between storage and database systems behind one access pattern so a Data Product can read from any registered source the same way. Key roles of a Depot in the lifecycle include: * **Data integration and access:** A Depot connects to file systems (AWS S3, Google GCS), databases (PostgreSQL, Snowflake), and event streams (Kafka) through Uniform Data Links (UDLs). Data Products read through the UDL without duplicating data. * **Standardization and abstraction:** By registering data locations within DataOS, a Depot standardizes access mechanisms, eliminating inconsistencies in connection protocols, schemas, and credentials across different systems. This simplifies Data Product development and ensures interoperability. * **Data transformation and querying:** Depots support ingestion, transformation, and query acceleration through DataOS Stacks (Vulcan, Nilus, and others), so data lands ready for analytical and operational use. * **Foundation for Data Products:** Depots are essential for creating Data Products, as they enable teams to query and transform source data. By using Depots, you can build scalable, secure, and efficient data ecosystems within DataOS, ensuring Data Products have reliable access to well-governed and high-quality data. {% endhint %} ### How a UDL is formed Once a source system is registered as a Depot, the Depot Service assigns it a unique address called a Uniform Data Link (UDL). This gives a Data Product one address for a dataset across reads, writes, and transformations, and removes the need to re-enter credentials every time. A UDL has two parts. #### Dataset mapping The first part is the dataset address, built as: ``` dataos://[depot]/[source-path] ``` Here, `[depot]` is the registered Depot's name, and `[source-path]` follows the native hierarchy of the connected source system — for a PostgreSQL Depot, for example, `public/orders` names the `public` schema and `orders` table within it, while object storage or other sources use their own native path structure. #### **Secret purpose** The second part is a `purpose` query parameter, written as `?purpose=`. Every Secret reference attached to a Depot carries a purpose — a logical label for a credential, such as read access versus write access — and this label must be unique within that Depot. The `purpose` parameter doesn't change the dataset or the path at all; it tells the Depot Service which purpose-specific Secret to resolve when a consumer connects, so the same path can use different credentials depending on what the connection is for. **Example** Putting both parts together: ``` dataos://icebase/analytics/orders?purpose=scan ``` Here `icebase` is the Depot, `analytics/orders` is the source path identifying the dataset, and `purpose=scan` selects the credential labelled `scan` for that connection. The same dataset address can be reused with a different purpose to resolve a different credential for a different kind of access — for instance, `purpose=query` for a read-oriented credential, or `purpose=rw` for a read/write credential — without the underlying path ever changing. Secret endpoints follow the same pattern: a call like `GET .../depots/{depot}/secrets?purpose=rw` returns the Secret mapped to the `rw` purpose for that Depot. #### **Valid purpose values and detailed rules** The `purpose` value must match one of the unique labels defined under `spec.secrets` in the Depot's configuration. Refer to the `secrets.purpose` configuration reference for the complete set of rules and valid entries — for example, valid values are lowercase and alphanumeric with underscores (`rw`, `read`, `write`, `scan`, `admin_key`), while values like `Read` (uppercase), `read_` (trailing underscore), or `read-write` (ending in a hyphen) are invalid. ### Depot access control Data Admins and other authorized administrators manage Depot access. They grant permissions to users or roles that need the data source. Developers can request access from a Data Admin. For permission types and grant instructions, see [Depot access control](https://v2.dataos.info/operate/access-control/roles-and-permissions/resource-level-permissions/depot-access-control). {% hint style="warning" icon="hand-back-point-right" %} A Depot provides access to data without moving or duplicating it. The data stays in the source system. When ingestion, querying, syndication, or copying is needed, use a DataOS Stack such as Nilus. {% endhint %} ### Depot connector support To see which data sources you can connect through a Depot, view [Supported connectors](/references/supported-connectors.md). It also shows connector support across Minerva, Nilus, and Vulcan. ### How to create a Depot? Use the [Data sources](/references/resources/depot/supported-sources.md) page to choose the source-specific guide for your system. ### Structure of a Depot manifest file To learn more about Depot manifest attributes, see [Manifest configurations](/references/resources/depot/configurations.md). ### Using Depots Once a Depot is created, you can use its Uniform Data Links (UDLs) to access data without physically moving data and perform further operations. The UDLs play a crucial role in various scenarios within DataOS. See [Using Depots](/references/resources/depot/consume.md) for common usage patterns. ### Best practices This section involves dos and don'ts for managing a Depot. * Give your Depot a meaningful name that represents its purpose. This makes it easier to manage and identify Depots. **Examples:** * `s3depot` instead of `s3-1`, as by `s3depot` it is easily interpretable that this Depot is establishing a connection with the Amazon S3 source. * `dbmysql` instead of `db1`, by `dbmysql` it is clear that this Depot is establishing the connection with the MySQL database, making it clear. * Before deleting a Depot, ensure no other downstream Resources are consuming it. ### Command reference A Depot has no dedicated `dataos-ctl` command group: manage it through the generic `resource` commands, scoped with `-t depot`. | Command | Purpose | Reference | | ----------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | `dataos-ctl resource apply -f ` | Create or update a Depot from a manifest | [→ full reference](https://v2.dataos.info/references/interfaces/command-line-interface/resource#dataos-ctl-resource-apply) | | `dataos-ctl resource get -t depot` | List Depots | [→ full reference](https://v2.dataos.info/references/interfaces/command-line-interface/resource#dataos-ctl-resource-get) | | `dataos-ctl resource get -t depot -n -d` | Detailed status of one Depot | [→ full reference](https://v2.dataos.info/references/interfaces/command-line-interface/resource#dataos-ctl-resource-get) | | `dataos-ctl resource delete -t depot -n ` | Delete a Depot | [→ full reference](https://v2.dataos.info/references/interfaces/command-line-interface/resource#dataos-ctl-resource-delete) | ### FAQs **1. How do I know which data sources are supported in DataOS?** You can refer to [Supported Connectors](/references/supported-connectors.md) to view all supported data sources. **2. What are Uniform Data Links (UDLs), and how do they work?** UDLs are standardized links that let you access and reference data across the environment without moving it. They provide consistent data access regardless of where the data resides within DataOS. **3. How can I use the data in a Depot without moving it?** DataOS allows compute-on-read and virtual access, so you can query and analyze data in-place without physically moving it. **4. Can users see everyone’s Depots or only their own?** Access to Depots is controlled by Data Admins. Users can only see the Depots they have access to. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://v2.dataos.info/references/resources/depot.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.