Depot
Depot is a Resource used to connect different data sources to DataOS.
Depot in DataOS is a Resource used to connect different data sources to DataOS by abstracting the complexities associated with the underlying source system (including protocols, credentials, and connection schemas). It enables users to establish connections and retrieve data from various data sources, such as file systems (e.g., AWS S3, Google GCS, Azure Blob Storage), data lake systems, database systems (e.g., Redshift, SnowflakeDB, BigQuery, Postgres), and event systems (e.g., Kafka), without moving the data.
What a UDL is, and why it matters
Once a source system is registered as a Depot, the Depot Service assigns it a unique address called a Uniform Data Link (UDL). Without a UDL, a Data Product would need to know the source system's native connection details and credentials directly — a different access pattern for every database, file store, or stream it touches. The UDL removes that friction: it gives a Data Product a single, consistent address for a dataset that works the same way across reads, writes, and transformations, regardless of what kind of system sits behind it, and it means credentials never need to be re-entered or hardcoded each time the dataset is accessed. In effect, the UDL is the handle through which everything else in DataOS — queries, pipelines, Data Products — refers to a piece of data, wherever it actually lives.
How a UDL is formed
Once a source system is registered as a Depot, the Depot Service assigns it a unique address called a Uniform Data Link (UDL). This gives a Data Product one address for a dataset across reads, writes, and transformations, and removes the need to re-enter credentials every time.
A UDL has two parts.
Dataset mapping
The first part is the dataset address, built as:
Here, [depot] is the registered Depot's name, and [source-path] follows the native hierarchy of the connected source system — for a PostgreSQL Depot, for example, public/orders names the public schema and orders table within it, while object storage or other sources use their own native path structure.
Secret purpose
The second part is a purpose query parameter, written as ?purpose=<value>. Every Secret reference attached to a Depot carries a purpose — a logical label for a credential, such as read access versus write access — and this label must be unique within that Depot. The purpose parameter doesn't change the dataset or the path at all; it tells the Depot Service which purpose-specific Secret to resolve when a consumer connects, so the same path can use different credentials depending on what the connection is for.
Example
Putting both parts together:
Here icebase is the Depot, analytics/orders is the source path identifying the dataset, and purpose=scan selects the credential labelled scan for that connection. The same dataset address can be reused with a different purpose to resolve a different credential for a different kind of access — for instance, purpose=query for a read-oriented credential, or purpose=rw for a read/write credential — without the underlying path ever changing. Secret endpoints follow the same pattern: a call like GET .../depots/{depot}/secrets?purpose=rw returns the Secret mapped to the rw purpose for that Depot.
Valid purpose values and detailed rules
The purpose value must match one of the unique labels defined under spec.secrets in the Depot's configuration. Refer to the secrets.purpose configuration reference for the complete set of rules and valid entries — for example, valid values are lowercase and alphanumeric with underscores (rw, read, write, scan, admin_key), while values like Read (uppercase), read_ (trailing underscore), or read-write (ending in a hyphen) are invalid.
Depot access control
Data Admins and other authorized administrators manage Depot access. They grant permissions to users or roles that need the data source. Developers can request access from a Data Admin.
For permission types and grant instructions, see Depot access control.
Depot connector support
To see which data sources you can connect through a Depot, view Supported connectors. It also shows connector support across Minerva, Nilus, and Vulcan.
How to create a Depot?
Use the Data sources page to choose the source-specific guide for your system.
Structure of a Depot manifest file
To learn more about Depot manifest attributes, see Manifest configurations.
Using Depots
Once a Depot is created, you can use its Uniform Data Links (UDLs) to access data without physically moving data and perform further operations. The UDLs play a crucial role in various scenarios within DataOS.
See Using Depots for common usage patterns.
Best practices
This section involves dos and don'ts for managing a Depot.
Give your Depot a meaningful name that represents its purpose. This makes it easier to manage and identify Depots.
Examples:
s3depotinstead ofs3-1, as bys3depotit is easily interpretable that this Depot is establishing a connection with the Amazon S3 source.dbmysqlinstead ofdb1, bydbmysqlit is clear that this Depot is establishing the connection with the MySQL database, making it clear.
Before deleting a Depot, ensure no other downstream Resources are consuming it.
Command reference
A Depot has no dedicated dataos-ctl command group: manage it through the generic resource commands, scoped with -t depot.
FAQs
1. How do I know which data sources are supported in DataOS?
You can refer to Supported Connectors to view all supported data sources.
2. What are Uniform Data Links (UDLs), and how do they work?
UDLs are standardized links that let you access and reference data across the environment without moving it. They provide consistent data access regardless of where the data resides within DataOS.
3. How can I use the data in a Depot without moving it?
DataOS allows compute-on-read and virtual access, so you can query and analyze data in-place without physically moving it.
4. Can users see everyone’s Depots or only their own?
Access to Depots is controlled by Data Admins. Users can only see the Depots they have access to.
Last updated
Was this helpful?