> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/v1/understand/connect-sources.md). # Connect sources Before DataOS can read or transform your data, it needs a registered, authenticated connection to the source. This is the first step of understanding a source: you make it reachable, then you scan and explore it. This page shows how to check whether a connection already exists and how to create one if it does not. A data connection in DataOS is two Resources working together: * **Secret** stores credentials (username, password, API keys, certificates) in an encrypted vault. Credentials never sit in manifests or code. See [Secret](https://v2.dataos.info/references/resources/secret) in References. * **Depot** connects DataOS to the external system and references the Secret for authentication. A Depot holds the connection details, where the source is and how to reach it. See [Depot](https://v2.dataos.info/references/resources/depot) in References. {% hint style="info" %} **Data connection is not the same as an engine.** A data connection (Secret + Depot) answers: *where does the source data live, and how do we authenticate to it?* It makes an external system reachable inside DataOS without moving data. An engine (Snowflake, Databricks, Postgres, Trino, Spark, SQL Server, or Microsoft Fabric) answers: *where do transformations run, and where do outputs land?* You configure an engine later, in [Productize → Connect engine](/build/v1/productize/connect-engine.md). The same system can play both roles. A Snowflake account can be a Depot (read source tables) and the engine (run transforms). Each role is configured separately. {% endhint %} ## Before you begin * Ensure the DataOS CLI is installed and initialized. See [CLI setup](/build/v1/get-started/prerequisites/cli-setup.md). * Ensure you have permissions to create Secrets and Depots. Check with `dataos-ctl user get`, or ask your administrator. Without the right role, applying a Secret or Depot fails with a permission error. ## 1. Check whether a connection already exists Many teams share Depots, so a connection to your source may already be live. Do not create a duplicate. List every Depot in your tenant: ```bash dataos-ctl resource get -t depot -a ``` ``` NAME | VERSION | TYPE | STATUS | RUNTIME | OWNER ---------------------+---------+--------+--------+---------+---------------- snowflake-sales | v2alpha | depot | active | | johndoetmdcio postgres-crm | v2alpha | depot | active | | johndoetmdcio ``` If a Depot for your source is `active`, note its name and use it directly through its Uniform Data Link (UDL): `dataos://[depot-name]/[source-path]`. Skip the rest of this page and go to [Scan metadata](/build/v1/understand/scan-metadata.md) or [Explore data](/build/v1/understand/explore-data.md). You can also check in the UI: open **Datasets** in the left navigation and look for your source in the source tree. See [Datasets](/build/v1/understand/explore-data/datasets.md) for the walkthrough. If no Depot covers your source, continue. ## 2. Create a Secret A Secret holds the credentials, DataOS uses to authenticate. The fields depend on the source; for source-specific examples, see [Secret setup guides](https://v2.dataos.info/references/resources/secret/data-sources) in References. Write the manifest: ```yaml name: ${{secret-name}} version: v2alpha type: secret description: ${{what this secret is for}} owner: ${{your-dataos-user-id}} secret: type: key-value data: username: ${{source-username}} password: ${{source-password}} ``` For **file-based credentials** (a BigQuery JSON key, a Snowflake RSA private key), use `secret.files` instead of `secret.data`. {% hint style="warning" %} Do not commit Secret manifests. The `secret.data` block is plaintext. After applying the Secret, delete the local file or move it outside your repo. {% endhint %} Apply and confirm: ```bash dataos-ctl resource apply -f ${{path-to-secret.yaml}} dataos-ctl resource get -t secret ``` A `STATUS` of `active` means it is ready. Note the name; the Depot references it next. ## 3. Create a Depot A Depot registers the source inside DataOS and references the Secret. Once active, DataOS addresses it through a UDL: `dataos://[depot-name]/[source-path]`. Every tool in DataOS uses this address, so there are no credentials at the point of use. ```yaml name: ${{depot-name}} version: v2alpha type: depot description: ${{what this depot connects to}} owner: ${{your-dataos-user-id}} layer: user spec: type: ${{SOURCE_TYPE}} # e.g. SNOWFLAKE, S3, POSTGRES, BIGQUERY spec: # source-specific connection fields secrets: - id: "${{tenant}}:${{secret-name}}" purpose: rw - id: "${{tenant}}:${{secret-name}}" purpose: scan - id: "${{tenant}}:${{secret-name}}" purpose: query ``` The `secrets` block ties the Depot to the Secret. A single Depot can have several Secrets attached, each representing a different purpose (permission context, for example, full access, masked or read-only access, or limited access.
Some common convention to define a purpose: | Purpose | Typical use | | -------- | ---------------------------------------------------------------- | | `rw` | Use this when read-write access is required and permitted. | | `scan` | Use this when metadata scanning or schema discovery is required. | | `query` | Use this when you only need to query and explore data. | | `full` | the credential has complete access to the data source | | `masked` | the credentials masks some of the data | {% hint style="info" %} What a credential can actually do (read, write, see masked data, or nothing) is set by the source system's own roles and policies. {% endhint %}
Snowflake example ```yaml name: snowflakesales version: v2alpha type: depot description: Snowflake depot for the sales schema owner: johndoetmdcio layer: user spec: type: snowflake spec: url: ${{account-url}} database: ${{database-name}} warehouse: ${{warehouse-name}} role: ${{role-name}} secrets: - id: "engineering:snowflake-cred" purpose: rw - id: "engineering:snowflake-cred" purpose: scan - id: "engineering:snowflake-cred" purpose: query ```
{% hint style="info" %} Reuse the same Secret for all three, or split them if your source needs different credentials per operation. {% endhint %} Apply the Depot: ```bash dataos-ctl resource apply -f ${{path-to-depot.yaml}} ``` For manifests covering other sources (S3, BigQuery, PostgreSQL, Kafka, and others), see [supported Depots](https://v2.dataos.info/references/resources/depot/supported-sources) in References. ## 4. Verify the connection ```bash dataos-ctl resource get -t depot ``` A `STATUS` of `active` means the Depot Service has registered it. You now reference data through the UDL. For the Snowflake example configured for `SALES_DB`, the `orders` table in the `public` schema is: ``` dataos://snowflakesales/public/orders ``` Use this address in scans, ingestion, transformations, and Workbench. ## Next The source is connected. If its metadata is not yet discoverable in DataOS, [scan it](/build/v1/understand/scan-metadata.md). Otherwise, [explore the data](/build/v1/understand/explore-data.md). ## Troubleshooting
Permission error when applying a Secret or Depot ``` ERROR: forbidden: user does not have required role data-dev ``` Ask your administrator to assign the required role (role names vary by tenant), then retry.
Depot status is not active after applying The Depot failed to validate the connection. Inspect it: ```bash dataos-ctl resource get -t depot -n ${{depot-name}} --details ``` Common causes: wrong credentials in the Secret, an incorrect host/port/database in `spec.spec`, or a network restriction between DataOS and the source.
Secret deletion blocked by a dependent Depot Delete the Depot first, then the Secret: ```bash dataos-ctl resource delete -t depot -n ${{depot-name}} dataos-ctl resource delete -t secret -n ${{secret-name}} ```
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/build/v1/understand/connect-sources.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.