> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/build/understand/connect-sources.md).

# Connect sources

Register an authenticated connection to a source so DataOS can reach it without moving the data.

Before DataOS can read or transform your data, it needs a registered, authenticated connection to the source. This is the first step of understanding a source: you make it reachable, then you scan and explore it. This page shows how to check whether a connection already exists and how to create one if it does not.

A data connection in DataOS is two Resources working together:

* **Secret** stores credentials (username, password, API keys, certificates) in an encrypted vault. Credentials never sit in manifests or code. See [Secret](https://v2.dataos.info/references/resources/secret) in References.
* **Depot** connects DataOS to the external system and references the Secret for authentication. A Depot holds the connection details: where the source is and how to reach it.

  See [Depot](https://v2.dataos.info/references/resources/depot) in References.

{% hint style="info" %}
**Data connection is not the same as an engine.**

A data connection (Secret + Depot) answers: *where does the source data live, and how do we authenticate to it?* It makes an external system reachable inside DataOS without moving data.

An engine (Snowflake, Databricks, Postgres, Trino, Spark, SQL Server, Microsoft Fabric, Amazon Redshift, or Amazon Athena) answers: *where do transformations run, and where do outputs land?* You configure an engine later, in [Productize → Connect engine](https://v2.dataos.info/build/productize/connect-engine.md).

The same system can play both roles. A Snowflake account can be a Depot (read source tables) and the engine (run transforms). Each role is configured separately.
{% endhint %}

## Before you begin

* Ensure the DataOS CLI is installed and initialized. See [CLI setup](https://v2.dataos.info/build/get-started/prerequisites/cli-setup.md).
* Ensure you have permissions to create Secrets and Depots. A user with the Tenant Admin, Data Admin, or Data Developer Tenant role can create a Depot. Check with `dataos-ctl user get`, or ask your administrator. Without the right role, applying a Secret or Depot fails with a permission error.

## 1. Check whether a connection already exists

Many teams share Depots, so a connection to your source may already be live. Do not create a duplicate.

List every Depot in your tenant:

```bash
dataos-ctl resource get -t depot -a
```

```
      NAME           | VERSION |  TYPE  | STATUS | RUNTIME |       OWNER
---------------------+---------+--------+--------+---------+----------------
 snowflakesales      | v2alpha | depot  | active |         | johndoetmdcio
 postgrescrm         | v2alpha | depot  | active |         | johndoetmdcio
```

If a Depot for your source is `active`, note its name and use it directly through its Uniform Data Link (UDL): `dataos://[depot-name]/[source-path]?purpose=[purpose]`. If you don't own the Depot, request a `Can Use` grant from its owner or a Data Admin first. Skip the rest of this page and go to [Scan metadata](https://v2.dataos.info/build/understand/scan-metadata.md) or [Explore data](https://v2.dataos.info/build/understand/explore-data.md).

You can also check in the UI: open **Datasets** in the left navigation and look for your source in the source tree. See [Datasets](https://v2.dataos.info/build/understand/explore-data/datasets.md) for the walkthrough.

If no Depot covers your source, continue.

## 2. Create a Secret

A Secret holds the credentials DataOS uses to authenticate. The fields depend on the source; for source-specific examples, see [Secret setup guides](https://v2.dataos.info/references/resources/secret/data-sources) in References.

Write the manifest:

```yaml
name: ${{secret-name}}
version: v2alpha
type: secret
description: ${{what this secret is for}}
owner: ${{your-dataos-user-id}}
secret:
  type: key-value
  data:
    username: ${{source-username}}
    password: ${{source-password}}
```

For **file-based credentials** (a BigQuery JSON key), use `secret.files` instead of `secret.data`. A Snowflake RSA key pair stays in `secret.data`: set `auth_mode: key-pair` and paste the private key inline under `key:`.

{% hint style="warning" %}
Do not commit Secret manifests. The `secret.data` block is plaintext. After applying the Secret, delete the local file or move it outside your repo.
{% endhint %}

Apply and confirm:

```bash
dataos-ctl resource apply -f ${{path-to-secret.yaml}}
dataos-ctl resource get -t secret
```

A `STATUS` of `active` means it is ready. Note the name; the Depot references it next.

## 3. Create a Depot

A Depot registers the source inside DataOS and references the Secret. Once active, DataOS addresses it through a UDL with two parts: the dataset address `dataos://[depot-name]/[source-path]`, and a `?purpose=<label>` parameter that tells the Depot which of its Secrets to use for the connection. Every tool in DataOS uses this address, so there are no credentials at the point of use.

The Depot name must be under 48 characters and match `[a-z]([a-z0-9]*)?`: lowercase letters and digits only, starting with a letter. Hyphens, underscores, and uppercase are not allowed.

```yaml
name: ${{depot-name}}
version: v2alpha
type: depot
description: ${{what this depot connects to}}
owner: ${{your-dataos-user-id}}
layer: user
spec:
  type: ${{source-type}}          # e.g. snowflake, s3, postgresql, bigquery
  spec:
    # source-specific connection fields
  secrets:
    - id: "${{tenant-id}}:${{secret-name}}"
      purpose: rw
    - id: "${{tenant-id}}:${{secret-name}}"
      purpose: scan
    - id: "${{tenant-id}}:${{secret-name}}"
      purpose: query
```

The `secrets` block ties the Depot to the Secret. A single Depot can have several Secrets attached, each representing a different purpose (a permission context, for example full access, masked or read-only access, or limited access).<br>

A purpose is a label you choose, and each label must be unique within a Depot. `rw`, `scan`, and `query` are the common conventions; any other label, such as `full` or `masked`, is a custom one:

| Purpose  | Typical use                                                                                                                                                                                  |
| -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `rw`     | Use this when read-write access is required and permitted.                                                                                                                                   |
| `scan`   | A common label for a metadata or schema-discovery credential. Metadata pipelines in [Scan metadata](https://v2.dataos.info/build/understand/scan-metadata.md) use `?purpose=rw`, not `scan`. |
| `query`  | Use this when you only need to query and explore data.                                                                                                                                       |
| `full`   | Custom label: the credential has complete access to the data source.                                                                                                                         |
| `masked` | Custom label: the credential masks some of the data.                                                                                                                                         |

{% hint style="info" %}
What a credential can actually do (read, write, see masked data, or nothing) is set by the source system's own roles and policies.
{% endhint %}

<details>

<summary>Snowflake example</summary>

```yaml
name: snowflakesales
version: v2alpha
type: depot
description: Snowflake depot for the sales schema
owner: johndoetmdcio
layer: user
spec:
  type: snowflake
  spec:
    url: ${{account-url}}
    account: ${{snowflake-account}}
    database: ${{database-name}}
    warehouse: ${{warehouse-name}}
    role: ${{role-name}}
  secrets:
    - id: "engineering:snowflake-cred"
      purpose: rw
    - id: "engineering:snowflake-cred"
      purpose: scan
    - id: "engineering:snowflake-cred"
      purpose: query
```

</details>

{% hint style="info" %}
Reuse the same Secret for all three, or split them if your source needs different credentials per operation.
{% endhint %}

Apply the Depot:

```bash
dataos-ctl resource apply -f ${{path-to-depot.yaml}}
```

For manifests covering other sources (S3, BigQuery, PostgreSQL, Kafka, and others), see [supported Depots](https://v2.dataos.info/references/resources/depot/supported-sources) in References.

## 4. Verify the connection

```bash
dataos-ctl resource get -t depot
```

A `STATUS` of `active` means the Depot Service has registered it. You now reference data through the UDL. For the Snowflake example configured for `SALES_DB`, the `orders` table in the `public` schema, read with the `query` credential, is:

```
dataos://snowflakesales/public/orders?purpose=query
```

Use this address in scans, ingestion, transformations, and Workbench. Change the `purpose` value to pick a different Secret for the same dataset.

## Next

The source is connected. If its metadata is not yet discoverable in DataOS, [scan it](https://v2.dataos.info/build/understand/scan-metadata.md). Otherwise, [explore the data](https://v2.dataos.info/build/understand/explore-data.md).

## Troubleshooting

<details>

<summary>Permission error when applying a Secret or Depot</summary>

```
ERROR: forbidden: user does not have required role data-dev
```

Ask your administrator to assign the required role, then retry. Creating a Depot needs the Tenant Admin, Data Admin, or Data Developer Tenant role.

</details>

<details>

<summary>Depot status is not <code>active</code> after applying</summary>

The Depot failed to validate the connection. Inspect it:

```bash
dataos-ctl resource get -t depot -n ${{depot-name}} --details
```

Common causes: wrong credentials in the Secret, an incorrect host/port/database in `spec.spec`, or a network restriction between DataOS and the source.

</details>

<details>

<summary>Secret deletion blocked by a dependent Depot</summary>

Delete the Depot first, then the Secret:

```bash
dataos-ctl resource delete -t depot -n ${{depot-name}}
dataos-ctl resource delete -t secret -n ${{secret-name}}
```

</details>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://v2.dataos.info/build/understand/connect-sources.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
