For the complete documentation index, see llms.txt. This page is also available as Markdown.

Secrets and projections

How Nilus resolves credentials at runtime: automatic depot inference for dataos:// addresses, explicit secret projection for direct connector URIs, and repo-auth for custom source sync.

This guide explains how secrets are resolved in current nilus resources. There are three secret-handling patterns in Nilus:

  1. Explicit projection under spec.use.projection for direct connector URIs.

  2. Automatic Depot inference for dataos:// source and sink addresses.

  3. Automatic repo-auth projection for spec.repo.secretId.

The shortest rule of thumb:

  • dataos://...purpose= means Nilus implies the Depot connection.

  • {ENV_VAR} inside a direct URI means you must project that value yourself.

  • spec.repo.secretId is only for repo-backed extension sync.

Mental model

Nilus secret handling has two layers:

  1. The Nilus domain template decides which Secrets or Depots must be projected into the runtime.

  2. The Nilus runtime consumes those projected values to build the final source and sink configuration.

Separate two questions:

  1. How does the stack project the values?

  2. How does Nilus consume those values in the authored resource?

Pattern 1: explicit projection for direct connector URIs

Use explicit projection when the source or sink address contains placeholders such as {USERNAME}, {PASSWORD}, {TOKEN}, or {CREDENTIALS_B64}. This is the right pattern for connectors configured directly in the URI rather than inferred from a dataos:// Depot. Typical cases:

  • API-style connectors configured directly in the URI.

  • Connectors that expect username/password values in the address.

  • Connectors that expect token-based values in the address.

  • Connectors that expect a base64 or file-derived credential blob.

How it works

Declare the secret under spec.use.projection.secrets, expose selected values through spec.use.projection.projections.envVars, then reference those values inside the authored URI.

Example: username / password / token projection
Example: encoded credential projection

Projection rules

  1. Put secret selection under spec.use.projection.secrets.

  2. Expose only the fields the connector needs.

  3. Reference those values through {ENV_VAR} placeholders in the authored URI.

  4. Match the secret shape to the connector contract.

  5. Prefer clear env-var names such as API_TOKEN, CONNECTOR_USERNAME, or CREDENTIALS_B64.

Pattern 2: automatic Depot inference for dataos://

If spec.source.address or spec.sink.address starts with dataos://, Nilus treats that address as a Depot reference and projects the matching Depot metadata and secret into the runtime automatically. This is the simplest path when a connector should be resolved from a Depot rather than assembled manually.

How purpose works

purpose is required on every dataos:// Depot address. Always add it, even when the Depot defines only a single secret purpose. Nilus does not assume a default purpose, so a Depot reference without ?purpose= fails to resolve.

Use ro for read-oriented source access and rw for write-oriented sink access. When a Depot exposes more than one secret purpose, purpose also selects which secret to project. When it exposes only one, state it explicitly.

Example: fully inferred Depot credentials

What happens behind the scenes

The domain template inspects the dataos:// addresses, projects the matching Depot metadata and purpose-specific secret, then Nilus converts those projected values into the actual connector configuration at runtime. That's why you don't need to hand-assemble a database URI when a Depot already represents that connection.

You can still add spec.use.projection.projections.envVars next to a dataos:// address when the runtime needs a tuning env var that the depot does not supply (for example MSSQL_IGNORE_NO_TRANSACTION_ON_ROLLBACK on Azure Synapse). Credentials stay inferred from the depot.

Source-specific environment variables

Project these with spec.use.projection.projections.envVars. They are not substitutes for source.options. Only document variables the Nilus runtime actually reads.

Variable
Sources
Default
When to set it

MSSQL_IGNORE_NO_TRANSACTION_ON_ROLLBACK

Azure Synapse and MS SQL Server depot sources

false

Set "true" when extract fails because the server rolled back with no active transaction. Common on Synapse dedicated SQL pools. Direct mssql:// URIs use the query parameter ignore_no_transaction_on_rollback=true instead. See Azure Synapse and MS SQL Server.

BQ_LOCATION

Google BigQuery depot sources

US

Set the BigQuery job location so it matches the dataset region (EU, asia-southeast1, …). If unset, Nilus logs a warning and uses US. Direct bigquery:// URIs use the location query parameter instead. See Google BigQuery.

SCHEMA_NAMING

Any source that goes through Nilus identifier mapping

default

Set direct to keep source table and column names (pkey stays pkey). This is env-only, do not put schema_naming under source.options. See Correctness knobs.

PostgreSQL, Snowflake, and other SQL sources do not currently take extra connector env vars beyond credentials. Use the depot or URI, plus source.options, for those connectors.

Literal values such as "true" do not need a secret. Put secrets in spec.use.projection.secrets only when the env var is a credential.

Pattern 3: repo secret inference

The Nilus resource can also infer repo-auth credentials for repo-backed extensions through spec.repo.secretId.

This secret is not used for source or sink connectivity. It is used by the stack when it needs to authenticate repository synchronization for a repo-backed extension.

When secrets are not inferred automatically

Secrets are not inferred just because a connector is sensitive. You still need explicit projection for:

  • Any direct connector URI you assemble yourself.

  • Any address containing {ENV_VAR} placeholders.

  • Token-based or username/password connectors that are not referenced through dataos://.

Decision guide

  • If the address is dataos://..., let the Depot supply the connection.

  • If the address contains placeholders like {USERNAME} or {TOKEN}, use spec.use.projection.

  • If the pipeline needs repository synchronization for an extension, use spec.repo.secretId.

Common mistakes

  • Expecting direct connector credentials to be inferred automatically.

  • Forgetting to project env vars for a non-dataos:// connector.

  • Omitting purpose from a dataos:// Depot address. It is required on every Depot reference, including single-purpose Depots.

  • Mixing a fully inferred Depot address with unnecessary explicit connector credentials.

  • Forgetting to project a required tuning env var, such as MSSQL_IGNORE_NO_TRANSACTION_ON_ROLLBACK on Synapse, when the depot does not set the matching parameter.

Last updated

Was this helpful?