> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/dataos-resources/nilus/concepts/understanding-data-masking.md).

# Data masking

> Nilus masks values during ingestion. The source data stays unchanged, and the destination receives only the masked values.

Data masking is useful when you need analytical access to real datasets but do not want sensitive values to land in development, shared, or compliance-sensitive environments.

## Overview

Nilus applies masking in-flight as records move through the ingestion pipeline. A `mask` map in connector options drives the masking contract, where each entry follows this format:

```yaml
mask:
  column_name: algorithm[:parameter]
```

Nilus does not create a parallel unmasked copy. If you need original values in the destination, run the ingestion again without masking.

## Current support model

* The runtime applies masking when the pipeline receives a `MASK` configuration.
* Define `mask` under `sink.options`. This placement is also consistent for CDC-oriented pipelines.
* Older examples may show `mask` under `source.options`. That still works for many batch flows, but sink-side placement is the clearer standard.
* When a masking algorithm changes the output type or representation, update `type_hints` so the destination type matches the masked value.

## Configuration example

```yaml
name: masked-users-batch
version: v1alpha
type: nilus
spec:
  type: batch
  compute: universe-compute
  source:
    address: postgresql://{PG_USERNAME}:{PG_PASSWORD}@postgres.example.com:5432/commerce
    options:
      source_table: public.users
      type_hints:
        registration_date: text
        salary: bigint
      mask:
        email: hash
        phone: partial:3
        ssn: ssn
        card_number: credit_card
        salary: round:5000
        registration_date: year_only
        
  sink:
    address: dataos://masked-warehouse?purpose=rw
    options:
      dest_table: pii.users_masked
      incremental_strategy: replace

```

## Masking algorithms

Nilus supports the following masking algorithms:

### Hash and redaction

| Algorithm | Behavior                                                                                                                                                                |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hash`    | Returns a SHA-256 hash of the value. Consistent across runs: the same input always produces the same output. Use for anonymous identifiers and consistent tokenization. |
| `md5`     | Returns an MD5 hash of the value. Faster than SHA-256 but less secure; adequate for non-security checksums and tokenization.                                            |
| `redact`  | Replaces the value with `REDACTED`.                                                                                                                                     |
| `stars`   | Replaces the value with asterisks of the same length.                                                                                                                   |

### Format-preserving and partial masks

| Algorithm      | Behavior                                                               |
| -------------- | ---------------------------------------------------------------------- |
| `phone`        | Keeps the first three digits and masks the rest as `123-***-****`.     |
| `ssn`          | Returns `***-**-` plus the last four digits for a valid 9-digit value. |
| `first_letter` | Keeps the first character and masks the rest.                          |

### Tokenization and randomization

| Algorithm    | Behavior                                                                      |
| ------------ | ----------------------------------------------------------------------------- |
| `uuid`       | Assigns a stable UUID per distinct source value during the run.               |
| `sequential` | Assigns a stable sequential integer per distinct source value during the run. |
| `random`     | Generates a random replacement of the same general type.                      |

### Numeric and date masks

| Algorithm           | Behavior                                             |
| ------------------- | ---------------------------------------------------- |
| `range:<n>`         | Returns an ASCII range bucket such as `40000-50000`. |
| `date_shift:<days>` | Shifts dates by up to the supplied number of days.   |
| `month_year`        | Returns `YYYY-MM`.                                   |

## Practical notes

* Use `type_hints` when the masked output no longer matches the destination type you want to preserve.
* For CDC pipelines, use sink-side `mask` configuration so the settings are passed through consistently.
* Keep HMAC keys simple because the masking syntax uses colon separators.
* Some masks are deterministic (`hash`, `uuid`, `sequential`). Others intentionally change values (`random`, `noise`, `date_shift`).

## Best practices

* Use `hash`, `md5`, `redact`, `phone`, or `ssn` when the priority is Personally Identifiable Information (PII) protection.
* Use `uuid`, `sequential`, or `partial` when you need testable but still relationally useful output.
* Use `round`, `range`, `date_shift`, or `month_year` when the goal is analytical usefulness rather than exact value preservation.
* Validate masked output types before promoting the pipeline to a production-like environment.

## Limitations

* Masking happens during ingestion. It does not modify the source system.
* Nilus does not perform dynamic masking after ingestion. Once masked values are written to the destination, the destination dataset contains only those masked or hidden values for all downstream users.
* Nilus does not create a second unmasked destination dataset automatically.
* Masking adds work to the pipeline. Very wide datasets or many masked fields can increase runtime cost.

## Related docs

* [Understanding Batch Pipeline Config](/references/dataos-resources/nilus/batch/pipeline-config.md)
* [Understanding CDC Pipeline Config](/references/dataos-resources/nilus/cdc/pipeline-config.md)
* [Batch Sample Configs](/references/dataos-resources/nilus/batch/sample-configs.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/references/dataos-resources/nilus/concepts/understanding-data-masking.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.