> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/concepts/resources/nilus/concepts/schema-evolution.md).

# Schema Evolution

Schema evolution is how a pipeline handles columns, types, nested fields, and source-side Data Definition Language (DDL) as the source system changes over time.

## How schema changes appear

| Change              | Batch pipelines                                                                                           | CDC pipelines                                                                    |
| ------------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| New nullable column | Usually appears on the next extraction when the connector reflects the source object.                     | Captured through the source change stream and stored with schema history.        |
| Removed column      | May disappear from new loads, but destination cleanup depends on destination behavior and write strategy. | Treat as a compatibility event. Consumers may still see older rows or events.    |
| Type change         | May require `type_hints` or a destination table migration.                                                | Requires extra care because earlier and later events can carry different shapes. |
| Nested field growth | Controlled by `max_table_nesting` where the connector emits nested records.                               | Depends on connector payload shape and chosen CDC strategy.                      |

## Stable authoring patterns

* Use `type_hints` when the destination type must be pinned.
* Keep `primary_key` stable for merge-style loads.
* Keep CDC source identity stable so offsets and schema history continue to line up.
* Use `full_refresh: true` or a new destination table when a schema change is easier to rebuild than migrate.
* For nested records, tune `max_table_nesting` only after inspecting the source payload.

## CDC schema history

Nilus persists CDC schema history through the observability layer so you can inspect the schema context tied to a pipeline. Use the schema-history endpoints when a CDC pipeline fails after source DDL, a restart, or a connector configuration change.

## Destination considerations

| Destination style         | Watch for                                                                                                                   |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Lakehouse tables          | Partition and clustering changes should be planned. Existing data files do not automatically become physically reorganized. |
| Warehouses                | Type widening is usually safer than type narrowing. Confirm destination permissions for table changes.                      |
| Document or search stores | Mapping conflicts can block writes if a field changes type. Predefine mappings for high-value indexes.                      |
| File destinations         | Appended files may contain different schemas across time unless the reader handles evolution.                               |

## Related docs

* [Understanding Batch Pipeline Config](/concepts/resources/nilus/batch/pipeline-config.md)
* [Understanding CDC Pipeline Config](/concepts/resources/nilus/cdc/service-config.md)
* [Observability API Endpoints](/concepts/resources/nilus/observability/api-endpoints.md)
* [Common Errors](/concepts/resources/nilus/troubleshooting/common-errors.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://v2.dataos.info/concepts/resources/nilus/concepts/schema-evolution.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.