Schema evolution
Learn how Nilus handles schema changes over time across batch and CDC pipelines.
Schema evolution is how a pipeline handles columns, types, nested fields, and source-side Data Definition Language (DDL) as the source system changes over time.
How schema changes appear
New nullable column on the source object
Usually appears on the next extraction when the connector reflects the source table or collection. Later pages in the same run can also add nullable columns on schema-flexible sources, such as MongoDB. Columns invented in a query: SELECT are not this case. See the callouts below.
Captured through the source change stream and stored with schema history.
Removed column
May disappear from new loads, but destination cleanup depends on destination behavior and write strategy.
Treat as a compatibility event. Consumers may still see older rows or events.
Type change
May require type_hints or a destination table migration.
Requires extra care because earlier and later events can carry different shapes.
Nested field growth
Controlled by max_table_nesting. Nested objects expand up to that depth. A nested array at that level becomes a child table, rather than a JSON column. See Shape Knobs → max_table_nesting.
Depends on connector payload shape and chosen CDC strategy.
Stable authoring patterns
Use
type_hintswhen the destination type must be pinned, or when a sparse column may be all-null on early pages.Keep
primary_keystable for merge-style loads.Keep CDC source identity stable so offsets and schema history continue to line up.
Use a new destination table, or
DROPthe existing table and reload, when rebuilding is easier than migration.For nested records, tune
max_table_nestingonly after inspecting the source payload. Arrays at that level become child tables, not JSON columns.
CDC schema history
Nilus persists CDC schema history through the observability layer so operators can inspect the schema context tied to a pipeline. Use the schema-history endpoints when a CDC pipeline fails after source DDL, a restart, or a connector configuration change.
Destination considerations
Lakehouse tables
Partition and clustering changes should be planned. Existing data files do not automatically become physically reorganized. Table reads and query: also write different decimal precision and scale. Do not switch extract styles on the same dest_table.
Warehouses
Type widening is usually safer than type narrowing. Confirm destination permissions for table changes.
Document or search stores
Mapping conflicts can block writes if a field changes type. Predefine mappings for high-value indexes.
File destinations
Appended files may contain different schemas across time unless the reader handles evolution.
Related docs
Last updated
Was this helpful?