For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs.
Skip to content

Schema evolution

Learn how Nilus handles schema changes over time across batch and CDC pipelines.

Schema evolution is how a pipeline handles columns, types, nested fields, and source-side Data Definition Language (DDL) as the source system changes over time.

How schema changes appear

Change
Batch pipelines
CDC pipelines

New nullable column on the source object

Usually appears on the next extraction when the connector reflects the source table or collection. Later pages in the same run can also add nullable columns on schema-flexible sources, such as MongoDB. Columns invented in a query: SELECT are not this case. See the callouts below.

Captured through the source change stream and stored with schema history.

Removed column

May disappear from new loads, but destination cleanup depends on destination behavior and write strategy.

Treat as a compatibility event. Consumers may still see older rows or events.

Type change

May require type_hints or a destination table migration.

Requires extra care because earlier and later events can carry different shapes.

Nested field growth

Controlled by max_table_nesting. Nested objects expand up to that depth. A nested array at that level becomes a child table, rather than a JSON column. See Shape Knobs → max_table_nesting.

Depends on connector payload shape and chosen CDC strategy.

Later pages can add nullable columns in the same run.

Batch extraction reads in pages. If later pages introduce fields that earlier pages lack, Nilus unions those columns onto the destination schema as nullable. This is common with MongoDB and other document sources. Documents without the field land as NULL. An existing destination table missing the column is widened on the next run.

This does not apply to columns invented by a SQL query:. Do not switch a query: onto a table-read dest_table, or the reverse.

A column that is all-null on every early page can be dropped from inferred schema before a later page has a value. Pin that column with type_hints so it remains in the schema. See Correctness Knobs.

Incompatible types still fail. The union supports new nullable columns, not type changes.

Stable authoring patterns

  • Use type_hints when the destination type must be pinned, or when a sparse column may be all-null on early pages.

  • Keep primary_key stable for merge-style loads.

  • Keep CDC source identity stable so offsets and schema history continue to line up.

  • Use a new destination table, or DROP the existing table and reload, when rebuilding is easier than migration.

  • For nested records, tune max_table_nesting only after inspecting the source payload. Arrays at that level become child tables, not JSON columns.

CDC schema history

Nilus persists CDC schema history through the observability layer so operators can inspect the schema context tied to a pipeline. Use the schema-history endpoints when a CDC pipeline fails after source DDL, a restart, or a connector configuration change.

Destination considerations

Destination style
Watch for

Lakehouse tables

Partition and clustering changes should be planned. Existing data files do not automatically become physically reorganized. Table reads and query: also write different decimal precision and scale. Do not switch extract styles on the same dest_table.

Warehouses

Type widening is usually safer than type narrowing. Confirm destination permissions for table changes.

Document or search stores

Mapping conflicts can block writes if a field changes type. Predefine mappings for high-value indexes.

File destinations

Appended files may contain different schemas across time unless the reader handles evolution.

Last updated

Was this helpful?