> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/references/dataos-resources/nilus/concepts/architecture-and-mechanism.md). # Architecture Enterprise data movement rarely fits one pattern. Some systems need scheduled bulk extraction. Others need continuous change capture. Teams also need a consistent way to onboard new sources and destinations without redesigning the runtime each time. Nilus gives batch and Change Data Capture (CDC) pipelines a common operating model while respecting the different semantics of each connector type. ## Overview Nilus is a unified, extensible data movement framework. It ingests data from diverse sources into lakehouse, warehouse, database, object-store, and other analytical destinations. It supports batch and CDC paradigms and abstracts extraction, normalization, and loading into one consistent pipeline shape. ## Core capabilities Nilus separates data movement into four distinct pipeline modes: ### Batch data movement * Batch pipelines collect and transfer data at scheduled intervals from databases, SaaS systems, files, or event-oriented sources into supported destinations. * This mode suits predictable, repeatable loads such as snapshots, periodic syncs, and analytical refreshes. * Connector-specific behavior still matters, but the execution model stays consistent at the Nilus layer. ### Change data capture * CDC pipelines listen to transactional change signals from supported operational databases and propagate inserts, updates, and deletes with low latency. * Nilus captures change events, applies the configured transformation strategy, and writes them into supported append-oriented or merge-capable destinations. * CDC support is curated more tightly than batch because each source must be validated for correctness, stability, and operational recovery. ### Metadata pipelines * Metadata pipelines extract catalog information rather than row-level data. They run against Snowflake, Databricks, and DataOS Lakehouse sources to harvest schema, table, and column metadata into the DataOS catalog. * Two depth modes exist: **shallow** extraction captures table and schema names without profiling data; **deep** extraction includes column statistics and profiling where the source supports it. * Metadata pipelines use the same `type: nilus` resource shape as batch pipelines and share the same secret and depot resolution model. ### Extensible connector model * You can add new sources without redesigning the runtime. * The product stays adaptable as source coverage expands. * The same Nilus resource shape can express very different connector contracts underneath. ## Flow * **Extract**: Nilus connects to the source and retrieves raw batch data or incremental change events. Extraction is schema-aware and tuned for connector-specific throughput. * **Normalize**: Nilus converts raw records into a consistent, consumable structure. It handles flattening, schema evolution, and variant cleanup where needed. * **Load**: Nilus writes normalized data into the destination using the connector's supported semantics. This includes object creation, chunked writes, retry-safe behavior, and operational monitoring. ## Constraints * Nilus exposes connector-specific capabilities through one consistent authoring model. Start with a familiar pipeline shape, then tune the details that matter for each source. * CDC coverage is narrower than batch coverage because it requires deeper source validation. * Destination readiness is connector-specific. Some destinations are mature; others are earlier in the lifecycle. * Throughput, memory, and file or batch sizing depend on workload shape and connector-specific options. ## Why this matters * You get one operating model for both batch and CDC instead of separate ingestion stacks. * The extract-normalize-load separation keeps pipelines easier to reason about, tune, and troubleshoot. * The connector model lets Nilus support a growing mix of sources and destinations without changing the core runtime contract. ## Related docs * [Understanding Batch Data Movement](/references/dataos-resources/nilus/batch.md) * [Understanding Change Data Capture](/references/dataos-resources/nilus/cdc.md) * [Understanding Stream Data Movement](/references/dataos-resources/nilus/stream.md) * [Metadata Pipelines](/references/dataos-resources/nilus/metadata-pipelines.md) * [Understanding Batch Pipeline Config](/references/dataos-resources/nilus/batch/pipeline-config.md) * [Understanding CDC Pipeline Config](/references/dataos-resources/nilus/cdc/pipeline-config.md) * [Sources](/references/dataos-resources/nilus/sources.md) * [Destinations](/references/dataos-resources/nilus/destinations.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/references/dataos-resources/nilus/concepts/architecture-and-mechanism.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.