> For the complete documentation index, see [llms.txt](https://v2.dataos.info/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://v2.dataos.info/concepts/resources/nilus/cdc/cdc-sources/ibm-db2.md). # IBM DB2 IBM DB2 is supported as a CDC source for capturing inserts, updates, and deletes at row granularity. Nilus's CDC engine reads from the CDC change tables maintained by the ASN capture service. Nilus persists offsets through its REST-backed offset store and persists DDL through the REST schema-history store, so a restart resumes from the last acknowledged position with the correct schema context. Unlike most Nilus database connectors, IBM DB2 has no DataOS depot type. Connect through a direct `db2://` URI and project the username and password as secrets. For `spec.type: cdc`, Nilus applies its Debezium CDC engine to that URI automatically. This page describes the CDC surface. If you need a periodic batch snapshot from DB2 instead, see [Custom Query (SQL Sources)](/concepts/resources/nilus/batch/custom-sources/custom-query-sql-sources.md) for the generic SQL batch manifest shape and `source_table` syntax. ## Connectivity IBM DB2 connects through a direct URI only; there is no DataOS depot variant for DB2. | Form | Address example | Notes | | -------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | Connection URI | `db2://{DB2_USERNAME}:{DB2_PASSWORD}@:50000/` | Direct DB2 URI (default port `50000`). For CDC pipelines, author `db2://`; Nilus applies the Debezium engine. | Defaults applied by Nilus: * `topic.prefix = cdc` (override per pipeline; the prefix is also used as the sink table prefix) * Each change event is emitted with the new row state and timestamp metadata; deletes preserve the original primary key. * DDL is persisted through the REST schema-history store. ## Requirements > *These requirements must be satisfied at the database before Nilus can run a DB2 CDC pipeline.* ### ASN capture service installed and started Nilus's DB2 CDC connector relies on DB2 SQL replication. The ASN capture components must be installed and the `ASNCDC` control schema available. ```sql -- Start (or verify) the capture service VALUES ASNCDC.ASNCDCSERVICES('start','asncdc'); -- Put a table into capture mode CALL ASNCDC.ADDTABLE('MYSCHEMA', 'MYTABLE'); -- Reinitialize after adding or removing captured tables VALUES ASNCDC.ASNCDCSERVICES('reinit','asncdc'); -- Stop / status when needed VALUES ASNCDC.ASNCDCSERVICES('stop','asncdc'); VALUES ASNCDC.ASNCDCSERVICES('status','asncdc'); ``` Tables must be put into capture mode before Nilus can stream them. Adding or removing tables requires a reinitialize call so the capture service rebuilds its change-table set. ### Privileges * The Nilus connection user needs network access to the DB2 server, `SELECT` on captured source tables, and `SELECT` on the metadata views the connector reads during the initial snapshot. * The administrator who runs `ADDTABLE` / `ASNCDCSERVICES` typically needs `DBADM` or equivalent privileges. * DB2 CDC requires the corresponding IIDR licensing, even though installing IIDR itself is not required. ### Data type and feature constraints * DB2's SQL replication path does not support `BOOLEAN` columns; tables that contain them are snapshot-only and cannot stream changes. Validate captured tables for `BOOLEAN` columns before rollout. * Tables without a primary key cannot stream updates and deletes reliably. * Schema changes on captured tables typically require redoing the `ADDTABLE` / `reinit` cycle. ## Core concepts 1. **Change tables, populated by the ASN capture agent** * DB2 CDC works through the ASN capture service, which reads the transaction log and writes changes into DB2 change tables under the CDC control schema. * Nilus reads from those change tables rather than tailing the transaction log directly. * The capture agent's health is part of the CDC contract, if it stops, no new change rows are produced. 2. **Snapshot + stream flow** * On first start, Nilus snapshots the included tables consistently and emits one event per row with `op = r`. * The connector then transitions to streaming changes from the capture tables. * On restart, the persisted offset is reused so the snapshot phase is skipped. 3. **Offset and schema-history persistence** * Offsets are persisted to the Nilus REST offset store; restart points to the last acknowledged change-table position. * DDL is persisted to the REST schema-history store, so the connector can rebuild schemas without reading from the source. 4. **Operational risk areas** * Capture-agent health, change-table latency, and reinitialization after onboarding new tables are the most common production concerns. * Long outages can push the persisted offset behind retention; a fresh snapshot is then required. ## Sample Nilus config
Nilus YAML, DB2 CDC → Lakehouse ```yaml name: ncdc-db2-to-lakehouse version: v1alpha type: nilus tags: - nilus-cdc description: Nilus CDC pipeline for IBM DB2 → DataOS Lakehouse spec: type: cdc compute: universe-compute use: projection: secrets: - id: engineering:db2-secret contextAlias: db2secret projections: envVars: - key: DB2_USERNAME template: "{{ secrets['db2secret'].username | base64_decode }}" - key: DB2_PASSWORD template: "{{ secrets['db2secret'].password | base64_decode }}" source: address: db2://{DB2_USERNAME}:{DB2_PASSWORD}@db2.example.com:50000/SAMPLE options: strategy: flatten cdc: table.include.list: "MYSCHEMA.CUSTOMERS,MYSCHEMA.ORDERS" topic.prefix: "db2_cdc" snapshot.mode: initial heartbeat.interval.ms: 60000 max.batch.size: 2048 max.queue.size: 8192 sink: address: dataos://analytics-lakehouse options: dest_table: db2_cdc_customers incremental_strategy: append ```
The source `address` is a direct `db2://` URI; `{DB2_USERNAME}` and `{DB2_PASSWORD}` are resolved from the projected secret at runtime. The Lakehouse sink still uses a DataOS depot (`dataos://analytics-lakehouse`). ## CDC options These are the most useful CDC connector properties for DB2. They go under `source.cdc` in the Nilus manifest. | Option | Default | Description | | -------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `table.include.list` | *No default* | Comma-separated list of fully qualified `schema.table` identifiers to capture. Tables must already be in capture mode. | | `table.exclude.list` | *No default* | Inverse of `table.include.list`. | | `column.include.list` | *No default* | `schema.table.column` patterns to include in change events. | | `column.exclude.list` | *No default* | Inverse of `column.include.list`. | | `topic.prefix` | `cdc` *(Nilus default)* | Logical namespace for this pipeline. Must be unique across CDC pipelines. Nilus also uses it as the sink table prefix. | | `snapshot.mode` | `initial` | Snapshot behavior on first start. `initial` snapshots when no offset is recorded; `always` re-snapshots every start; `schema_only` captures only schema; `when_needed` snapshots when the recorded offset is no longer available. | | `heartbeat.interval.ms` | `0` (off) | Periodic heartbeats keep the persisted offset advancing on idle databases. Recommended: `60000`. | | `max.batch.size` | `2048` | Maximum number of change events processed per batch. | | `max.queue.size` | `8192` | Maximum buffered change events before back-pressure kicks in. | | `offset.flush.interval.ms` | `60000` | How often offsets are persisted to the Nilus REST offset store. | | `strategy` | `flatten` | `flatten` emits row-shaped records (one row per change). `changelog` preserves the full CDC envelope. Set under `source.options`, not `source.cdc`. | ## Behavior and capabilities * **Pipeline mode**: IBM DB2 CDC runs as a `cdc` source. * Re-run the `reinit` ASN service action after onboarding new tables or making capture-mode changes; otherwise the connector does not see the new change tables. * Keep `topic.prefix` stable after the first production deployment because downstream naming depends on it. * Monitor capture-agent lag at the DB2 layer as well as Nilus lag. If the ASN agent falls behind, downstream symptoms appear late and after the connector is already healthy-looking. * Use narrow `table.include.list` patterns instead of broad capture scopes when onboarding a new database. * Validate `BOOLEAN`-column tables ahead of time, they are snapshot-only on the SQL replication path. ## Troubleshooting * **Symptom:** The pipeline starts but no change events arrive. * **Cause:** The ASN capture service is not running, or the required table was never added to capture mode. * **Recovery:** Verify with `VALUES ASNCDC.ASNCDCSERVICES('status', ...)`, add the table with `ASNCDC.ADDTABLE`, reinitialize, and redeploy. * **Symptom:** DB2 changes are visible in the source table but not downstream. * **Cause:** The capture agent is lagging or failing before change tables are updated. * **Recovery:** Check capture-agent health, review DB2-side errors, and confirm change tables are advancing. * **Symptom:** The pipeline fails after a long outage or a schema change. * **Cause:** Nilus can no longer resume cleanly from the retained change history, or the captured table definition changed. * **Recovery:** Revalidate captured tables, reinitialize the capture service if needed, and start with a fresh snapshot. * **Symptom:** Updates / deletes are missing from the change stream for a specific table. * **Cause:** The table has no primary key, or it contains `BOOLEAN` columns that the SQL replication path cannot stream. * **Recovery:** Add a primary key, exclude the unsupported columns, or treat the table as snapshot-only. ## Related docs * [Understanding CDC Pipeline Config](/concepts/resources/nilus/cdc/service-config.md) * [CDC Sample Configs](/concepts/resources/nilus/cdc/sample-configs.md) * [Understanding Change Data Capture](/concepts/resources/nilus/cdc.md) * **DataOS Lakehouse destinations**: see the [AWS-backed DataOS Lakehouse](/concepts/resources/nilus/destinations/dataos-lakehouse/aws-backed.md), [Azure-backed DataOS Lakehouse](/concepts/resources/nilus/destinations/dataos-lakehouse/azure-backed.md), or [GCP-backed DataOS Lakehouse](/concepts/resources/nilus/destinations/dataos-lakehouse/gcp-backed.md) variants for writing CDC change events into a Lakehouse. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://v2.dataos.info/concepts/resources/nilus/cdc/cdc-sources/ibm-db2.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.