For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs.
Skip to content

MongoDB CDC BufferingChangeStreamCursor

This warning comes from the Nilus MongoDB CDC reader. It means the runtime's in-memory buffer is full and cannot accept more events until the downstream consumer catches up.

This article explains the BufferingChangeStreamCursor warning, what it means for your CDC pipeline, and how to resolve it.

  • This warning indicates that the MongoDB CDC pipeline temporarily stopped reading new changes because its internal buffer filled up.

  • It is not an error by itself, but if it continues for long periods, it can cause CDC lag, resume-token failures, or change stream interruptions.

To prevent the issue

  • Reduce CDC batch size.

  • Increase service memory and set memory limits (if not set already).

  • Make sure sink writes are healthy (check source event lag and sink commit behavior).

  • Avoid long-running Iceberg commits or high write spikes.

If it occurs

  • Verify whether CDC is still progressing.

  • Check sink performance.

  • Restart the CDC service only if offsets stop advancing.

  • Take action before the MongoDB oplog window is exceeded.

When the runtime's in-memory buffer is full and cannot accept more events until the downstream consumer catches up, the following warning comes from the Nilus MongoDB CDC reader:

Unable to acquire buffer lock, buffer queue is likely full

Key points

  • The warning does not mean data loss.

  • It can indicate backpressure in the pipeline.

  • If backpressure persists long enough, it can eventually lead to MongoDB change stream errors, such as:

  • ChangeStreamHistoryLost

  • InvalidResumeToken

This document explains why it happens, how to mitigate it proactively, and what to do if it occurs.

Root cause

The warning appears when the event-production rate (Mongo writes) temporarily exceeds the event-consumption rate (Nilus processing + Iceberg writes). The typical contributing factors:

  1. Downstream sink is slow (most common)

Nilus writes the CDC data to a destination. If these operations slow down, for example, due to:

  • Large Iceberg commits

  • Compaction or manifest rewrites

  • S3 throttling or retry storms

  • High latency writes

Then the Nilus CDC runtime cannot drain its queue fast enough. Result -> buffer fills -> warning appears.

  1. Large CDC batch size

max.batch.size: 2048 (default)

  • May cause the CDC runtime to process very large chunks at once.

  • Large batches increase processing time and memory consumption, slowing down queue draining.

  1. Insufficient memory

This can lead to:

  • Very small heap → too many minor GCs

  • Very large heap → long GC pauses

  • GC pauses → slow consumer thread → queue full

  1. No Kubernetes memory limits

If container memory limits are not set:

  • The JVM may assume it has access to full node memory.

  • It may pick an inappropriate heap size automatically.

  • During heavy load, this causes GC pressure and stalls.

  1. CPU contention

Heavy pipeline activity plus large batches may saturate CPU in a 1-replica setup.

  1. High write bursts from MongoDB

During traffic spikes, the change stream volume can exceed regular processing capacity.

Prevention

The following configuration adjustments significantly reduce the likelihood of buffer-full conditions.

  1. Reduce CDC batch size

Smaller batches mean faster downstream commits and steady buffer drain.

Example
  1. Allocate sufficient (or more) memory to the service

This stops the service from operating at the edge of its memory budget.

  1. Define resource memory limits

Why this matters:

  • Predictable heap sizing

  • Reduced GC stalls

  • Improved CDC throughput

Resolution

Use the following checklist to assess whether the warning is transient or serious. Steps

  1. Verify whether CDC is still progressing

  • Check the sink dataset:

  • Are new rows appearing?

  • Is the CDC timestamp (_ts or equivalent) moving forward?

  • Check offset logs:

  • If offsets are updating → pipeline is healthy, warning was transient.

  • Check if heartbeats are processing:

  • Are new heartbeats committed to the heartbeat dataset?

  • Latest heartbeat was commit timestamp; how far is it since the current timestamp?

  1. Measure CDC lag

  • Compare the last ingested timestamp vs the MongoDB server time.

How to check MongoDB Server Time

Replica set members rely on synchronized clocks for:

  • Oplog timestamp ordering

  • Heartbeat timeouts and election timing

  • Write concern “majority” acknowledgment

Run this in your MongoDB shell:

**Check the time on all Replica Set Members**

  1. Check sink performance

  • Query the Nilus metadata table (stored in PostgreSQL) and check for the throughput.

Query
  1. Assess the stability of the warning

  • If the warning lasts < 5 seconds:

  • Normal temporary backpressure.

  • No action needed.

  • If the warning lasts ~5–30 seconds:

  • Monitor closely; CDC lag may grow.

  • Check sink and memory usage.

  • If the warning persists > 30 seconds:

  • Risk zone for ChangeStreamHistoryLost.

  • Take the following actions.

  • Restart the service.

This resets the Nilus CDC reader threads while preserving the pipeline position in the stream.

Useful MongoDB shell commands

  1. Check oplog size and window

This helps determine the available oplog window (how far the CDC runtime can fall behind).

  1. Estimate oplog window duration

This is crucial: if the CDC lag exceeds the oplog window, the change stream will fail.

  1. Check recent write rate

  1. Check collection-level throughput

Last updated

Was this helpful?