MongoDB CDC BufferingChangeStreamCursor
This warning comes from the Nilus MongoDB CDC reader. It means the runtime's in-memory buffer is full and cannot accept more events until the downstream consumer catches up.
This article explains the BufferingChangeStreamCursor warning, what it means for your CDC pipeline, and how to resolve it.
This warning indicates that the MongoDB CDC pipeline temporarily stopped reading new changes because its internal buffer filled up.
It is not an error by itself, but if it continues for long periods, it can cause CDC lag, resume-token failures, or change stream interruptions.
To prevent the issue
Reduce CDC batch size.
Increase service memory and set memory limits (if not set already).
Make sure sink writes are healthy (check source event lag and sink commit behavior).
Avoid long-running Iceberg commits or high write spikes.
If it occurs
Verify whether CDC is still progressing.
Check sink performance.
Restart the CDC service only if offsets stop advancing.
Take action before the MongoDB oplog window is exceeded.
When the runtime's in-memory buffer is full and cannot accept more events until the downstream consumer catches up, the following warning comes from the Nilus MongoDB CDC reader:
Unable to acquire buffer lock, buffer queue is likely fullKey points
The warning does not mean data loss.
It can indicate backpressure in the pipeline.
If backpressure persists long enough, it can eventually lead to MongoDB change stream errors, such as:
ChangeStreamHistoryLostInvalidResumeToken
This document explains why it happens, how to mitigate it proactively, and what to do if it occurs.
Root cause
The warning appears when the event-production rate (Mongo writes) temporarily exceeds the event-consumption rate (Nilus processing + Iceberg writes). The typical contributing factors:
Downstream sink is slow (most common)
Nilus writes the CDC data to a destination. If these operations slow down, for example, due to:
Large Iceberg commits
Compaction or manifest rewrites
S3 throttling or retry storms
High latency writes
Then the Nilus CDC runtime cannot drain its queue fast enough. Result -> buffer fills -> warning appears.
Large CDC batch size
max.batch.size: 2048 (default)
May cause the CDC runtime to process very large chunks at once.
Large batches increase processing time and memory consumption, slowing down queue draining.
Insufficient memory
This can lead to:
Very small heap → too many minor GCs
Very large heap → long GC pauses
GC pauses → slow consumer thread → queue full
No Kubernetes memory limits
If container memory limits are not set:
The JVM may assume it has access to full node memory.
It may pick an inappropriate heap size automatically.
During heavy load, this causes GC pressure and stalls.
CPU contention
Heavy pipeline activity plus large batches may saturate CPU in a 1-replica setup.
High write bursts from MongoDB
During traffic spikes, the change stream volume can exceed regular processing capacity.
Prevention
The following configuration adjustments significantly reduce the likelihood of buffer-full conditions.
Reduce CDC batch size
Smaller batches mean faster downstream commits and steady buffer drain.
Allocate sufficient (or more) memory to the service
This stops the service from operating at the edge of its memory budget.
Define resource memory limits
Why this matters:
Predictable heap sizing
Reduced GC stalls
Improved CDC throughput
Resolution
Use the following checklist to assess whether the warning is transient or serious. Steps
Verify whether CDC is still progressing
Check the sink dataset:
Are new rows appearing?
Is the CDC timestamp (
_tsor equivalent) moving forward?Check offset logs:
If offsets are updating → pipeline is healthy, warning was transient.
Check if heartbeats are processing:
Are new heartbeats committed to the heartbeat dataset?
Latest heartbeat was commit timestamp; how far is it since the current timestamp?
Measure CDC lag
Compare the last ingested timestamp vs the MongoDB server time.
Check sink performance
Query the Nilus metadata table (stored in PostgreSQL) and check for the throughput.
Assess the stability of the warning
If the warning lasts < 5 seconds:
Normal temporary backpressure.
No action needed.
If the warning lasts ~5–30 seconds:
Monitor closely; CDC lag may grow.
Check sink and memory usage.
If the warning persists > 30 seconds:
Risk zone for
ChangeStreamHistoryLost.Take the following actions.
Restart the service.
This resets the Nilus CDC reader threads while preserving the pipeline position in the stream.
Useful MongoDB shell commands
Check oplog size and window
This helps determine the available oplog window (how far the CDC runtime can fall behind).
Estimate oplog window duration
This is crucial: if the CDC lag exceeds the oplog window, the change stream will fail.
Check recent write rate
Check collection-level throughput
Last updated
Was this helpful?