Observability
Index of Nilus observability surfaces (logs, API, metrics, Grafana) and the core signals to check for pipeline health.
Nilus observability is the set of signals that helps operators answer four questions: did the pipeline run, did it move the expected data, is it getting slower, and what should be inspected next?
Observability surfaces
Main API
Run history, tenant-scoped pipeline listings, CDC offsets, and schema history.
Metrics server
Prometheus scrape target, health, and readiness for metrics.
Grafana dashboards
Triage, baseline drift, throughput, resource pressure, and freshness views.
Core signals
pipeline_status
Latest run reports success.
It is 0, absent, or stale.
duration_sec
Stable for similar input volumes.
It grows faster than row count or exceeds the schedule window.
records_processed
Matches the expected source volume.
It drops unexpectedly or is much larger than expected.
cpu_percent and memory_mb
Resource use correlates with useful throughput.
Resource use rises while throughput falls.
Push age
Recent for active resources.
The series has not refreshed after a run or service heartbeat.
Triage flow
Start with the latest run status and run age.
If the run failed, open logs before changing config.
If the run succeeded but the dataset is wrong, compare source row counts, destination row counts, and write strategy.
If the run is slow, compare duration, throughput, extract time, load time, CPU, and memory.
If CDC is involved, check source retention, connector lag, offsets, and schema history.
Record the resource ID, tenant, owner, pipeline mode, and exact error before escalating.
Related docs
Last updated
Was this helpful?