For the complete documentation index, see llms.txt. This page is also available as Markdown.

Observability

Index of Nilus observability surfaces (logs, API, metrics, Grafana) and the core signals to check for pipeline health.

Nilus observability is the set of signals that helps operators answer four questions: did the pipeline run, did it move the expected data, is it getting slower, and what should be inspected next?

Observability surfaces

Surface
What it answers
Reference

Pipeline logs

Runtime errors, connector messages, stack traces, and final push status.

Main API

Run history, tenant-scoped pipeline listings, CDC offsets, and schema history.

Metrics server

Prometheus scrape target, health, and readiness for metrics.

Grafana dashboards

Triage, baseline drift, throughput, resource pressure, and freshness views.

Core signals

Signal
Healthy shape
Investigate when

pipeline_status

Latest run reports success.

It is 0, absent, or stale.

duration_sec

Stable for similar input volumes.

It grows faster than row count or exceeds the schedule window.

records_processed

Matches the expected source volume.

It drops unexpectedly or is much larger than expected.

cpu_percent and memory_mb

Resource use correlates with useful throughput.

Resource use rises while throughput falls.

Push age

Recent for active resources.

The series has not refreshed after a run or service heartbeat.

Triage flow

  1. Start with the latest run status and run age.

  2. If the run failed, open logs before changing config.

  3. If the run succeeded but the dataset is wrong, compare source row counts, destination row counts, and write strategy.

  4. If the run is slow, compare duration, throughput, extract time, load time, CPU, and memory.

  5. If CDC is involved, check source retention, connector lag, offsets, and schema history.

  6. Record the resource ID, tenant, owner, pipeline mode, and exact error before escalating.

Last updated

Was this helpful?