Zebra metrics

Zebra exposes counters, gauges, and timing distributions through a Prometheus HTTP exporter. Prometheus stores the samples; Grafana queries Prometheus to render dashboards. Metrics complement health checks: a successful scrape does not establish that the node is ready to serve clients.

Enable the exporter

Use a build with the prometheus feature.

If no configuration file exists, create one:

zebrad generate -o zebrad.toml

In zebrad.toml, set [metrics] endpoint_addr:

[metrics]
endpoint_addr = "127.0.0.1:9999"

Start Zebra with that configuration and retrieve the metrics:

zebrad -c zebrad.toml start

From another terminal:

curl --fail http://127.0.0.1:9999/metrics

The exporter provides no authentication or TLS configuration. Bind it to loopback when Prometheus runs on the same host. For remote scrapes, bind to a private interface and restrict access with a firewall or authenticated proxy. Failure to bind the configured address terminates startup.

For Prometheus running on the same host, use:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: zebra
    metrics_path: /metrics
    static_configs:
      - targets: ["127.0.0.1:9999"]

For container deployments, use an address reachable from Prometheus; the supplied Compose stack uses zebra:9999.

Configure Grafana’s Prometheus data source with the Prometheus server URL, not Zebra’s exporter URL. For a same-host deployment without containers, the usual Prometheus URL is http://127.0.0.1:9090. The Compose stack provisions its own data source automatically.

Metric names and types

Zebra’s source uses dot-separated names, such as sync.estimated_distance_to_tip. The exporter replaces dots with underscores, so Prometheus queries use sync_estimated_distance_to_tip. Counter names are not automatically given a _total suffix: use the exported name as written.

Counters accumulate during a process run; use rate() or increase() for activity over time. Gauges represent a current or most recently measured value. A metric can be absent until its recording path executes; absence is not equivalent to zero.

Zebra exports timing distributions as Prometheus summaries, with a quantile label and _sum and _count series, not _bucket series:

rpc_request_duration_seconds{quantile="0.99"}

Do not apply histogram_quantile() to these Zebra summaries or average their quantiles across nodes. For a mean duration over five minutes, divide the rate of _sum by the rate of _count. Metrics generated separately by Jaeger’s span-metrics connector have their own types and naming.

Principal operator metrics

The table uses source names; replace dots with underscores in Prometheus. Suggested alert conditions are operational choices, not Zebra defaults. Choose sustained thresholds that distinguish initial sync from steady-state operation and accommodate the network’s block production intervals.

Prometheus also generates up{job="zebra"} for scrape success. Alert on zero or missing scrape data separately from chain progress. This metric comes from Prometheus, not Zebra.

Network metric aggregation

Network metrics omit per-peer labels to bound the number of time series. Update dashboards, recording rules, and joins that depend on peer identities:

• Connection-state, request, response, message, and byte metrics have no addr label. The zebra.net.connection.state gauge counts live connections by command; transitions and disconnects decrement the old state. Request, response, and message counters retain command labels.
• The counters zcash.net.peers.connected, zcash.net.peers.obsolete, and zcash.net.peers.missing_services have no remote_ip or user_agent labels. They count handshake activity, not currently connected peers.
• The gauges zcash.net.peers.version.connected and zcash.net.peers.version.obsolete contain the remote protocol version from the most recent corresponding handshake, without remote_ip labels. They are not peer counts or a distribution of connected versions.
• The zebra.net.in.errors counter has no addr label.

The counter zcash.net.peer.version.outcomes.total provides a software breakdown through the bounded implementation values zebra, zakura, legacy_zcashd, and other. Select outcome="success" for successful handshakes. These remain cumulative handshake counts, not current peer counts.

Other bounded labels include:

Mining metrics

Prometheus replaces dots in metric names with underscores. The RPC dashboard includes mining outcomes and mean build and announcement durations. Zebra exports duration metrics as Prometheus summaries, with quantiles plus _sum and _count series.

Submission labels follow the zcashd-compatible result, with accepted representing JSON null. Operational JSON-RPC errors remain in rpc.errors.total; they are not consensus rejections. Some result variants are reserved by the response type and are not currently returned.

Build timing covers the shared constructor used by background and fallback builders, not state or mempool reads, executor scheduling, long-poll waits, or cache hits. It records each completed build, including work superseded before a miner requests it.

First-template latency uses the state’s monotonic publication time, not the block header timestamp or the time the RPC notices a change. It records once per tip transition across RPC clones; repeated requests do not add samples. Tips with no served template and templates on uncommitted candidates are excluded. Returning to an earlier tip after a reorg starts a new interval. At startup, timing begins when the state publishes its loaded tip. The measurement ends when the RPC returns the template, before network delivery or pool job distribution; it also includes time until a miner requests work.

Announcement completion means every selected connection completed its local write, not that peers received, validated, or relayed the block. Failed attempts include write errors and timeouts; a retry records another outcome. Successful retry durations retain the original gossip receipt time, including queueing, peer-readiness waits, and retry delays. Ordinary tip and early-relay adverts are excluded. No peer count is exported because the broadcast response does not carry one.

Watch increases in rejected submissions and failed announcements alongside rising announcement durations. A successful local write is not proof against stale blocks, and a stalled attempt has no completion sample yet. These metrics do not measure network-wide propagation or stale-block rates.

Observability stack

The Compose stack builds Zebra and starts Prometheus, Grafana, Jaeger, and Alertmanager using configuration in docker/observability/. It persists Zebra’s state, Prometheus, Grafana, and Alertmanager in volumes.

The supplied stack is a development deployment, not a hardened public service. Zebra’s RPC and metrics ports stay on the Compose network, and RPC requires cookie authentication. The remaining port mappings bind on all host interfaces; restrict published management ports to loopback or a private network before starting it. Prometheus’s admin and lifecycle APIs are disabled.

Export GF_SECURITY_ADMIN_PASSWORD with a strong, unique password; Compose rejects an unset or empty value. Run from the repository root:

docker compose -f docker/docker-compose.observability.yml up -d
docker compose -f docker/docker-compose.observability.yml logs -f zebra

Log in to Grafana as admin with the password supplied through GF_SECURITY_ADMIN_PASSWORD. It initializes new Grafana volumes only; change existing account passwords in Grafana. The stack provisions the Prometheus and Jaeger data sources and the dashboards in docker/observability/grafana/dashboards/. The home dashboard is zebra_overview.json; other dashboards include peers.json, network_health.json, syncer.json, mempool.json, rocksdb.json, rpc_metrics.json, rpc_tracing.json, and value_pools.json. For a separate Grafana deployment, import the JSON files and select its Prometheus data source.

The file docker/observability/prometheus/prometheus.yaml scrapes zebra:9999 and Jaeger’s span metrics at jaeger:8889, every 15 seconds. Zebra exports traces to http://jaeger:4318 with 100% sampling in this stack. Reduce [tracing] opentelemetry_sample_percent through the Compose environment when lower trace volume is required; see OpenTelemetry export.

Review docker/observability/prometheus/rules/zebra_alerts.yml before relying on its alerts. Its latency rules use exported summary quantiles: RPC p99 above two seconds and successful-handshake p95 above two seconds, each sustained for five minutes. Zebra abandons handshakes after three seconds, so slower ones never reach the handshake series. The RPC latency panels show per-method quantiles, not a combined percentile across methods or nodes. The receivers in docker/observability/alertmanager/alertmanager.yml have no notification destinations; configure them before expecting notifications.

Stop the stack without deleting its volumes:

docker compose -f docker/docker-compose.observability.yml down