Zebra
Zebra is an independent implementation of the Zcash protocol, written in Rust
by the Zcash Foundation. Its binary, zebrad, is a full
node: it downloads every block from the peer-to-peer network, checks it against
the consensus rules of the protocol
specification and the
ZIPs, and maintains the resulting chain state. The
original node, zcashd, has
reached end of life.
What Zebra does
- Validation. Blocks pass structural checks while parsing, semantic checks that need no chain context (proofs, signatures, scripts), and contextual checks against the chain they extend (double spends, value pools, difficulty). By default, blocks up to the last hard-coded checkpoint get only proof-of-work and Merkle-root checks and are accepted once their hashes chain to a checkpoint. Architecture describes the pipeline.
- Chain state. The finalized chain lives in RocksDB; the most recent blocks of every competing chain stay in memory, so Zebra can switch to a chain with more work.
- Network. Zebra speaks the Zcash peer-to-peer protocol and relays blocks and, once near the tip, unmined transactions from its mempool.
- Interfaces. A JSON-RPC server that follows the
zcashdAPI serves light-wallet servers, wallets, explorers, exchanges, and mining pools.
Zebra holds no keys. Wallets use Zallet; light clients use a
light-wallet server such as Zaino or lightwalletd, both
backed by Zebra. Systems that still depend on the zcashd wallet RPC can run
zcashd behind Zebra in zcashd-compat mode; the
API migration guide records which
component now owns each zcashd method.
Networks and network upgrades
Zebra runs on Mainnet, Testnet, Regtest, and custom test networks. Zebra 7.0.0 implements every network upgrade through NU7, at these compiled-in heights:
| Network upgrade | Mainnet height | Testnet height | Deployment ZIP |
|---|---|---|---|
| Overwinter | 347,500 | 207,500 | ZIP 201 |
| Sapling | 419,200 | 280,000 | ZIP 205 |
| Blossom | 653,600 | 584,000 | ZIP 206 |
| Heartwood | 903,000 | 903,800 | ZIP 250 |
| Canopy | 1,046,400 | 1,028,500 | ZIP 251 |
| NU5 | 1,687,104 | 1,842,420 | ZIP 252 |
| NU6 | 2,726,400 | 2,976,000 | ZIP 253 |
| NU6.1 | 3,146,400 | 3,536,500 | ZIP 255 |
| NU6.2 | 3,364,600 | 4,052,000 | ZIP 257 |
| NU6.3 (Ironwood) | 3,428,143 | 4,134,000 | ZIP 258 |
| NU7 | 3,543,405 | 4,465,026 | ZIP 259 |
A release that does not implement an upgrade cannot follow the chain past its activation height, and on Mainnet every release halts at its end-of-support height. Upgrading Zebra covers both deadlines.
Getting started
Run the latest release in Docker, with the P2P port published and the state on a named volume:
docker run -d --name zebra \
-p 8233:8233 \
-v zebrad-cache:/home/zebra/.cache/zebra \
zfnd/zebra:latest
Or install the pre-built binary for Linux on x86_64 or aarch64:
cargo binstall zebrad
zebrad start
Installing Zebra, Running Zebra, and Zebra with Docker cover the details.
Documentation
The user documentation is for operators and integrators, the developer documentation for contributors, and the API reference points to the crates’ rustdoc.
Support and security
Report bugs in the
issue tracker;
Troubleshooting lists known problems. Discussion
happens in the zebra-support channel of the
Zcash Foundation Discord. Report
vulnerabilities privately, as the
security policy
describes.
Zebra is dual-licensed under MIT and Apache 2.0; some crates are MIT-only,
as their Cargo.toml states.
User documentation
These pages describe Zebra 7.0.0 for node operators and for the software that depends on their nodes.
- Deploy: requirements, platforms, installation, configuration, running, Docker. A state snapshot avoids a sync from genesis.
- Operate: upgrading, tracing, metrics, health endpoints, troubleshooting.
- Integrate: JSON-RPC, Zallet,
lightwalletd, mining. Operators
leaving
zcashdstart with the API migration guide and zcashd-compat mode. - Test networks: custom testnets, Regtest, forking the Testnet.
System requirements
Size the host for initial synchronization and the services that share it, not only for an idle node at the chain tip. See platform support for operating-system coverage.
Resource use
| Resource | Consumers | Capacity planning |
|---|---|---|
| CPU | Proof and signature verification, transaction execution, database compaction, and RPC requests | Reserve capacity for synchronization and client traffic; see CPU worker configuration. |
| RAM | Blocks awaiting verification, recent non-finalized chains, the mempool, proof parameters, and RocksDB caches and write buffers | Account for all processes and container memory limits. Reducing synchronization concurrency reduces the number of in-flight blocks; the mempool cost limit is not a process-wide memory limit. |
| Disk | Blocks, transactions, state indexes, recent-state backups, and configured log files | Keep free space for chain growth, database compaction, and upgrades. Account separately for snapshots and other services. |
| Network | Initial block download, new blocks and transactions, peer discovery, and responses to peer or RPC requests | Allow sustained outbound connectivity. Upload demand depends on the peers and clients served by the node. |
The configuration reference documents these controls. Monitor metrics and host resource usage while synchronizing and under the expected RPC load.
Reference deployment
The Zcash Foundation’s
node deployment
and
integration-test
use Google Compute Engine c2d-standard-2 instances (2 vCPUs, 8 GB RAM) with a
400 GB balanced persistent state disk. This is not a minimum or an RPC-load
capacity guarantee.
Disk capacity
The published Mainnet state snapshot of 2026-10-23 reports a source-node height of 3,527,378 and an archive size of 273,763,413,895 bytes (255 GiB). The Testnet archive of the same day, at finalized height 4,531,081, is 12,221,184,309 bytes (11.4 GiB). These are zstd-compressed tar archive sizes, not measurements of the extracted databases. The snapshot index at https://snapshots.zfnd.org/ lists current archive sizes. The database grows with every block.
Store state on persistent storage configured with [state] cache_dir. Mainnet
and Testnet have separate databases. Running both requires space for both.
The zebra.state.rocksdb.total_disk_size_bytes metric measures the RocksDB
table files, not all files on the filesystem. Check filesystem free space as
well.
A state snapshot needs space for the downloaded archive and the extracted database at the same time. Retaining an existing database during replacement needs additional space. Do not use the compressed archive size as the required capacity of the destination disk.
zcashd-compat requirements
The zcashd-compat mode runs Zebra with a zcashd sidecar. Its Linux preflight checks enforce these thresholds; they are not standalone Zebra benchmarks.
| Resource | Minimum | Recommended |
|---|---|---|
| Logical CPUs available to the process | 4 | 8 |
| Effective RAM | 16 GiB | 32 GiB |
| Mainnet disk provisioning | 275 GiB for Zebra and 275 GiB for zcashd | 1 TiB combined |
| Testnet / Regtest disk provisioning | 30 GiB for Zebra and 30 GiB for zcashd | 100 GiB combined |
Effective RAM is the smaller of host RAM and the detected container memory
limit. The disk check measures total filesystem capacity, not free space.
Shared-filesystem minimums add: 550 GiB on Mainnet or 60 GiB on Testnet and
Regtest. Each filesystem counts once toward the combined recommendation.
Zebra checks the sidecar’s disk only when it manages the sidecar or
[zcashd_compat] zcashd_datadir identifies the external sidecar’s data directory.
A remote or otherwise unspecified sidecar needs separate capacity planning.
A failed minimum check stops startup. Values below the recommendation produce
warnings. The zebrad start --unsafe-low-specs option bypasses preflight
failures; it does not reduce resource use or make an undersized host safe.
Network requirements and ports
Zebra needs outbound TCP connections to Zcash peers to synchronize. Default peer discovery also needs working DNS resolution. Inbound peer connections are optional for synchronization; enable them to let other nodes connect to Zebra.
Configure the P2P listener and advertised address, then permit its port through the firewall and publish it when running in Docker. Do not expose management endpoints merely to make peer-to-peer networking reachable.
Platform support
The platform tier policy defines maintenance commitments; the tables below list Zebra 7.0.0 release targets and CI coverage. A Rust target triple does not specify a Linux distribution or library versions.
Minimum supported Rust version (MSRV)
The workspace and library crates require Rust 1.89. For the daemon’s minimum and pinned toolchain, see building from source. CI toolchains are listed below.
Tier 1
The production release targets are 64-bit GNU/Linux on x86 and ARM:
| Target | Release archive build host | Artifacts |
|---|---|---|
x86_64-unknown-linux-gnu | Ubuntu 22.04 | Binary archive and linux/amd64 image |
aarch64-unknown-linux-gnu | Ubuntu 22.04 ARM | Binary archive and linux/arm64 image |
See pre-built binary requirements and Docker images. The GNU/Linux archives do not run natively on Alpine Linux.
The release-binary workflow builds both targets natively. The Docker workflow builds test images only for AMD64; the main unit-test workflow has no per-change ARM64 job. See the Tier 1 policy.
Tier 2
The additional CI environment, x86-64 Ubuntu 26.04, runs workspace and crate builds, minimum-version checks, Clippy, and unit tests. It produces no separate Ubuntu-specific release archive.
| Target | Environment | Toolchain coverage |
|---|---|---|
x86_64-unknown-linux-gnu | GitHub Actions ubuntu-26.04 | Rust 1.97.0 builds and tests; Rust 1.91.0 daemon build; Rust 1.89 library-crate builds |
The target triple is the same as the x86-64 Tier 1 release target; the tier here describes an additional OS/toolchain environment. See the Tier 2 policy.
Scheduled or manually triggered advisory jobs run Clippy and unit tests with beta Rust on Ubuntu 26.04. They do not guarantee that each change builds with beta.
Tier 3
The following platforms have no official zebrad release archives and no
full-node build-and-test guarantee:
| Target | Platform | CI limitation |
|---|---|---|
x86_64-apple-darwin | macOS on Intel | No macOS build or test job |
aarch64-apple-darwin | macOS on Apple silicon | No macOS build or test job |
x86_64-pc-windows-msvc | Windows with the MSVC toolchain | CI tests only the zebra-test helper crate, not zebrad |
Build from source on macOS; use a supported Linux environment for production. The zcashd-compat preflight supports Linux only, independently of these native-node tiers. See the Tier 3 policy.
Platform tier policy
General
Zebra’s platform tiers define maintenance commitments, following the model of the Rust target tier policy. A platform includes its architecture, operating system, toolchain, and required native libraries. The platform support matrix lists the release targets and the CI coverage for Zebra 7.0.0.
The Zebra team approves additions, promotions, demotions, and removals. Each tier inherits the requirements of the tier below it. A platform must first be accepted at the lower tier and demonstrate sustained maintenance before promotion; the Zebra team determines the required period. Meeting a checklist does not automatically confer a tier.
The words must and must not express requirements. The words should and should not express requirements for which the Zebra team can approve an exception. The word may expresses permission. This terminology follows RFC 2119.
Tier 3 platform policy
Tier 3 offers no build, test, or release guarantee. Other contributors need not maintain or test its platform-specific support.
- A platform must have identified maintainers who can receive reports of platform-specific issues.
- Target names should identify the actual environment without ambiguity.
- Changes must not break Tier 1 or Tier 2 platforms. A known regression on another Tier 3 platform requires agreement from that platform’s maintainers or the Zebra team.
- Tier 3 failures must not block unrelated pull requests. Automated failure notifications must go only to people who have opted into them.
- Support must not impose a disproportionate maintenance burden on the rest of Zebra.
The Zebra team may remove an inactive or unmaintained Tier 3 platform, or one whose support impedes the codebase. A removal proposal must notify the platform maintainers so they can respond.
Tier 2 platform policy
Tier 2 requires reliable builds in CI. A platform-specific build failure in a required job blocks acceptance of the affected change. Tier 2 does not promise that every test passes or that the project publishes binaries for that platform.
The DevOps team must approve the platform’s CI integration.
- The platform must implement Zcash consensus rules. The Zebra team may approve excluding other features or development tools.
- At least one designated developer must maintain the platform and respond to platform-specific build failures.
- The platform must be a Rust Tier 1 target and have a usable Rust toolchain.
- Documentation must specify CPU and OS baselines, native dependencies, build commands, and test commands. It must identify required hardware or emulation when an ordinary development host cannot run the target.
- Required CI builds must be reliable. Their duration and infrastructure cost must not impose an excessive burden on the project.
- Platform maintainers should run the test suite regularly and fix failures. The Zebra team may require selected tests as part of the platform’s CI contract.
Contributors need not become platform experts. The Zebra team handles test failures according to their impact: it may require a fix, disable a test, or demote the platform.
Tier 1 platform policy
Tier 1 adds production support and release artifacts to the Tier 2 requirements. The Zebra team, release team, and DevOps team must approve a promotion and the infrastructure needed to sustain it.
- The platform must support the standard production feature set: peer-to-peer networking, consensus verification, the mempool, persistent state, and RPC services. The Zebra team must approve exceptions.
- The platform must provide adequate security, performance, and reliability for production use, and serve users across multiple organizations or projects.
- Required builds and tests must pass reliably. The platform must not achieve this by disabling an excessive part of the test suite.
- Building and running tests must not take substantially longer than on other supported platforms.
- Release artifacts must have a documented runtime baseline and a maintained build and publication path.
- Maintainers must provide any additional hardware, cloud capacity, or emulation needed for CI and platform-specific debugging. CI resources must be continuously available for the project’s exclusive use; debugging resources must be exclusive while in use.
- Emulation used for required checks must accurately represent native behavior. Discrepancies affecting test results must be treated as high-priority platform, compiler, or emulator defects.
The Zebra team decides whether a failure requires a fix before other changes merge, a fix before release, a test exception, or a tier change. Changes to a Tier 1 CPU or OS baseline require team approval and communication to users.
Tier 1 or Tier 2 demotion or removal proposals must notify maintainers and the Zcash community before a stable release. Notice depends on severity and urgency. Changes normally apply to development and future releases, not retroactively.
Building and installing Zebra
Install a release archive on supported Linux systems, use the Docker image, or build from source. Check system requirements and platform support before provisioning the node.
Pre-built binaries
The GitHub release provides these archives:
| Architecture | Archive |
|---|---|
| 64-bit x86 | zebrad-7.0.0-x86_64-unknown-linux-gnu.tar.gz |
| 64-bit ARM | zebrad-7.0.0-aarch64-unknown-linux-gnu.tar.gz |
Both require glibc 2.34 or newer and the C++ runtime library, libstdc++.
Documented runtime baselines include Ubuntu 22.04, Debian 12, RHEL 9, and
Amazon Linux 2023, or newer releases. RocksDB and the Zcash C++ script library
are linked statically. These are GNU/Linux binaries, not macOS, Windows, or
musl binaries.
Each archive contains zebrad, both license files, and a README. The release
also supplies a checksum sidecar for each archive, a combined SHA256SUMS,
and its Cosign signature bundle, SHA256SUMS.sigstore.json. GitHub stores a
build-provenance attestation for each archive.
Install with cargo-binstall
After installing cargo-binstall, install the latest published Zebra release:
cargo binstall zebrad
To select this release explicitly:
cargo binstall --version 7.0.0 zebrad
The package metadata does not configure verification of Zebra’s signed checksum manifest. Use the manual procedure below when that verification is required.
Download and verify an archive
Install GitHub CLI and
Cosign.
The following Linux procedure also uses GNU sha256sum, tar, and install.
Select aarch64-unknown-linux-gnu instead of x86_64-unknown-linux-gnu on ARM64.
Run the commands in a new directory so unrelated files do not affect checksum
verification.
set -eu
version=7.0.0
target=x86_64-unknown-linux-gnu
archive="zebrad-${version}-${target}.tar.gz"
mkdir "zebrad-${version}-${target}"
cd "zebrad-${version}-${target}"
gh release download "v${version}" --repo ZcashFoundation/zebra \
--pattern "${archive}" \
--pattern SHA256SUMS \
--pattern SHA256SUMS.sigstore.json
gh attestation verify "${archive}" \
--repo ZcashFoundation/zebra \
--signer-workflow ZcashFoundation/zebra/.github/workflows/zfnd-release-binaries.yml \
--source-ref "refs/tags/v${version}"
cosign verify-blob SHA256SUMS \
--bundle SHA256SUMS.sigstore.json \
--certificate-identity="https://github.com/ZcashFoundation/zebra/.github/workflows/zfnd-release-binaries.yml@refs/tags/v${version}" \
--certificate-oidc-issuer='https://token.actions.githubusercontent.com'
sha256sum --ignore-missing --check SHA256SUMS
tar -xzf "${archive}"
install -d "${HOME}/.local/bin"
install -m 755 zebrad "${HOME}/.local/bin/zebrad"
export PATH="${HOME}/.local/bin:${PATH}"
zebrad --version
Stop if any verification fails. The attestation binds the archive to the expected repository, builder workflow, and release tag. Cosign authenticates the checksum manifest; the checksum check then compares the downloaded archive with that signed manifest. An unsigned checksum alone detects file corruption but does not authenticate the publisher.
An attestation is not an independent rebuild.
Keep ${HOME}/.local/bin in the service or shell’s PATH, or invoke the
installed binary by its full path. Continue with running Zebra.
Building from source
The zebrad binary requires Rust 1.91 or newer; Zebra 7.0.0 pins Rust 1.91.0
in rust-toolchain.toml. Install rustup,
then select the release toolchain:
rustup toolchain install 1.91.0
rustup default 1.91.0
The bundled RocksDB and Zcash script library require a C compiler, a C++17
compiler, and Clang’s shared library, libclang, for generated Rust bindings.
Install Git for the tagged-source procedure below.
The RPC build script
uses checked-in generated files unless the Protocol Buffers compiler, protoc,
and source .proto files are available. Install protoc only to regenerate
these files or build with the lightwalletd-grpc-tests feature:
| Platform | Package |
|---|---|
| Debian / Ubuntu | protobuf-compiler |
| Arch Linux / Homebrew | protobuf |
Debian and Ubuntu
sudo apt-get update
sudo apt-get install build-essential clang libclang-dev git
Arch Linux
sudo pacman -Syu --needed base-devel clang git
Fedora and RHEL
On Fedora:
sudo dnf install gcc gcc-c++ make clang clang-libs clang-devel git
On RHEL 9, install the LLVM toolset and libclang runtime from the enabled RHEL repositories:
sudo dnf install gcc gcc-c++ make llvm-toolset clang-libs git
macOS
Install the Xcode Command Line Tools for the compiler and SDK, and Homebrew LLVM for libclang:
xcode-select --install
brew install llvm
export LIBCLANG_PATH="$(brew --prefix llvm)/lib"
Complete the Command Line Tools installation before building.
Install from crates.io
Build and install the latest published release with its dependency lockfile:
cargo install --locked zebrad
To pin Zebra 7.0.0:
cargo install --locked --version 7.0.0 zebrad
Cargo installs the executable in its installation root, normally
${HOME}/.cargo/bin. Keep that directory in PATH. The --locked option
retains the release’s dependency versions; do not remove it as a generic
workaround for a build failure.
Install from a release tag
Build and install directly from the Git tag:
cargo install --locked \
--git https://github.com/ZcashFoundation/zebra \
--tag v7.0.0 zebrad
Alternatively, build a checkout without installing it:
git clone --branch v7.0.0 https://github.com/ZcashFoundation/zebra.git
cd zebra
cargo build --locked --release --bin zebrad
target/release/zebrad --version
Run the installed binary with zebrad start, or the checkout build with
target/release/zebrad start. See running Zebra for configuration,
network selection, and shutdown.
Dynamic RocksDB linking
The default build statically links bundled RocksDB. To avoid recompiling it, link an installed shared library instead.
Zebra 7.0.0 locks rocksdb to 0.24.0 and librocksdb-sys to
0.17.3+10.4.2, which bundles RocksDB 10.4.2. Use matching 10.4.2 headers
and a shared library built with LZ4 support. Do not assume that an arbitrary
system package supplies the required API; the build script does not validate
the library’s version.
For a matching installation under /opt/rocksdb-10.4.2, with libraries in
lib and headers in include:
unset ROCKSDB_STATIC ROCKSDB_COMPILE
export ROCKSDB_LIB_DIR=/opt/rocksdb-10.4.2/lib
export ROCKSDB_INCLUDE_DIR=/opt/rocksdb-10.4.2/include
cargo build --locked --release --bin zebrad
The librocksdb-sys build script
uses ROCKSDB_LIB_DIR to skip the bundled C++ build. It selects dynamic
linking when ROCKSDB_STATIC is absent; even 0 selects static linking.
The ROCKSDB_INCLUDE_DIR variable selects bindgen’s headers; libclang is
still required. Set both paths to the actual installation.
Configure the operating system’s dynamic-library loader to find RocksDB and
its dependencies when Zebra starts. The resulting binary depends on those
libraries remaining installed. Unset ROCKSDB_LIB_DIR and
ROCKSDB_INCLUDE_DIR to return to the bundled build.
Building on ARM
Use a native 64-bit ARM Rust toolchain and native C/C++ dependencies:
aarch64-unknown-linux-gnu for Linux or aarch64-apple-darwin for Apple
silicon. Do not select an x86-64 toolchain. There is no official 32-bit ARM
archive.
Build troubleshooting
| Failure | Action |
|---|---|
| Cargo rejects the Rust version | Select Rust 1.91.0 or newer. Check for a RUSTUP_TOOLCHAIN override or a different cargo earlier in PATH. |
Bindgen cannot find libclang.so or libclang.dylib | Install the Clang shared library and set LIBCLANG_PATH to its containing directory if automatic discovery fails. A compiler executable alone is not sufficient. |
| The loader cannot open a versioned libclang library | Install the matching runtime package. Pointing at headers or a different library version does not repair a missing runtime dependency. |
| The C/C++ compiler or linker is missing | Install the native build-tool packages above. Select a compiler explicitly with CC and CXX when necessary. |
The build cannot find protoc | Install it when using lightwalletd-grpc-tests. Remove an invalid PROTOC override; a normal RPC build falls back to generated files only when no compiler is selected or found. |
| Dynamic RocksDB linking reports missing symbols or Zebra cannot load RocksDB | Check that the headers, shared library, and loader paths refer to the same compatible installation, or return to the bundled build. |
| Compilation exhausts memory | Reduce Cargo’s parallel jobs, for example cargo build --locked --release --bin zebrad --jobs 1. This changes build concurrency, not node runtime requirements. |
Tor feature
Zebra 7.0.0 has no buildable zebra-network/tor feature; selecting it fails
Cargo feature resolution. See Arti support.
Cargo features
Cargo features select code at build time; configuration selects runtime
behavior. The table covers every explicit feature in
zebrad/Cargo.toml.
“Default” includes features enabled transitively by default.
Mining RPCs require no optional Cargo feature; internal-miner adds the
built-in solver, not the mining API.
| Feature | Default | Purpose | Runtime prerequisites or configuration |
|---|---|---|---|
default | Yes | Production feature selector; enables default-release-binaries. | Each included component retains its own runtime activation requirements. |
default-release-binaries | Yes | Production bundle: release_max_level_info, progress-bar, prometheus, sentry, and opentelemetry. | The archive workflow explicitly selects this bundle. |
indexer | No | Production integration: adds state indexes mapping spent outputs and revealed nullifiers to spending transactions. | No runtime switch; opening a populated state adds the indexes. A build without the feature removes them. The indexer gRPC listener is separate and uses [rpc] indexer_listen_addr without requiring this feature. |
internal-miner | No | Experimental CPU Equihash miner; runs one solver and checks the target difficulty. | Set [mining] miner_address and internal_miner = true; see internal miner setup. Use generation RPCs, not this solver, for deterministic Regtest block production. |
elasticsearch | No | Experimental integration: exports finalized block data to Elasticsearch. | Requires an Elasticsearch service and [state] elasticsearch_url, elasticsearch_username, and elasticsearch_password; see Elasticsearch. |
zmq | No | Pool integration: publishes zcashd-compatible ZMQ hashblock notifications, using a pure-Rust implementation. | Set [notify] zmqpubhashblock; without the feature, configuring it is a startup error. See ZMQ block notifications. |
sentry | Yes | Production monitoring: sends diagnostic events to Sentry. | Set SENTRY_DSN to activate; see Sentry monitoring. |
journald | No | Production logging: writes tracing events to systemd-journald. | Requires Linux with a reachable journal and [tracing] use_journald = true. |
filter-reload | No | Debugging and operations: changes tracing filters through HTTP. | Set [tracing] endpoint_addr. The endpoint is unauthenticated; restrict it to trusted access. It cannot restore logs removed at compile time. |
opentelemetry | Yes | Production observability: exports traces using OpenTelemetry. | Set [tracing] opentelemetry_endpoint or OTEL_EXPORTER_OTLP_ENDPOINT to an OTLP collector; see trace export. |
progress-bar | Yes | Operator display: renders synchronization, mempool, and network progress. | Set [tracing] progress_bar to "summary" or "detailed". Zebra redirects logs to a file when progress display is active. |
prometheus | Yes | Production monitoring: exposes Prometheus metrics. | Set [metrics] endpoint_addr; see metrics. No listener is enabled by default. |
error-debug | No | Debugging: enables error caller tracking and span-trace capture through color-eyre. | No Zebra configuration switch; error reporting uses the configured tracing context. |
release_max_level_warn | No | Production log-volume control: removes tracing levels more verbose than warn in release builds. | Runtime filters can select only levels retained by the build. |
release_max_level_info | Yes | Production logging: removes debug and trace calls from both tracing and log in release builds. | Runtime filters cannot enable debug or trace output in the default release binary. |
release_max_level_debug | No | Debugging: retains tracing through debug, but removes trace, in release builds. | Disable the default info ceiling before selecting it, then configure [tracing] filter. |
max_level_warn | No | Production log-volume control: removes tracing levels more verbose than warn in every build profile. | No runtime switch can raise this ceiling. |
max_level_info | No | Production log-volume control: removes tracing levels more verbose than info in every build profile. | No runtime switch can raise this ceiling. |
max_level_debug | No | Debugging: removes trace calls from both tracing and log in every build profile. | Disable lower default ceilings when debug output is needed. |
flamegraph | No | Debugging: records tracing spans for flamegraph output. | Set [tracing] flamegraph to the output path; see tracing. |
proptest-impl | No | Tests: enables property-test data generators in Zebra crates. | Used by test code; no node configuration switch. |
zebra-checkpoints | No | Tests: enables checkpoint-generation test support and the zebra-utils/zebra-checkpoints feature. | Checkpoint tests need a synced cached state and the zebra-checkpoints executable; see checkpoint generation. |
lightwalletd-grpc-tests | No | Tests: generates the lightwalletd gRPC test client. | Requires protoc while building and an installed lightwalletd binary for the tests. It is not required to serve normal wallet traffic. |
tokio-console | No | Debugging: enables Tokio task instrumentation through console-subscriber. | Build with RUSTFLAGS="--cfg tokio_unstable", disable default log ceilings, and use the tokio-console client; see setup. |
comparison-interpreter | No | Tests: runs the Rust and C++ script interpreters and reports disagreements. | No runtime activation flag; do not use for production. |
Cargo also exposes implicit features for optional dependencies. The default
bundle enables howudoin, indicatif, and metrics-exporter-prometheus.
The remaining dependency features are console-subscriber, inferno,
proptest, proptest-derive, thread-priority, tonic-prost-build,
tracing-flame, tracing-journald, and zebra-utils. Selecting a dependency
alone does not activate the corresponding Zebra feature; use the functional
features in the table.
Add features without removing the default bundle:
cargo install --locked --version 7.0.0 --features "indexer,journald" zebrad
Features are additive. To retain debug logging in a release build, remove the default info ceiling and select the desired production components explicitly:
cargo install --locked --version 7.0.0 --no-default-features \
--features "progress-bar,prometheus,sentry,opentelemetry,release_max_level_debug" \
zebrad
Set [tracing] filter to request the retained debug events. Avoid
--all-features for an operator build: it includes test features and
incompatible logging/instrumentation choices.
Configuring Zebra
Zebra 7.0.0 reads configuration from a TOML file and environment variables.
The file represents the
ZebradConfig Rust type.
Keep only settings that must differ from the defaults; copying every generated
value into a permanent configuration prevents later releases from updating those
values automatically.
Without -c or --config, Zebra looks for zebrad.toml in the user’s preference
directory:
| Platform | Default configuration file |
|---|---|
| Linux | $XDG_CONFIG_HOME/zebrad.toml, or $HOME/.config/zebrad.toml |
| macOS | $HOME/Library/Preferences/zebrad.toml |
| Windows | %LOCALAPPDATA%\zebrad.toml, in the local application-data directory |
Zebra uses defaults and environment variables when the default file is absent. Starting a node with an explicitly selected file requires that file to exist and be readable. Select a file with:
zebrad --config /path/to/zebrad.toml start
Generate a default configuration with:
zebrad generate --output-file zebrad.toml
The command overwrites the output file. Without --output-file, it writes to
standard output. It generates built-in defaults, not the merged configuration
of a running node. Optional fields whose default is absent are not included.
The generated file reflects the features compiled into that binary. The
generate command ignores configuration-loading errors so it can produce a
replacement for an invalid configuration.
The loader applies these sources, from highest to lowest priority:
- Environment variables with the
ZEBRA_prefix. - The selected TOML file.
- Built-in defaults for omitted fields and sections.
Command-specific overrides apply afterward. See Tracing for filter syntax and Docker for container configuration.
Every top-level section listed below rejects unknown keys, and the root rejects
unknown sections. The nested network-parameter, activation-height,
funding-stream, recipient, and lockbox-disbursement structs also reject unknown
keys. The Regtest selector wrapper described below is an exception: it does not
reject unknown sibling keys beside params and regtest. A field compiled out
of the schema, such as an Elasticsearch state field without the elasticsearch
feature, is an unknown field. Fields whose implementation is feature-gated can
still be accepted; their behavior is specified below. Unknown fields and invalid
types fail configuration loading rather than silently using defaults.
In the tables, absent means None: omit the key to leave it unset. TOML has
no null; an empty string, false, or empty array is a value, not an unset
instruction. Optional fields can have nonempty defaults. Removing a file
setting does not cancel an environment override.
Write durations as quoted humantime strings,
such as "5m", "1h", or "1m 1s". Address fields are strings containing an IP
address and port, with brackets around IPv6 addresses. Only the two peer-to-peer
(P2P) address fields permit an omitted port. A block height is an integer;
validated consensus heights are at most 2147483647. Amounts are in zatoshis,
with 100000000 zatoshis per ZEC.
The default cache directory used below is:
| Platform | Default cache directory |
|---|---|
| Linux | $XDG_CACHE_HOME/zebra, or $HOME/.cache/zebra |
| macOS | $HOME/Library/Caches/zebra |
| Windows | %LOCALAPPDATA%\zebra |
| No platform cache directory available | cache/zebra under the current working directory |
On Linux, the XDG directory variables must contain absolute paths. Empty or relative values use the corresponding home-directory fallback.
The P2P cache, state database, and RPC cookie have separate settings even when their defaults share this directory. Give each concurrently running node a separate state directory and RPC cookie directory. Protect configuration files containing credentials and directories writable by a privileged Zebra process from modification by other users.
Environment variables
- Zebra strips the exact, case-sensitive
ZEBRA_prefix, lowercases the key, and replaces each__with a nested-table separator. Single underscores remain in field names. - Use leaf variables or TOML, not JSON or TOML objects as whole-table values.
Configure activation heights in TOML: lowercasing loses case-sensitive
names such as
CanopyandNU7. - Values are parsed as case-insensitive booleans, signed 64-bit integers, floating-point numbers, then strings. Empty values are not ignored. Shell-quote spaces in durations and filters; shell quotes are not part of the value.
- There is no list separator or general array parser. Put peer arrays, network
magic, checkpoints, and funding streams in TOML, not comma-separated or
JSON-array environment values. Only the two JSON-array-string fields in
[zcashd_compat]accept arrays through environment values. - Zebra rejects leaf keys ending in
password,secret,token,cookie, orprivate_key, ignoring case, even for unknown fields. Store these values in the file. This rejectsZEBRA_STATE__ELASTICSEARCH_PASSWORD, but not path fields such ascookie_dir. - For
copy-state, useZEBRA_TARGET_variables with--target-config-pathto override target settings, for exampleZEBRA_TARGET_STATE__CACHE_DIR=/srv/zebra-target. The source/base loader ignores this prefix. Only the target’s[state]settings are used, although the whole file is deserialized. Without--target-config-path, the target uses an ephemeral state configuration and ignores target overrides.
| Environment variable | TOML setting |
|---|---|
ZEBRA_NETWORK__NETWORK=Testnet | [network] network = "Testnet" |
ZEBRA_RPC__LISTEN_ADDR=127.0.0.1:8232 | [rpc] listen_addr = "127.0.0.1:8232" |
ZEBRA_STATE__CACHE_DIR=/srv/zebra | [state] cache_dir = "/srv/zebra" |
ZEBRA_HEALTH__READY_MAX_TIP_AGE=10m | [health] ready_max_tip_age = "10m" |
ZEBRA_NETWORK__TESTNET_PARAMETERS__CHECKPOINTS=true | [network.testnet_parameters] checkpoints = true |
export ZEBRA_ZCASHD_COMPAT__ZCASHD_EXTRA_ARGS='["-debug=1"]'
export ZEBRA_ZCASHD_COMPAT__BLOCK_GOSSIP_PEER_IPS='["127.0.0.1","::1"]'
[consensus]
Rust API: zebra_consensus::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
checkpoint_sync | Boolean | true | Use the selected network’s full checkpoint list during synchronization. Setting false starts full verification earlier, but does not disable mandatory checkpoints. Developer-only option; disabling it makes synchronization slower and weakens protection against chain attacks. |
debug_skip_parameter_preload | Boolean | false | Obsolete compatibility key. Zebra accepts it, ignores its value, and omits it when generating configuration. |
[network]
Rust API: zebra_network::Config
and CacheDir.
| Field | Type | Default | Meaning |
|---|---|---|---|
listen_addr | IP address string, optional port | "[::]", resolved to "[::]:8233" on Mainnet | P2P listener. An omitted port becomes 8233 on Mainnet or 18233 on Testnet and Regtest. The generated Mainnet configuration includes the explicit port. |
external_addr | Optional IP address string, optional port | Absent | Address advertised to peers instead of the listener address, for network address translation (NAT), containers, or load balancers. Does not change the bind address. Uses the selected network’s default port if omitted. |
network | Network-name string or parameter table | "Mainnet" | Select "Mainnet", "Testnet", or "Regtest", or use a configured-network table as described below. |
experimental_early_block_relay | Boolean | false | Experimental next-tip block relay with peers that also enable it. Negotiated peers can serve authenticated blocks before full transaction validation; Zebra still fully verifies every block, and ordinary relay of committed blocks is unchanged. Uses a temporary service bit until a production one is allocated. |
initial_mainnet_peers | Array of address strings | Mainnet seed list below | Initial Mainnet peers; each IP address or DNS name must include a port. Duplicate strings are removed while preserving order. |
initial_testnet_peers | Array of address strings | Testnet seed list below | Initial Testnet peers. Regtest removes public seed names and retains only resolved loopback addresses. |
cache_dir | Boolean or path string | true | Use the default cache directory when true, disable the peer disk cache when false, or use the supplied directory. Independent of [state] cache_dir. |
peerset_initial_target_size | usize | 25 | Initial peer target. Also sets the outbound connection limit to three times this value and the inbound limit to five times it. Zero produces a warning and becomes 25. |
crawl_new_peer_interval | Duration string | "1m 1s" | Interval between routine peer-address crawls. Zebra also crawls when all peers are busy and no connection candidates remain. The old key new_peer_interval is an accepted alias. |
max_connections_per_ip | usize | 1 | Concurrent peer-connection limit per IPv4 address or IPv6 /64 subnet. Also affects inbound reconnection rate limiting. Zero produces a warning and becomes 1; increasing it reduces peer diversity protections. |
testnet_parameters | Optional table | Absent | Compatibility layout for Testnet or Regtest parameters, used with a string-valued network selector. Ignored for Mainnet and when network itself contains parameters. |
The exact default Mainnet seed list is:
initial_mainnet_peers = [
"dnsseed.str4d.xyz:8233",
"dnsseed.z.cash:8233",
"mainnet.seeder.shieldedinfra.net:8233",
"mainnet.seeder.zfnd.org:8233",
"seeder.zec.rocks:8233",
]
The exact default Testnet seed list is:
initial_testnet_peers = [
"dnsseed.testnet.z.cash:18233",
"seeder.testnet.zec.rocks:18233",
"testnet.seeder.zfnd.org:18233",
]
Place either array in [network]. An empty array removes those seeds; it does
not disable inbound connections or the disk peer cache. Regtest skips loading
the peer cache. Peer cache files use
<cache_dir>/network/<network-directory>.peers.
The listener and external address parsers accept IP literals, not DNS names. The default unspecified listener accepts connections without advertising an unspecified address.
Configured network layouts
Use Custom Testnets or Regtest for complete network examples. Both of these layouts select the same parameter schema:
[network]
network = "Testnet"
[network.testnet_parameters]
checkpoints = true
[network.network]
checkpoints = true
A table-valued selector with a params child selects Regtest instead:
[network.network.params.activation_heights]
NU5 = 1
The Regtest selector wrapper has these fields:
Field under [network.network] | Type | Default | Meaning |
|---|---|---|---|
params | Parameter table | Required for the Regtest table form | Selects Regtest and contains its parameters. Omit the entire table to use the string selector instead. |
regtest | Optional boolean | Absent | Accepted marker in the Regtest table form; the deserializer ignores its value. The presence of params, not this boolean, selects Regtest. |
[network.testnet_parameters]
These fields also apply under [network.network] for configured Testnet or
[network.network.params] for Regtest.
| Field | Type | Default | Meaning |
|---|---|---|---|
network_name | Optional string | Absent; configured name "UnknownTestnet" | Custom Testnet name, at most 30 UTF-8 bytes, containing only alphanumeric characters or _. Case-insensitive reserved names are Mainnet, Testnet, Regtest, MainnetKind, TestnetKind, and RegtestKind. Unchanged public parameters can retain the canonical name Testnet. Ignored on Regtest, whose name is Regtest. |
network_magic | Optional array of four u8 integers | Absent; [250, 26, 249, 191] on Testnet | P2P wire identifier. Mainnet [36, 233, 39, 100] and Regtest [170, 232, 63, 95] magic values are reserved. On Regtest, only [170, 232, 63, 95] is accepted. |
slow_start_interval | Optional block height | Absent; 20000 on Testnet | Subsidy slow-start interval in blocks; the slow-start shift is half this value. On Regtest, only 0 is accepted. |
target_difficulty_limit | Optional hexadecimal unsigned 256-bit integer string | Absent; compact-rounded 2^251 - 1 on Testnet | Maximum expanded proof-of-work target, not a fixed target for every block. Compact conversion can discard low-order precision; invalid or zero values fail startup. On Regtest, only the compact-rounded 256-bit value consisting of 32 0x0f bytes is accepted. |
disable_pow | Optional boolean | Absent; false on Testnet | Disable proof-of-work checks for an isolated test network. On Regtest, only true is accepted. |
genesis_hash | Optional 64-digit hexadecimal string | Absent; 05a60a92d99d85997cce3b87616c089f6124d7342af37106edc76126334a2c38 | Testnet genesis block hash. On Regtest, only 029f11d80ef9765602235e1bc9727e3eb6ba20839319f761fee920d63401e327 is accepted. |
activation_heights | Optional table | Absent; public Testnet schedule below | Replace the activation schedule, not individual public-network defaults. Without overrides, Regtest activates through Canopy at height 1; later upgrades remain inactive. |
nsm_reissuance_height | Optional block height | Absent | Override the calculated first Network Sustainability Mechanism (NSM) reissuance height. Must be nonzero and within the height range. Without an override, Zebra calculates the ZIP 237 scheduled-issuance crossover; a custom schedule can have no crossover. |
pre_nu6_funding_streams | Optional funding-stream table | Absent | Legacy funding-stream group prepended before the current list. Missing group fields inherit the first public Testnet group. |
post_nu6_funding_streams | Optional funding-stream table | Absent | Legacy group inserted after the pre-NU6 legacy group and before the current list. Missing group fields inherit the second public Testnet group. |
funding_streams | Optional array of funding-stream tables | Absent; public Testnet groups below, or [] on Regtest | Replace the funding-stream list. An explicit [] disables streams. |
pre_blossom_halving_interval | Optional u32 | Absent; 840000 on Testnet | Pre-Blossom halving interval in blocks, in 1..=2147483647. Later intervals are derived. On Regtest, only 144 is accepted, giving 288 after Blossom and 864 after NU7. |
lockbox_disbursements | Optional array of disbursement tables | Absent; ten public Testnet outputs below, or [] on Regtest | Expected one-time lockbox outputs at NU6.1 activation. An explicit [] removes them. |
checkpoints | Boolean, path string, or array of [height, hash] pairs | false in a parameter table | With false, use only the configured genesis checkpoint; with true, use the public Testnet checkpoint list. Strings are file paths, not a "default" selector. An array supplies the list directly. Plain network = "Testnet" without a parameter table uses the full public list instead. Regtest defaults to genesis only. |
extend_funding_stream_addresses_as_required | Optional boolean | Absent; no extension | When true, repeat configured funding-stream address sequences to cover their required address periods. |
temporary_orchard_disabling_soft_fork_height | Optional block height | Absent; 4048500 on Testnet | Override the start height of the temporary Orchard-action restriction, which ends at NU6.2 activation. Omission does not disable it on Testnet, and there is no TOML disabling switch. Rejected on Regtest, where the restriction is disabled. |
should_allow_unshielded_coinbase_spends | Optional boolean | Absent; true on Regtest | Regtest-only permission for transactions spending coinbase outputs to have transparent outputs. Zebra rejects this field on configured Testnet, even when set to false. |
initial_nsm_value_balance | Optional integer amount | Absent; 55768414957 zatoshis with public Testnet magic, otherwise 0 | NSM reserve seeded at NU7. Explicit values must be between 0 and 2100000000000000 zatoshis. Regtest defaults to 0. |
See Custom Testnets for cross-field validation and activation, subsidy, funding-stream, and checkpoint rules.
For inline checkpoint pairs, each hash is an array of 32 u8 values in
serialized byte order: reverse the byte order of the displayed hexadecimal
hash, not its individual digits. A hexadecimal string is not accepted here.
For human-readable hashes, use a checkpoint file: each line contains a height
and its displayed hexadecimal hash, separated by one space.
Activation heights
The activation table is
[network.testnet_parameters.activation_heights] in the compatibility layout.
All fields are optional u32 heights and default to absent in an explicitly
supplied table. The public schedule column applies only when the whole table is
omitted on Testnet.
| Field | Type | Default in supplied table | Public Testnet height |
|---|---|---|---|
BeforeOverwinter | Optional height | Absent | 1 |
Overwinter | Optional height | Absent | 207500 |
Sapling | Optional height | Absent | 280000 |
Blossom | Optional height | Absent | 584000 |
Heartwood | Optional height | Absent | 903800 |
Canopy | Optional height | Absent | 1028500 |
NU5 | Optional height | Absent | 1842420 |
NU6 | Optional height | Absent | 2976000 |
"NU6.1" | Optional height | Absent | 3536500 |
"NU6.2" | Optional height | Absent | 4052000 |
"NU6.3" | Optional height | Absent | 4134000 |
NU7 | Optional height | Absent | 4465026 |
ZFuture | Optional height | Absent | Not activated; key exists only with --cfg zcash_unstable="zfuture" |
Quote upgrade names containing dots so TOML does not interpret them as nested
keys. Each supplied height must be in 1..=2147483647.
Funding streams and lockbox outputs
Each funding_streams array element or legacy table configures one group:
| Field | Type | Default | Meaning |
|---|---|---|---|
height_range | Optional table with start and end | Absent; inherit the corresponding public Testnet group | Inclusive start, exclusive end of the group’s block heights. Entries beyond the three public default groups must provide this field. |
recipients | Optional array of recipient tables | Absent; inherit the corresponding public Testnet group | Replace the group’s recipients. An explicit [] leaves the group without recipients. Entries beyond the three public default groups must provide this field. |
Each supplied height_range has these required fields:
| Field | Type | Default | Meaning |
|---|---|---|---|
start | Block height | Required | First height in the group. Must not exceed end. |
end | Block height | Required | First height after the group; equal start and end produce an empty range. |
Each supplied recipient has these fields:
| Field | Type | Default | Meaning |
|---|---|---|---|
receiver | String enum | Required | One of "ECC", "ZcashFoundation", "MajorGrants", or "Deferred". |
numerator | u64 | Required | Share of the block subsidy over a fixed denominator of 100. |
addresses | Optional array of strings | Absent; empty address sequence | Transparent funding addresses in period order. Deferred recipients do not require addresses. Non-deferred recipients use P2PKH or P2SH addresses, not tex addresses. |
On Testnet and Regtest, Zebra rejects invalid ranges, missing defaults,
numerator totals above 100, and malformed addresses during loading.
Non-deferred recipients need Testnet addresses covering every address period.
Extension can repeat a nonempty list but cannot fill a required empty list.
The three public Testnet groups are, in order:
| Height range, end exclusive | Receiver shares | Address defaults |
|---|---|---|
1028500..2796000 | ECC 7/100, Zcash Foundation 5/100, Major Grants 8/100 | The 51-entry FUNDING_STREAM_ECC_ADDRESSES, FUNDING_STREAM_ZF_ADDRESSES, and FUNDING_STREAM_MG_ADDRESSES lists in the Testnet funding constants. |
2976000..3396000 | Deferred 12/100, Major Grants 8/100 | No deferred addresses; 13 copies of t2HifwjUj9uyxr9bknR8LFuQbc98c3vkXtu for Major Grants. |
3536500..4497948 | Deferred 12/100, Major Grants 8/100 | No deferred addresses; 27 copies of t2HifwjUj9uyxr9bknR8LFuQbc98c3vkXtu for Major Grants. |
Public address sequences are consensus data, not operator payout addresses.
Each lockbox_disbursements element has these fields:
| Field | Type | Default | Meaning |
|---|---|---|---|
address | String | Required | Transparent address receiving the specified one-time disbursement. |
amount | Integer amount | Required | Output value in zatoshis, from 0 through 2100000000000000. |
Zebra validates lockbox addresses during loading. The sum of all disbursements
must not exceed 2100000000000000 zatoshis.
The default Testnet list contains ten identical outputs, each paying
787500000000 zatoshis to t2RnBRiqrN1nW4ecZs1Fj3WWjNdnSs4kiX8.
Rust API: ConfiguredActivationHeights,
ConfiguredFundingStreams,
ConfiguredFundingStreamRecipient,
and ConfiguredLockboxDisbursement.
[state]
Rust API: zebra_state::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
cache_dir | Path string | Default cache directory | Root of the state database and non-finalized-state backup. Does not relocate peer caches or RPC cookies. |
ephemeral | Boolean | false | Store the database in a temporary directory instead of cache_dir, and disable non-finalized-state backups. The temporary database is removed on successful cleanup; a crash can leave it behind. |
should_backup_non_finalized_state | Boolean | true | Back up recent blocks outside the finalized database so they can be restored after a restart. Disabled by ephemeral; otherwise updates occur no more frequently than every five seconds. |
delete_old_database | Boolean | true | Check for and delete obsolete state database versions. Setting false disables old-version cleanup, not migration of a compatible previous database. |
debug_stop_at_height | Optional u32 | Absent | Commit blocks to finalized state up to the selected height, then exit. Developer option; omission keeps syncing. |
debug_validity_check_interval | Optional duration string | Absent | Run additional periodic state-validity checks. Without it, normal format checks still run at startup and shutdown. |
debug_skip_non_finalized_state_backup_task | Boolean | false | Developer option: replace the periodic backup task with synchronous writes before non-finalized-state and chain-tip updates. Does not disable restoring backups or override ephemeral/backup disabling. |
elasticsearch_url | String; elasticsearch feature only | "https://localhost:9200" | Elasticsearch service URL for experimental indexing. |
elasticsearch_username | String; elasticsearch feature only | "elastic" | Elasticsearch HTTP Basic authentication username. |
elasticsearch_password | String; elasticsearch feature only | "" | Elasticsearch HTTP Basic authentication password. Redacted from debug output, but stored in plaintext in the configuration file. Environment overrides are rejected. |
elasticsearch_skip_cert_validation | Boolean; elasticsearch feature only | false | Disable HTTPS certificate validation. Use only for trusted local clusters with self-signed certificates; credentials and indexed data can be intercepted. |
Persistent state uses <cache_dir>/state/v29/<network-directory> in Zebra
7.0.0. Non-finalized backups use
<cache_dir>/non_finalized_state/<network-directory>. Do not share a live state
directory between node processes. Deleting state forces the node to download
and verify the chain again; it is not a configuration-reset procedure.
The Elasticsearch client validates HTTPS certificates by default. Configure a
trusted server certificate, or explicitly opt out for a trusted local cluster
with elasticsearch_skip_cert_validation = true. HTTP never protects credentials.
See Elasticsearch indexing.
[sync]
Rust API: zebrad::components::sync::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
download_concurrency_limit | usize | 50 | Parallel block-download requests. Values below 1 are raised to 1 with a warning. The old key max_concurrent_block_requests is an accepted alias. |
checkpoint_verify_concurrency_limit | usize | 1000 | Blocks submitted concurrently to checkpoint verification. Values below 400 are raised to 400 with a warning. The old key lookahead_limit is an accepted alias. |
full_verify_concurrency_limit | usize | 20 | Blocks submitted concurrently to full verification. Values below 1 are raised to 1 with a warning. |
parallel_cpu_threads | usize | 0 | Rayon worker threads for CPU-intensive verification. Zero delegates thread-count selection to Rayon; a positive value overrides its automatic selection. |
Do not supply both an alias and its current key: they deserialize as duplicate fields.
With zero threads configured, Rayon
uses a positive RAYON_NUM_THREADS environment value, otherwise the available
logical CPU count, with a one-thread fallback if detection fails. If that
environment variable is absent or invalid, Rayon also checks the deprecated
RAYON_RS_NUM_CPUS before detecting the CPU count. An explicit
RAYON_NUM_THREADS=0 selects CPU detection without checking the deprecated
variable.
[mempool]
Rust API: zebrad::components::mempool::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
tx_cost_limit | u64 | 80000000 | Maximum sum of ZIP 401 transaction costs. Each transaction costs the greater of its serialized byte size and 10000; this is not a literal RAM-byte limit. |
eviction_memory_time | Duration string | "1h" | Retention limit for randomly evicted, mined, expired, and chain-conflicting transaction IDs in rejection caches. Capacity eviction or a mempool reset can remove entries earlier. Does not set transaction lifetimes or retention for every rejection reason. |
debug_enable_at_height | Optional u32 | Absent | Force-enable the mempool once the best chain reaches this height, regardless of synchronization status. Omission retains sync-status gating except on Regtest, where startup always overrides this field to 0. |
max_datacarrier_bytes | Optional u32 | 83 | Maximum standard OP_RETURN script size in bytes, including the opcode and pushdata overhead. Omission still uses 83. This is relay policy, not a block-consensus limit. |
[rpc]
Rust API: zebra_rpc::config::rpc::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
listen_addr | Optional socket-address string | Absent | Enable the HTTP JSON-RPC server at this IP address and port. Conventional loopback ports are 8232 for Mainnet and 18232 for Testnet; neither is enabled automatically. |
indexer_listen_addr | Optional socket-address string | Absent | Enable the indexer gRPC server. The listener itself does not require the indexer feature; that feature enables additional state spend indexes. |
lightwalletd_listen_addr | Optional socket-address string | Absent | Enable the experimental lightwalletd-compatible CompactTxStreamer gRPC server, for example at 127.0.0.1:9067. |
parallel_cpu_threads | usize | 0 | Deprecated compatibility field. Accepted but has no effect on the current JSON-RPC server. |
debug_force_finished_sync | Boolean | false | Test-only override that reports synchronization finished regardless of the estimated tip or local clock. |
cookie_dir | Path string | Default cache directory | Directory containing the JSON-RPC authentication cookie. Must be writable when cookie authentication is enabled. |
enable_cookie_auth | Boolean | true | Require cookie-based HTTP authentication for JSON-RPC. Does not authenticate either gRPC listener. |
max_response_body_size | u32 | 52428800 | Maximum JSON-RPC response body size in bytes, exactly 50 MiB. Does not configure the request-size limit. |
Bind RPC listeners to loopback or a restricted private interface; see JSON-RPC authentication. Neither gRPC listener uses cookie authentication. The lightwalletd-compatible listener serves plaintext HTTP/2 without authentication or TLS; terminate TLS at a controlled proxy before exposing it to clients. See Mining and Docker RPC.
[mining]
Rust API: zebra_rpc::config::mining::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
miner_address | Optional Zcash address string | Absent | Payout address for miner subsidy and transaction fees in generated block templates. Must be valid for the selected network and supported by the coinbase builder. |
extra_coinbase_data | Optional string | Absent | Text appended to the coinbase input after Zebra’s marker and separator. Maximum 86 UTF-8 bytes, not 86 characters; larger values fail configuration loading. Empty text adds no separator. |
miner_memo | Optional string | Absent | Memo text for a shielded coinbase output, at most 512 UTF-8 bytes. Zebra validates the length even for a transparent-only payout, which does not include the memo; an oversized value prevents template generation. |
internal_miner | Boolean | false | Start the experimental built-in miner when compiled with internal-miner. Accepted but has no effect without that feature. |
The obsolete miner_data key is rejected. Coinbase tag and memo strings are not
hex-decoded: "4142" means four UTF-8 bytes, not two.
See Mining for payout validation and template behavior. The internal miner uses one solver; its thread count is not configurable. Treat coinbase metadata as public, including shielded memos; see ZIP 213.
[tracing]
Rust API: zebrad::components::tracing::Config
and InnerConfig.
| Field | Type | Default | Meaning |
|---|---|---|---|
use_color | Boolean | true | Use ANSI colors when standard output is a terminal. This test also controls formatted logs sent to log_file; set false for plain-text log files. |
force_use_color | Boolean | false | Force ANSI colors even for non-terminal output and even when use_color is false. |
filter | Optional string | Absent | Tracing filter expression for start and copy-state; command-line filters override it. The fallback is info, or debug with -v. The generate and tip-height commands ignore configured and command-line filters: they always use warn, or info with -v. |
buffer_limit | usize | 128000 | Maximum queued log lines for the nonblocking writer. Values below 100 become 100. A full queue drops log lines. |
endpoint_addr | Optional socket-address string | Absent | Enable the runtime filter-control HTTP endpoint; requires filter-reload. Without the feature, Zebra warns and does not open it. |
flamegraph | Optional path string | Absent | Enable tracing-span flamegraph collection for start and copy-state; requires flamegraph. Writes .folded data and an .svg graph using the supplied path with its extension replaced. Without the feature, Zebra warns and does not collect it. The generate and tip-height commands ignore this setting. |
progress_bar | Optional string enum | Absent | Enable progress rendering with "summary" or "detailed"; requires progress-bar to render. Both modes currently render the same bars. Other string values deserialize as "summary". Setting this field also selects a default log file if none was specified. |
log_file | Optional path string | Absent | Log destination instead of standard output. When progress_bar is set, omission selects the platform log path below. |
use_journald | Boolean | false | Send tracing events to systemd-journald; requires journald. Without the feature, Zebra warns and does not install that layer. |
opentelemetry_endpoint | Optional URL string | Absent | Enable OpenTelemetry trace export over OTLP HTTP; requires opentelemetry. Falls back to OTEL_EXPORTER_OTLP_ENDPOINT. Zebra appends /v1/traces when absent from the URL suffix. |
opentelemetry_service_name | Optional string | Absent | Exported service name. Falls back to OTEL_SERVICE_NAME, then "zebra". |
opentelemetry_sample_percent | Optional u8 | Absent | Trace sampling percentage. Falls back to OTEL_TRACES_SAMPLER_ARG, then 100; values from 101 through 255 are clamped to 100. Zero samples no traces. |
The progress-bar log fallback is $XDG_STATE_HOME/zebrad.log or
$HOME/.local/state/zebrad.log on Linux,
$HOME/Library/Application Support/zebrad.log on macOS, and
%LOCALAPPDATA%\zebrad.log on Windows. Selecting a progress mode changes this
log destination even when the binary lacks progress rendering. If no platform
state directory is available, Zebra tries the local data directory; if neither
path is available, logs remain on standard output.
All tracing fields remain recognized without their optional implementation
features. An OpenTelemetry endpoint configured without the feature produces a
warning. Exporter initialization failure disables export, not the node.
The fallback OTEL_TRACES_SAMPLER_ARG is an integer percentage: 10 means 10%;
0.1 fails parsing and falls back to 100%. Prefer the explicit Zebra setting.
The filter-control endpoint has no authentication. Keep it private: remote changes can increase log volume and reveal operational details. Log files, flamegraphs, and exported traces can also contain node data.
Sentry has no TOML field or [sentry] section. With the sentry feature, set
SENTRY_DSN to configure reporting; SENTRY_ENVIRONMENT supplies its environment
metadata. See Tracing,
Sentry monitoring, and
OpenTelemetry export.
[metrics]
Rust API: zebrad::components::metrics::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
endpoint_addr | Optional socket-address string | Absent | Enable the Prometheus HTTP metrics listener. Requires prometheus; without it, Zebra accepts the field, warns, and leaves the endpoint disabled. |
The listener has no configured authentication or TLS. Restrict access to the monitoring network. Failure to bind a configured listener stops startup. See Metrics.
[health]
Rust API: zebrad::components::health::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
listen_addr | Optional socket-address string | Absent | Enable the health HTTP server. |
min_connected_peers | usize | 1 | Minimum recently live peers for /healthy and, when enforced, /ready. Zero disables the peer-count requirement. |
ready_max_blocks_behind | i64 | 2 | Maximum estimated lag in blocks accepted by /ready. Negative lag estimates are clamped to zero; a negative configured threshold cannot pass the enforced lag check. |
enforce_on_test_networks | Boolean | false | With false, /ready returns HTTP 200 on Testnet, custom Testnets, and Regtest without running readiness checks. Does not bypass /healthy. |
ready_max_tip_age | Duration string | "5m" | Maximum time since the sampled best height last exceeded its previous high-water mark. This is local progress freshness, not the latest block header’s timestamp. |
An enforced readiness check also requires a populated state and sync status close to the network tip. The HTTP endpoints are unauthenticated. Bind them to a private interface and restrict access with network policy. See Health checks.
[notify]
Rust API: zebrad::components::notify::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
block_notify_command | Optional string | Absent | Shell command launched after a best-chain tip change when Zebra is close to the network tip. Every %s becomes the new tip’s 64-digit hexadecimal block hash. |
zmqpubhashblock | Optional string | Absent | ZMQ PUB endpoint for best-chain tip hashes, equivalent to zcashd’s -zmqpubhashblock, for example "tcp://127.0.0.1:28332". Requires a binary built with the zmq feature. Startup fails if the endpoint is invalid or unavailable, or if the feature is missing. The socket has no authentication; expose it only to trusted clients. |
Zebra executes the command through /bin/sh -c on Unix or cmd /C on Windows,
with the node process’s privileges. Standard input, output, and error are
redirected to the null device. Treat this setting as executable code; use only
a trusted command or script.
Notification commands run independently of block validation. Notifications are best effort: fast synchronization and reorganizations can coalesce intermediate tips, and a slow command does not prevent later commands from starting. Do not use notifications as an exactly-once block index.
ZMQ notifications follow the same rules, and the publisher also drops messages when its bounded queue overflows. Subscribers must reconcile through RPC after a gap or a reconnect.
[zcashd_compat]
Rust API: zebrad::components::zcashd_compat::Config.
| Field | Type | Default | Meaning |
|---|---|---|---|
enabled | Boolean | false | Enable zcashd-compat mode for a zcashd P2P sidecar. The start --zcashd-compat flag also enables it. |
manage_zcashd | Boolean | false | Spawn and supervise the sidecar when compatibility mode is enabled. Otherwise zcashd is managed externally. |
zcashd_source | String enum | "path" | Binary source: "path" uses zcashd_path; "embedded" downloads and caches the release selected by Zebra’s embedded manifest. An explicit path overrides either selection. |
zcashd_path | Optional path string | Absent | Explicit sidecar executable, which must be resolvable for supervised "path" mode. This mode does not default to searching for an executable named zcashd. |
zcashd_datadir | Optional path string | Absent | Base sidecar data directory; falls back to <state.cache_dir>/zcashd-compat-zcashd. A valid extra -datadir=<path> argument overrides it. |
zcashd_extra_args | Array of strings, or JSON-array string | [] | Extra supervised-child arguments. Zebra rejects peer-selection arguments and appends its enforced isolation and foreground flags after these arguments. |
p2p_connect_addr | Optional socket-address string | Absent | Zebra P2P address passed to the supervised child’s -connect. Defaults to Zebra’s actual bound listener, substituting loopback in the same IP family for an unspecified address. |
block_gossip_peer_ips | Array of IP strings, or JSON-array string | [] | Trusted sidecar source IPs. An empty list becomes 127.0.0.1 and ::1 while compatibility mode is enabled. A nonempty list is rejected when compatibility mode is disabled. |
startup_delay | Duration string | "1s" | Delay before the first supervised spawn attempt. |
restart_backoff | Duration string | "2s" | Initial restart delay for exponential backoff. Must be greater than zero when supervising. |
restart_backoff_max | Duration string | "5m" | Maximum restart delay. Must be greater than zero when supervising. |
restart_reset_after | Duration string | "1h" | Child uptime that resets the consecutive restart count. Setting "0s" disables the reset. |
shutdown_grace_period | Duration string | "5m" | Time allowed for clean child shutdown after SIGTERM before forced termination. |
The embedded source supports Linux x86-64. Startup runs compatibility-mode preflight checks.
The extra-argument validator rejects -connect, -addnode, and -seednode,
including negated forms and variants with extra leading dashes or different
case. The supervisor sets its own peer connection, disables incoming P2P,
DNS seeding, onion listening, and discovery, and forces foreground operation.
For data-directory and configuration-file preparation, Zebra uses the last
valid -datadir=<path> or -conf=<path> argument, also accepting -- forms.
It warns about duplicates, empty paths, or paired arguments such as
["-conf", "file"]; paired values are passed through but not inferred for
preparation. Relative configuration paths resolve under the selected sidecar
data directory.
Trust every process able to connect from a listed sidecar IP; never list an address shared with untrusted tenants. The zcashd-compat page describes the sidecar’s reserved inbound capacity, reconnection exemptions, and pinned block gossip.
The connect-address override describes the address the sidecar dials. It does not identify the sidecar’s source IP for privileges. Across containers or hosts, configure the source IP list separately. A non-loopback connect override with an empty trusted list produces a warning because privileges still default to loopback.
Running Zebra
Zebra downloads, verifies, stores, and relays the Zcash chain. The zebrad
process runs continuously; it does not stop when the initial sync finishes.
Enable JSON-RPC to serve applications such as
lightwalletd, Zallet, and
mining pools.
Commands
Run Zebra with its default Mainnet configuration:
zebrad start
Omitting the subcommand also starts Zebra. Put top-level options, including
--config, before the subcommand:
zebrad --config /path/to/zebrad.toml start
| Command | Purpose |
|---|---|
zebrad start | Run the node. Optional positional arguments override the tracing filter, for example zebrad start info,zebra_network=debug. |
zebrad generate | Print a default TOML configuration to standard output. Use --output-file /path/to/zebrad.toml to write it to a file. |
zebrad tip-height --network Mainnet | Print the finalized chain tip height stored on disk. Use --cache-dir /path/to/cache to override the configured cache root. |
zebrad copy-state --target-config-path /path/to/target.toml | Copy finalized blocks to another state database for debugging. The source uses the main configuration; only [state] settings are used from the target configuration. |
zebrad help COMMAND | Show the options for a command. |
zebrad --version | Print the build version. |
The tip-height command opens a read-only state instance, does not connect to
peers, and does not upgrade an older database. Its output is not the live,
non-finalized chain tip. On a read failure it logs an error rather than printing
a height; check for numeric output, not only its exit status.
The copy-state command is an expert diagnostic tool, not a snapshot download
or full revalidation command. Use a distinct target cache directory. With
--target-config-path, ZEBRA_TARGET_ environment variables override target
settings without changing the source. Without that option, the command ignores
target overrides and uses temporary state that is deleted on a successful exit.
The --max-source-height HEIGHT option limits the copy.
Configuration and network
See configuration for file selection, precedence, and environment overrides.
Generate a starting configuration on Linux:
mkdir -p "${XDG_CONFIG_HOME:-$HOME/.config}"
zebrad generate --output-file "${XDG_CONFIG_HOME:-$HOME/.config}/zebrad.toml"
Keep only fields that need an explicit setting.
The [network] network field selects "Mainnet" by default. To use public
Testnet, set:
[network]
network = "Testnet"
When switching networks, remove an explicit [network] listen_addr to use the
new network’s default port, or change its port. To select Testnet through the
environment:
ZEBRA_NETWORK__NETWORK=Testnet zebrad start
Use "Regtest" for a local development chain; see Regtest.
See custom Testnets for networks with other consensus
parameters. Separate processes need separate listener addresses and must not
write the same state database. Give each RPC server its own [rpc] cookie_dir,
even on different networks, so one server does not overwrite another’s cookie.
State directory
The state cache contains a finalized RocksDB database of blocks, unspent transaction outputs, and validation/query indexes, plus backups of recent blocks that can still be reorganized.
Zebra’s chain state contains no wallet keys. Deleting it requires downloading
and verifying the chain again. Do not enable [state] ephemeral for a
persistent node. See state snapshots for restoration and
upgrading for format migrations and rollback.
First sync
An empty node starts with the genesis block and downloads the chain from peers.
A checkpoint is a block height and hash compiled into Zebra that identifies a
known chain. With the default [consensus] checkpoint_sync = true, Zebra
checks the chain of block hashes against those checkpoints through the last
checkpoint. It does not repeat all proof and signature checks for those
blocks. Above that height, Zebra fully verifies new blocks, including their
transactions, before accepting them.
The protocol’s blockchain rules require validators to follow the chain containing the settled network upgrades. Zebra’s checkpoint routing enforces known checkpoint hashes both while syncing and when checking existing state at startup.
The checkpoint_sync = false setting is for development. It starts full
verification after the first checkpoint at or above the block immediately
before Canopy activation. It does not disable mandatory checkpoints or
provide full verification from genesis: Zebra does not implement every legacy
pre-Canopy validation rule. Disabling checkpoint sync is slower and removes
protection provided by later checkpoints; keep the default for production.
A state snapshot replaces the initial download and validation of archived blocks with trust in the snapshot producer. See Startup and sync for the task sequence and log messages.
Progress and readiness
Zebra logs sync progress approximately once a minute; see progress messages for the fields. Keep the system clock synchronized: progress estimates use local time.
For automated readiness checks, configure the health server and
query /ready. A listening RPC port alone does not establish that the node
is synced.
Enable terminal progress bars with:
[tracing]
progress_bar = "summary"
Progress bars require the progress-bar
Cargo feature and redirect logs to a file.
See tracing configuration for modes and log paths, and
metrics for monitoring without a terminal.
Ports
The table describes native zebrad defaults, not container port mappings.
All endpoints use TCP.
| Endpoint | Configuration field | Default |
|---|---|---|
| Mainnet P2P | [network] listen_addr | [::]:8233 |
| Public Testnet P2P | [network] listen_addr | [::]:18233 |
| Regtest P2P | [network] listen_addr | [::]:18233, the same port as Testnet |
| JSON-RPC | [rpc] listen_addr | Disabled; no default port. Recommended explicit ports are 8232 for Mainnet and 18232 for Testnet. |
| Indexer gRPC | [rpc] indexer_listen_addr | Disabled; no default port. |
| Experimental lightwalletd-compatible gRPC | [rpc] lightwalletd_listen_addr | Disabled; no default port. |
| Prometheus metrics | [metrics] endpoint_addr | Disabled; no default port; requires the prometheus feature. |
| HTTP health and readiness | [health] listen_addr | Disabled; no default port. |
| Tracing-filter reload | [tracing] endpoint_addr | Disabled; no default port; requires the filter-reload feature. |
| Tokio console | TOKIO_CONSOLE_BIND environment variable | 127.0.0.1:6669; requires the tokio-console feature and tokio_unstable compiler configuration. |
The P2P wildcard listener accepts IPv6 and IPv4 connections where the operating system supports dual-stack sockets. Set a specific address to restrict the listener. Zebra’s Regtest default differs from zcashd’s port 18344.
Bind administrative and monitoring endpoints to loopback or a protected network. See RPC authentication before exposing RPC; the indexer, lightwalletd-compatible, and health endpoints do not inherit JSON-RPC authentication.
Stopping Zebra
On Unix, press Ctrl-C or send SIGINT or SIGTERM to the zebrad process.
Zebra logs received SIGINT, starting shutdown or the corresponding SIGTERM
message. On Windows, Ctrl-C starts shutdown; Zebra does not implement Windows
service-control messages directly.
Zebra stops its asynchronous runtime and waits up to 20 seconds for runtime-owned work to stop. Asynchronous tasks are canceled rather than drained. State cleanup closes the block-write channels and flushes RocksDB data and its write-ahead log. This does not guarantee that every in-flight block or latest non-finalized block has reached disk: cleanup can be interrupted, and the non-finalized backup is periodic. Zebra resumes from persisted state on restart.
Wait for the process to exit before moving, copying, or replacing its state.
Avoid SIGKILL for routine shutdown because it skips cleanup. If Zebra manages
a zcashd child in compatibility mode, shutdown also waits
for that child using [zcashd_compat] shutdown_grace_period.
Exit codes
| Status | Meaning |
|---|---|
0 | Normal command completion, help/version output, or a handled shutdown signal. |
1 | Application initialization failure, a returned runtime error, or a failure during application shutdown. |
2 | Command-line usage error. The application also reserves this code for its explicit Shutdown::Crash path. |
| Platform-dependent | Termination by a signal, an abort, or another operating-system failure. |
A panic is not reliably exit code 2. Zebra’s development and release profiles
use panic = "abort"; a panic terminates the process without normal unwinding.
On Unix, process supervisors distinguish this signal termination from a normal
exit status. Do not interpret every code 2 as a crash or every code 0 as proof
that a utility produced valid output; in particular, check tip-height output.
End of support
Zebra enforces a height-based support window on Mainnet. Zebra 7.0.0 uses these compiled constants:
| Constant | Value | Effect |
|---|---|---|
ESTIMATED_RELEASE_HEIGHT | 3,526,000 | Starting height for the support-window calculation. |
EOS_WARN_AFTER | 42 days | Warn after the tip exceeds the height estimated 42 days after the starting height. |
EOS_PANIC_AFTER | 56 days | Panic after the tip exceeds the height estimated 56 days after the starting height. |
The calculation advances from the starting height using each network upgrade’s target block spacing, including spacing changes within the window. It is not a timer measured from installation, process startup, or the release’s calendar date. The last supported height is inclusive; the first unsupported height is one block higher.
The task waits ten seconds after startup, then checks the local best tip once
per hour when a tip exists. It does not enforce expiry on Testnet, configured
Testnets, or Regtest. Mainnet warnings begin with
Your Zebra release is too old and it will stop running; expiry panics begin
with Zebra refuses to run. Upgrade to a supported release
before reaching the reported cutoff. Restarting the same binary does not
extend its support window.
Startup and sync
An open listener does not establish readiness; use the health checks. Startup tasks run concurrently, so log messages interleave. The examples omit timestamps, tracing spans, and variable fields. See tracing for filters and metrics for dashboards.
Load configuration and diagnostics
Zebra loads configuration, applies command-line overrides, and initializes logging and the CPU thread pool. It logs the build version, network, compiled features, state format, and loaded configuration.
Without a configuration file, the log includes:
No config file provided, using default configuration
initialized rayon thread pool for CPU-bound tasks
Environment overrides still apply when this message says “default
configuration.” With a file, Zebra instead logs Using config file at: and
the selected path. Configured metrics and tracing-filter reload listeners
start during this application initialization, before the node state opens.
The startup command then logs:
Starting zebrad
On Linux, it checks the TCP slow-start-after-idle setting and warns if it is enabled. See Linux TCP tuning. In zcashd-compat mode, Zebra performs its preflight checks before opening state. It also resolves the zcashd binary when supervision is enabled.
Open the state
Zebra selects the checkpoint boundary and opens or creates the state database, restoring eligible non-finalized block backups. See upgrading for format migrations.
initializing node state
opening database, this may take a few minutes
loaded Zebra state cache
logging database metrics on startup
The Opened Zebra state cache at message identifies the actual database
path. The loaded Zebra state cache event includes chain_tip: None for
empty state, or the restored block hash and height. Check these fields after
restoring a snapshot or changing [state] cache_dir.
Database opening precedes block downloads; a pause here is not a peer-discovery failure. Format upgrades and checks run on a separate thread and overlap syncing.
Initialize peer connections
Zebra opens [network] listen_addr and combines configured seed peers with
cached peer addresses.
initializing network
starting the address book updater
resolved seed peer IP addresses
connecting to initial peer set
starting the peer address crawler
The DNS message appears when a seed resolves. The loaded cached peer IP addresses event appears when Zebra reads a usable cache. The listener reports
Opened Zcash protocol endpoint at followed by its bound address. Individual
seed lookups and peer connections can fail while other connections succeed.
Initialize verification and the mempool
The mempool holds verified, unmined transactions.
initializing verifiers
initializing block verifier router
initializing syncer
initializing mempool
fully initializing inbound peer request handler
The router’s log includes tip and max_checkpoint_height; see
first sync for checkpoint and full verification rules.
A separate startup task checks stored block hashes against the compiled checkpoints. Its first message is:
starting state checkpoint validation
This check does not fully revalidate the stored database. A mismatch aborts startup or operation with an invalid-chain error. With incomplete state and checkpoint sync enabled, Zebra reports that remaining checkpoints will be verified during syncing.
On Mainnet and public Testnet, the mempool activates when Zebra is near the chain tip, rather than during the bulk initial download. Regtest enables it from height zero. Transactions admitted to the mempool use transaction verification, not checkpoint verification.
Start application interfaces
Zebra starts JSON-RPC when [rpc] listen_addr is set, logging
Opened RPC endpoint at and its bound address without waiting for sync.
Additional services depend on configuration:
| Service | Startup condition | Message |
|---|---|---|
| Block-template updater | Mining configuration enables template preparation, and JSON-RPC or the internal miner can use it | spawned block template updater task |
| Managed zcashd | Both [zcashd_compat] enabled and manage_zcashd are true | zcashd-compat mode enabled |
| Indexer gRPC | [rpc] indexer_listen_addr is set | spawning indexer RPC server |
| Experimental lightwalletd-compatible gRPC | [rpc] lightwalletd_listen_addr is set | spawning lightwalletd gRPC server |
See RPC, mining, and
zcashd compatibility for the interface contracts.
The indexer warning configure an indexer_listen_addr to start the indexer RPC server means that this optional listener is disabled; it is not a sync error.
Launch background tasks
These markers indicate task startup, not completed work:
| Task | Work | Message |
|---|---|---|
| Block gossip | Advertise new best-tip block hashes to peers | spawning block gossip task |
| Block notification | Run [notify] block_notify_command when configured | spawning block notify task |
| Mempool queue checker | Poll the mempool to collect completed transaction-verification work | spawning mempool queue checker task |
| Transaction gossip | Advertise transactions added to the mempool | spawning mempool transaction gossip task |
| Old database cleanup | Remove eligible old database directories when [state] delete_old_database is true | spawning delete old databases task |
| Progress reporting | Estimate sync progress, log it, and update health data | spawning progress logging task |
| Health server | Serve health and readiness when [health] listen_addr is set | initializing health endpoints |
| End-of-support check | Enforce the Mainnet release support window | spawning end of support checking task |
| Mempool crawler | Request unmined transactions from peers when enabled near the tip | spawning mempool crawler task |
| Block syncer | Download and verify blocks needed to extend local state | spawning syncer task |
The block-notification marker appears even without a configured
command. The health marker appears even when
disabled; an enabled listener also logs opened health endpoint at and its
address. See end of support for the release cutoff.
On Regtest, Zebra commits the genesis block locally if it is absent, then runs
the syncer to recover blocks from peers. The internal miner starts last when
its build feature and configuration enable it, with spawning Zcash miner.
Once these tasks have been launched, Zebra logs:
spawned initial Zebra tasks
This marks task startup, not readiness. Ordinary sync retries do not require restarting the process.
Sync the chain
Download genesis and discover extensions
On Mainnet or Testnet with empty state, Zebra downloads and verifies the genesis block. The syncer and progress reporter emit:
starting genesis block download and verify
initial sync is waiting to download the genesis block
With existing state, Zebra resumes from its stored chain. It sends block
locators, which identify known chain ancestors, to peers and requests hashes
of later blocks. Responses contain at most 500 block hashes. The syncer
schedules downloads and verification concurrently within [sync] limits,
then asks for further extensions:
starting sync, obtaining new tips
extending tips
Verify checkpoint ranges
Successful checkpoint ranges produce:
verified checkpoint range
This event includes block_count and current_range. Range verification and
state writes are separate stages; incoming blocks do not immediately imply a
higher committed tip.
Fully verify recent blocks
Above the checkpoint boundary, Zebra fully verifies blocks and transactions. The progress reporter logs:
estimated progress to chain tip
Its fields include current_height, remaining_sync_blocks, sync_percent,
and time_since_last_state_block. The estimate uses block times, target block
spacing, and the local clock. It is not a guaranteed completion percentage or
remaining-time prediction.
Follow new blocks
Near the chain tip, the bulk syncer and block announcements from peers overlap. Zebra reports the transition with:
close to finishing initial sync, confirming using syncer and gossiped blocks
finished initial sync to chain tip, using gossiped blocks
The syncer remains active to recover missing blocks while the inbound handler downloads and verifies announced blocks. Monitor tip freshness and peers; see troubleshooting if height stops advancing.
Syncing from a state snapshot
The Zcash Foundation publishes compressed Zebra state archives at https://snapshots.zfnd.org/. Zebra resumes syncing from the archived tip. Check database compatibility before downloading.
On trust
A snapshot substitutes the producer’s validation for your own initial sync. Zebra checks database structure and stored checkpoint hashes on startup, but it does not fully revalidate every stored block and index. Those checks do not make an untrusted snapshot safe. Sync from genesis if your security policy requires validating the chain without trusting a state provider.
A SHA-256 checksum detects a damaged or incomplete download. A checksum served beside an archive does not authenticate its producer: a compromised server can replace both files.
A GitHub artifact attestation authenticates the manifest’s producing workflow; the manifest’s archive digest binds it to the archive. It does not prove that the chain data is valid. For unsigned manifests, obtain the expected digest through an independent trusted channel if you need protection against compromise of the download server.
Check the database format version
Zebra 7.0.0 uses database format 29.0.0, with finalized state under
<cache_dir>/state/v29/<network>/. Compare the manifest’s
database_format_major_version with the state format, not
the producer’s Zebra release number.
Zebra also reuses compatible format 28 state without a resync, moving
state/v28/<network>/ to state/v29/<network>/ when no usable format 29
database exists there. The old state must resolve inside the cache root;
symlinks outside it are skipped. Reuse considers only the immediately
preceding major version, not format 27 or older. Do not rename a database
directory manually to claim compatibility.
Restore into a fresh cache root. Keep an independent backup for rollback: reuse moves the database rather than copying it. See upgrading for migration and rollback procedures.
Resolve the archive
Install curl, jq, GNU tar with Zstandard support, zstd, and
sha256sum. Run the following commands in the same shell and download
directory. Use network=testnet for public Testnet.
The per-network index is newest first. The latest role is not filtered by
database version, so select the newest format that Zebra 7.0.0 can restore:
network=mainnet
curl -fLsS "https://snapshots.zfnd.org/$network/snapshots.json" \
-o snapshots.json
snapshot=$(jq -ce '
map(select(.db_major == 29 or .db_major == 28))
| first // error("no compatible snapshot")
' snapshots.json)
url=$(printf '%s\n' "$snapshot" | jq -er '.url')
manifest_url=$(printf '%s\n' "$snapshot" | jq -er '.manifest_url')
archive=${url##*/}
manifest=${manifest_url##*/}
curl -fLsS "$manifest_url" -o "$manifest"
jq '{network, chain_tip, database_format_version, archive}' "$manifest"
Stop if a command fails. Confirm the manifest’s network, database format,
archive.extraction_path, and chain_tip.note.
Allocate space for the compressed archive, extracted database, and ongoing
growth. The manifest’s archive.size_bytes is only the compressed size.
Verify and download
Signed manifests
If the publication site lists an attestation for the selected manifest, download its bundle and verify it with the GitHub CLI. Obtain the trusted roots from the CLI itself, not from the snapshot server:
curl -fLsS "$manifest_url.sigstore.json" -o "$manifest.sigstore.json"
gh attestation trusted-root > trusted_root.jsonl
gh attestation verify "$manifest" \
--bundle "$manifest.sigstore.json" \
--custom-trusted-root trusted_root.jsonl \
--repo ZcashFoundation/zcash-fleet-ops \
--signer-workflow ZcashFoundation/zcash-fleet-ops/.github/workflows/snapshot-cut.yml \
--source-ref refs/heads/main
Do not use a snapshot whose manifest verification fails. An unsigned manifest cannot pass this step; use the trust policy described in On trust rather than treating a missing attestation as successful verification.
Archive checksum
Download the archive and verify it against the manifest’s digest before extracting it:
curl -fL "$url" -o "$archive"
printf '%s %s\n' "$(jq -er '.archive.sha256' "$manifest")" "$archive" \
| sha256sum --check
For an unsigned manifest, this checks transfer integrity only. For a verified manifest, it also checks that the archive matches the authenticated digest. Do not extract a file that fails verification. Re-download it and investigate persistent mismatches.
Do not pipe an unverified download directly into the live state directory. Streaming extraction writes files before the complete archive digest is known and can leave partial state after a failed transfer.
Extract into your cache directory
Stop Zebra before replacing its state. Extract into a new cache root, not over an existing RocksDB database. Keep the old root until the restored node works and any rollback requirements are satisfied.
Replace /path/to/zebra-cache with the new cache root:
cache_dir=/path/to/zebra-cache
mkdir -p "$cache_dir"
tar --zstd -xf "$archive" -C "$cache_dir"
The archives contain the state/v<major>/<network>/ prefix. Do not extract into
that subdirectory a second time; the resulting path is, for example,
/path/to/zebra-cache/state/v28/mainnet/.
Make the extracted files writable by the account that runs Zebra. Set the same root in its configuration, leaving persistent state enabled:
[state]
cache_dir = "/path/to/zebra-cache"
ephemeral = false
Match [network] network to the archive. Check
environment overrides, container mounts,
and state paths.
Start Zebra with that configuration:
zebrad --config /path/to/zebrad.toml start
Check Opened Zebra state cache at for the selected path and chain_tip in
loaded Zebra state cache for the restored height. Reuse logs
moved state cache from with the old and new paths.
Docker
Stop the Zebra container before restoring its volume. The example below uses
the zebrad-cache volume and /home/zebra/.cache/zebra mount from the
Docker quick start. Use a new volume when replacing
an existing database, then update the Zebra container’s volume mapping.
After downloading and verifying the archive on the host, extract it through a temporary container:
docker run --rm \
-v zebrad-cache:/home/zebra/.cache/zebra \
-v "$PWD":/snapshots:ro \
-e ARCHIVE="$archive" \
alpine sh -c 'apk add --no-cache zstd tar &&
tar --zstd -xf "/snapshots/$ARCHIVE" -C /home/zebra/.cache/zebra'
The standard image entrypoint repairs cache ownership on startup.
A custom non-root entrypoint must arrange write permissions itself. Keep the
volume mount and ZEBRA_STATE__CACHE_DIR consistent.
Troubleshooting
- Zebra syncs from genesis. Check the network, effective cache root, extraction prefix, and database format. An unsupported major directory is not a substitute for the directory Zebra opens. A missing usable database causes Zebra to create an empty one.
- Zebra ignores the restored database. Check the reuse conditions; restore into a fresh root rather than combining databases.
- The database cannot open. Preserve the full error and follow database troubleshooting.
- The starting height differs from the manifest. Read
chain_tip.note. Older manifests can describe a live source-node tip rather than the archived finalized tip. A restart can also restore separately cached non-finalized blocks; do not mix unrelated backups into the new root.
See troubleshooting for runtime failures after restoration.
Upgrading Zebra
Two deadlines force upgrades. A network upgrade changes the consensus rules at a fixed height, and only releases that implement it can follow the chain past that height. On Mainnet, each release also halts once the chain passes its end-of-support height. Releases can also change the state format, the configuration schema, RPC responses, metrics, and logs; the release notes list every change that needs operator action under Breaking Changes.
When to upgrade
| Trigger | Deadline | Where to find it |
|---|---|---|
| Network upgrade | Before its activation height | Activation heights; release notes of the first release that supports it |
| End of support (Mainnet only) | Before the chain passes the release’s support height | Startup log Zebra release is supported until block N; the getdeprecationinfo RPC field end_of_service.block_height |
| Security fix | On release | Security advisories and the release’s Security notes |
Zebra logs Your Zebra release is too old and it will stop running at block N
from 14 days before the support height. The getdeprecationinfo field
end_of_service.estimated_time reports the halt as a Unix timestamp, 24 hours
early. Both estimates assume the node is synced to the network tip.
The zebrad version is a SemVer version. A major
release adds Mainnet network-upgrade support or breaks an operator interface:
configuration, CLI, RPC, or a state format that older releases cannot read.
Minor and patch releases are compatible; see
Zebra versioning. Release candidates
are prereleases for testing and do not move the Docker latest tag.
Before upgrading
Choose a supported release that implements upgrades active now and those activating before your next planned update.
- Read the release notes of every release after the one you run, up to the target. Act on each Breaking Changes entry: Zebra refuses to start with a configuration field it does not recognize, and dashboards, alerts, and RPC clients break on renamed metrics, log lines, or response fields.
- Check whether the target changes the major state format. The release notes
say so; the running format is the
v<major>directory under<[state] cache_dir>/state/. If the major changes, take a backup: no older release can read the upgraded state. If your state is older than the target’s immediately preceding major, the target cannot reuse it; see first start. - Upgrade the software that depends on Zebra in the same window (see dependent software).
- Download and verify the new release as for a new installation.
Backing up the state
A backup is a copy of one network’s state directory, taken while no process
has the database open. Keep it outside [state] cache_dir, where no Zebra
process moves, upgrades, or deletes it.
-
Stop Zebra (Stopping Zebra) and every process that opens its database directly.
-
Copy the directory into a new, dated backup directory. For a Mainnet node with the default cache directory on Linux, before upgrading to Zebra 7.0.0:
backup=/srv/zebra-backup/$(date +%F)-v28 mkdir -p /srv/zebra-backup mkdir "$backup" && cp -a ~/.cache/zebra/state/v28/mainnet "$backup/"For the Docker image, copy out of the state volume:
docker run --rm \ -v zebrad-cache:/cache -v /srv/zebra-backup:/backup alpine \ sh -c 'b="/backup/$(date +%F)-v28" && mkdir "$b" && cp -a /cache/state/v28/mainnet "$b/"'The
mkdirof the dated directory fails if that backup already exists, so a repeated command never mixes two copies.
The copy needs as much free space as the directory occupies (du -sh it
first). A recent snapshot of the old format can replace a
backup for rollback, at the cost of syncing the blocks after its height.
Upgrading
Stop the old release, replace it, and start the new one with the same
configuration and state directory. Never run two zebrad processes on the same
state.
Binary installations
Replace the executable with one of:
cargo binstall [email protected]
cargo install --locked --version 7.0.0 zebrad
or with the verified release archive. Then start zebrad as before.
Docker
Stop the container without Docker’s default 10-second kill deadline, then recreate it from the new tag with the same volume, ports, and environment:
docker stop --timeout -1 zebra
docker rm zebra
docker run -d --name zebra --restart unless-stopped \
-p 8233:8233 -v zebrad-cache:/home/zebra/.cache/zebra \
zfnd/zebra:7.0.0
Zebra with Docker documents the image and its tags.
What happens on first start
The new release looks for its own major-version directory,
<cache_dir>/state/v<major>/<network>, and acts on what it finds:
| On disk | First start of the new release |
|---|---|
| The current major directory | Opens it. If the on-disk minor or patch format is older than the release’s, a background task upgrades it in place while the node runs and syncs. A newer on-disk format is opened as compatible and marked as downgraded (see rolling back). |
| Only the immediately preceding major, when the release registers a reusable upgrade from it | Renames the directory to the new major (log moved state cache from … to …), then upgrades it in place. No copy and no resync; the old directory no longer exists. |
That directory, but resolving outside [state] cache_dir through a symlink | Does not move it (log skipped reusing previous state cache) and syncs a new database from genesis. |
| Only older majors | Syncs a new database from genesis. Cleanup (below) keeps reusable predecessors on disk. |
Format upgrades log trying to open older database format: launching upgrade task, then Zebra automatically upgraded the database format and database format is valid when done. Stopping Zebra during an upgrade is safe: the
version file advances only after each step completes, and the next start
resumes the remaining work.
The setting [state] delete_old_database (default true) makes each start
delete older major-version directories under the cache directory, except the
predecessors of registered reusable upgrades (v25 to v28 in 7.0.0).
Setting it to false keeps every directory, but does not prevent the rename
above. Renaming or symlinking a versioned directory by hand does not convert
its format.
To avoid a sync from genesis when skipping majors, restore a snapshot whose format the target can open or reuse, instead of starting from the old directory.
Dependent software
| Software | Coupling | Action |
|---|---|---|
Processes that open the database directly: Zallet’s zebra backend, Zaino’s read-state source | Built against a zebra-state version that reads one major format | Upgrade them to a build that reads the new format. Start Zebra first, wait for database format is valid, then start them. |
RPC clients: lightwalletd, Zaino’s RPC source, Zallet’s zaino backend, mining pools | JSON-RPC and gRPC responses | Check the release notes’ RPC entries. |
Managed zcashd-compat sidecar ([zcashd_compat] zcashd_source = "embedded") | Each zebrad release pins one sidecar build | Nothing: the new release downloads its pinned build. |
Sidecar from zcashd_source = "path" or run externally | The sidecar must also implement each network upgrade | Install the build pinned in zcashd-compat-manifest.json; see zcashd-compat mode. |
| Dashboards and alerts | Metric names and labels, log lines | Check the release notes’ metrics and log entries. |
Record the Zebra revision used to build each direct reader, including cached binaries; the downstream application’s revision alone does not establish state-format compatibility.
Verifying the upgrade
- The executable reports its version:
zebrad --version. A running node reports it in thegetinfoRPC fieldbuild, in the Prometheus metriczebrad_build_info{version="…"}, and in its peer user agent/Zebra:<version>/. - The logs show the state handling above and then sync progress.
- The readiness endpoint returns
200once the node is back near the network tip. - The
getblockchaininfoRPC lists the network upgrades the release knows inupgrades, with their activation heights.
Rolling back
Rolling back is limited by the same deadlines: the older release cannot follow the chain past a network upgrade it does not implement, and on Mainnet it halts at its own support height.
Within the same major state format, start the older release on the existing
state. It opens a newer minor or patch format and records the downgrade
(trying to open newer database format: data should be compatible).
Across a major format change, the older release looks for its own, older directory, which the upgrade renamed away. Without the backup it syncs from genesis. To restore the backup:
-
Stop Zebra and every process that opens its database directly.
-
Move the new major directory aside, so that a later upgrade reuses the restored directory instead, then restore the backup. Each
mkdirfails instead of reusing an existing directory:state=~/.cache/zebra/state backup=/srv/zebra-backup/2026-10-23-v28 # the backup to restore mkdir /srv/zebra-backup/abandoned-v29 && mv "$state/v29/mainnet" /srv/zebra-backup/abandoned-v29/ mkdir -p "$state/v28" mkdir "$state/v28/mainnet" && cp -a "$backup/mainnet/." "$state/v28/mainnet/" -
Start the older release.
Roll back direct readers to builds for the older format at the same time.
Zebra 7.0.0
- NU7. Mainnet activates NU7 at height 3,543,405; 6.x releases cannot follow the chain past it. After activation, Mainnet nodes disconnect peers that advertise a protocol version below 170190.
- State format 28 → 29. The first start renames
state/v28/<network>tostate/v29/<network>without a resync. Zebra 6.x cannot read format 29: back upstate/v28first. A format-27 state (Zebra 2.4 to 5.x) is not reused; restore a snapshot instead of syncing from genesis. - Support window. 7.0.0 is supported for 56 days after its estimated release height, instead of the usual 12 weeks.
- Dependent software.
- The zcashd-compat sidecar must be
zebra-compat-v1.3.0; earlier builds stop at NU7. - Zallet 0.1.0-beta.3’s
zebrabackend reads format 28 and cannot read a 7.0.0 state; its RPC-basedzainobackend is unaffected.
- The zcashd-compat sidecar must be
- Breaking changes. These are in the notes for
7.0.0 and
7.0.0-rc.0:
- Per-peer labels are removed from the network metrics.
getblocktemplateadvertises onlytimeas mutable.- Custom testnets reject configurations that were previously accepted and store their state in new directories; resync custom networks.
Zebra with Docker
The Zebra runtime image packages zebrad and its startup script. It includes
no configuration file, including docker/default-zebra-config.toml, no chain
database, and no zcashd. Containerized nodes have the same
resource requirements as native nodes.
Images and tags
Releases and pre-releases publish to Docker Hub
as zfnd/zebra; the same tag selects linux/amd64 or linux/arm64.
| Tag | Meaning |
|---|---|
7.0.0 | The image built for the v7.0.0 release; Docker version tags omit the v. |
latest | Updated by stable release publication; also selected when the image reference omits a tag. |
7.1.0-alpha.0 | One exact pre-release. Release candidates (-rc.N) are cut from main; other pre-releases come from the next branch. |
edge | The most recent pre-release from next; may not exist. Never a release candidate. |
Release builds do not publish commit-SHA tags. Pre-releases never move
latest, and publishing workflows do not maintain moving beta or rc tags.
Use a version tag for an explicit upgrade policy, or an image digest to pin exact image content. Pulling a newer image does not replace a running container; recreate the container with its existing state volume.
Non-release image publishing belongs to continuous delivery.
Quick start
Run a Mainnet node with a persistent named volume:
docker run -d \
--name zebra \
--restart unless-stopped \
-p 8233:8233 \
-v zebrad-cache:/home/zebra/.cache/zebra \
zfnd/zebra:7.0.0
To start from existing state, populate the volume from a state snapshot before starting Zebra.
Use a separate volume and port for a Testnet node:
docker run -d \
--name zebra-testnet \
--restart unless-stopped \
-e ZEBRA_NETWORK__NETWORK=Testnet \
-e 'ZEBRA_NETWORK__LISTEN_ADDR=[::]:18233' \
-p 18233:18233 \
-v zebrad-testnet-cache:/home/zebra/.cache/zebra \
zfnd/zebra:7.0.0
Inspect startup and synchronization logs with docker logs -f zebra. Stop
Zebra without Docker’s forced-kill deadline before replacing its container or
restoring state:
docker stop --timeout -1 zebra
Docker’s default Linux stop timeout is ten seconds; use -1
to allow Zebra’s shutdown to finish. Stop and remove
the old container before reusing its name and state volume. Never run two
Zebra processes against the same state database.
Paths and permissions
The image’s zebra account has UID and GID 10001 and home /home/zebra:
| Container path | Purpose |
|---|---|
/usr/local/bin/zebrad | Node executable. |
/usr/local/bin/entrypoint.sh | Startup script. |
/home/zebra/.cache/zebra | Default state-cache, peer-cache, and RPC-cookie directory. |
/home/zebra/.cache/zebra/.cookie | RPC authentication file, created only when RPC and cookie authentication are enabled. |
/home/zebra/.config/zebrad.toml | Optional configuration file discovered by Zebra. |
The entrypoint starts as root, prepares writable directories, then executes
Zebra through setpriv --reuid=10001 --regid=10001 --init-groups.
Before dropping privileges, the entrypoint recursively changes ownership of
the state-cache directory, cookie directory, and configured log-file directory.
It also changes ownership of each immediate parent unless that parent is /
or $HOME. Bind mounts therefore change ownership on the host, not just
inside the container. Mount dedicated directories, not a shared home or log
tree. Recursive ownership changes also traverse an existing state database on
every startup.
Mount cache directories read-write. Mount configuration files read-only and
make them readable by UID 10001; the entrypoint does not repair config-file
permissions. Paths configured only in TOML must already be writable by the
container user because directory preparation reads environment variables, not
the TOML file.
The Docker --user option bypasses root startup but does not skip the
entrypoint’s ownership operations. It can therefore fail on mounts whose
ownership the selected user cannot change. To bypass preparation deliberately,
use both --user 10001:10001 and --entrypoint /usr/local/bin/zebrad, and
prepare every writable mount beforehand. This also bypasses CONFIG_FILE_PATH;
pass --config directly when using that override.
Configuring Zebra
Use Zebra’s environment variables or mount a TOML configuration.
Mounted configuration
Generate a configuration on the host:
docker run --rm zfnd/zebra:7.0.0 zebrad generate > zebrad.toml
Edit the fields you need, then mount the file at the default discovery path:
docker run -d \
--name zebra \
-p 8233:8233 \
-v zebrad-cache:/home/zebra/.cache/zebra \
--mount type=bind,src="$PWD/zebrad.toml",dst=/home/zebra/.config/zebrad.toml,readonly \
zfnd/zebra:7.0.0
For a different path, mount the file there and set CONFIG_FILE_PATH to its
container path. The entrypoint supplies --config only when that path is an
existing regular file. A missing path does not cause the entrypoint to fail;
Zebra falls back to normal configuration discovery. To require a particular
file, pass zebrad --config /path/in/container/zebrad.toml start after the
image name instead.
Include zebrad before CLI options or subcommands in Docker commands. The
entrypoint removes its first argument when that argument starts with -, so
passing a bare --config or --help after the image name loses that option.
Entrypoint environment variables
The entrypoint interprets these variables; other settings go to Zebra’s configuration loader.
| Variable | Default or behavior |
|---|---|
HOME | Image default /home/zebra; supplies the fallback cache and cookie paths. |
UID | Image default 10001; target owner and execution UID. Changing it at runtime does not create a corresponding account for setpriv --init-groups. |
GID | Image default 10001; target owner and execution GID. Prefer changing account IDs at build time. |
ZEBRA_STATE__CACHE_DIR | When unset or empty, directory preparation uses $HOME/.cache/zebra. |
ZEBRA_RPC__COOKIE_DIR | When unset or empty, directory preparation uses $HOME/.cache/zebra, independently of the state-cache setting. |
ZEBRA_TRACING__LOG_FILE | Unset by default. When nonempty, prepares the file’s parent directory. |
ZEBRA_ZCASHD_COMPAT__ZCASHD_DATADIR | Unset by default. When nonempty, creates the directory and changes only its top-level ownership, not the existing database tree. A failed ownership change warns rather than aborting. |
CONFIG_FILE_PATH | Unset by default; see mounted configuration. |
ZCASHD_COMPAT_ENABLED | Ignored. Values true, TRUE, 1, yes, YES, on, and ON print a removal warning; other values have no effect. |
NEXTEST_PROFILE | Unset by default. In test mode, selects cargo nextest run instead of the supplied command, except for an explicit zebrad command. |
NEXTEST_FILTER | Unset by default. Adds --filter-expr in nextest mode; required for profiles ci-stateful and ci-e2e. |
FEATURES | Image default default-release-binaries; passed to nextest in test mode. Setting it on a runtime container cannot change compiled features. |
SHORT_SHA | Preserved in test images from the build argument. The entrypoint unsets an empty value in test mode to avoid unnecessary Cargo rebuilds. |
Use docker/.env for node configuration examples and docker/test.env for
test settings. ZEBRA_SECTION__KEY variables override values in mounted
configuration files; unknown ZEBRA_* configuration fields fail loading.
RPC
Enable RPC on the container’s network interface so Docker forwarding or other containers can reach it. Publish it only on host loopback, never on a public interface:
docker run -d \
--name zebra \
-p 8233:8233 \
-p 127.0.0.1:8232:8232 \
-e ZEBRA_RPC__LISTEN_ADDR=0.0.0.0:8232 \
-v zebrad-cache:/home/zebra/.cache/zebra \
zfnd/zebra:7.0.0
For clients on the same private Docker network, omit the RPC -p mapping
entirely and connect to Zebra’s service name and container port. Container
loopback (127.0.0.1) is not the host loopback or another container’s loopback.
Use cookie authentication; the container’s default cookie path is listed above.
Make an authenticated request inside the container without publishing RPC:
docker exec --user 10001:10001 zebra sh -c '
curl --fail --silent --show-error \
--user "$(cat /home/zebra/.cache/zebra/.cookie)" \
--header "Content-Type: application/json" \
--data '\''{"jsonrpc":"2.0","id":1,"method":"getblockchaininfo","params":[]}'\'' \
http://127.0.0.1:8232
'
For a client in another container, mount a dedicated ZEBRA_RPC__COOKIE_DIR
read-only and give its UID permission to read the cookie. This avoids sharing
the state database. Share the directory, not a copied credential, so the
client can reload the cookie after Zebra restarts.
Set ZEBRA_RPC__ENABLE_COOKIE_AUTH=false only for a client that cannot use
cookie authentication, on an isolated network with trusted clients. Cookie
authentication is not a reason to publish RPC publicly.
Health endpoints
The runtime image includes curl but no Docker HEALTHCHECK. To enable
health checks, add these fields to the Zebra service in a Compose file:
environment:
- ZEBRA_HEALTH__LISTEN_ADDR=127.0.0.1:8080
healthcheck:
test: ["CMD", "curl", "--fail", "--silent", "http://127.0.0.1:8080/healthy"]
interval: 30s
timeout: 10s
retries: 3
start_period: 90s
Use /ready to gate dependent traffic, not to require a new node
to finish synchronizing during startup. Health checks need no RPC or cookie.
For probes outside the container, bind to 0.0.0.0:8080 and restrict exposure
to the probe network.
P2P networking
Without a published P2P port, a bridge-networked container synchronizes through outbound connections but cannot receive unsolicited connections from outside its Docker network.
When a firewall, network address translation (NAT), or load balancer changes
the externally reachable address, configure [network] external_addr with
the public IP and mapped port. For example, add
-e ZEBRA_NETWORK__EXTERNAL_ADDR=203.0.113.42:8233, replacing the documentation
address with your node’s actual address. Open the corresponding P2P port in
the firewall. Publishing a port does not enable its listener.
Custom images
Build from the repository root using the multi-stage Dockerfile:
git clone --depth 1 --branch v7.0.0 https://github.com/ZcashFoundation/zebra.git
cd zebra
docker build --file docker/Dockerfile --target runtime --tag zebra:local .
The build uses BuildKit bind and cache mounts. Select runtime for a node;
other stages serve the build and test pipeline:
| Target | Contents |
|---|---|
deps | Rust toolchain and build dependencies on Debian Trixie. |
tests | Source tree, Cargo, nextest, prebuilt release tests, dynamically linked RocksDB, and the lightwalletd v0.5.3 executable; defaults to entrypoint.sh test cargo test. |
release | Builds only the zebrad release executable for the runtime stage. |
runtime | Debian 13 (trixie-slim), zebrad, curl, and the entrypoint; the final/default stage. |
Building with custom features
The FEATURES build argument adds Cargo features.
The Dockerfile does not pass --no-default-features, so an empty argument does
not disable the default feature set.
For example, add spend indexing with the indexer feature:
docker build \
--file docker/Dockerfile \
--target runtime \
--build-arg FEATURES="default-release-binaries indexer" \
--tag zebra:custom-features \
.
The Dockerfile declares these build arguments:
| Argument | Default | Purpose |
|---|---|---|
FEATURES | default-release-binaries | Features passed to Cargo. |
RUST_VERSION | 1.91.0 | Rust base-image tag; the FROM instruction also pins a digest, which must match any replacement toolchain image. |
UID | 10001 | Created user’s numeric ID. |
GID | The UID argument | Created group’s numeric ID. |
USER | zebra | Account name. |
HOME | /home/${USER} | Home and working directory. |
CARGO_HOME | ${HOME}/.cargo | Cargo dependency and tool cache. |
CARGO_TARGET_DIR | ${HOME}/target | Build artifacts. |
CARGO_INCREMENTAL | Unset; the build sets 0 | Cargo incremental compilation. |
SOURCE_DATE_EPOCH | 0 | Build timestamp input; publishing workflows supply the source commit timestamp. |
SHORT_SHA | Unset | Commit identifier embedded in the build; retained at runtime in test images. |
SKIP_IPV6_TESTS | Unset; the test image sets 1 | Test-image environment setting. |
BUILDPLATFORM | Supplied by BuildKit | Selects the nextest download architecture in the test stage. |
To use different filesystem IDs, build with both --build-arg UID=... and
--build-arg GID=.... Update mount permissions and any hard-coded Compose paths
when changing the account or home directory.
zcashd-compat image removal
Zebra 7.0.0 removes the runtime-zcashd-compat Docker target and its
ZCASHD_COMPAT_ARCHIVE_URL and ZCASHD_COMPAT_ARCHIVE_SHA256 build arguments.
The runtime image no longer bundles zcashd or zcash-cli.
Use zcashd-compat mode with a separately supplied zcashd process and configuration; the image no longer fills in sidecar configuration.
Compose deployments
Run these commands from the repository root. Before running examples concurrently, give each unique container names, port mappings, and state volumes.
| File | Purpose and differences |
|---|---|
docker/docker-compose.yml | Single Mainnet node using zfnd/zebra (latest), docker/.env, the mounted docker/default-zebra-config.toml, and a persistent cache volume. Publishes P2P port 8233; pins linux/amd64. |
docker/docker-compose.lwd.yml | Zebra plus lightwalletd, with separate state volumes. Enables unauthenticated RPC and plaintext lightwalletd endpoints; pins both services to linux/amd64. Requires the network restrictions below. |
docker/docker-compose.observability.yml | Builds the runtime image locally and adds Prometheus, Grafana, Jaeger, and Alertmanager. Enables metrics, tracing, and unauthenticated RPC. Requires the network restrictions below. |
docker/docker-compose.test.yml | Builds the tests target, loads docker/test.env, and mounts Zebra and lightwalletd caches. |
docker/mining/docker-compose.yml | Mining-pool example with Zebra, Redis, S-NOMP, and optional nheqminer under the miner profile. Defaults to Testnet and keeps RPC and Zebra’s health listener internal. |
For the base node, pin the service’s image to zfnd/zebra:7.0.0 if a moving
latest tag is not appropriate. Remove or change its platform: linux/amd64
setting to select the ARM64 runtime image on an ARM64 host. Then start it:
docker compose -f docker/docker-compose.yml up -d
To switch the base Compose file to Testnet, change the network, its explicitly configured listen address, and the published P2P port.
Running Zebra with lightwalletd
The supplied example disables Zebra’s cookie authentication but does not
publish RPC to the host. Lightwalletd connects to zebra:8232 over the Compose
network. Keep that network limited to trusted containers.
The example also runs lightwalletd with --no-tls-very-insecure. For local
testing, restrict its port mappings to 127.0.0.1:9067:9067 (gRPC) and
127.0.0.1:9068:9068 (HTTP). Do not expose these plaintext endpoints as a
production service. Configure TLS and access controls for an externally
accessible deployment; see lightwalletd.
After making these changes, start the example:
docker compose -f docker/docker-compose.lwd.yml up -d
The dependency condition waits for the Zebra container to start, not for
Zebra to synchronize. This file does not load docker/.env; put overrides
in its service environment settings or explicitly add an env_file.
Running Zebra with Prometheus and Grafana
The observability example exports metrics to Prometheus and traces to Jaeger.
It sets ZEBRA_TRACING__OPENTELEMETRY_ENDPOINT=http://jaeger:4318, service name
zebra, and sample percentage 100. Reduce
ZEBRA_TRACING__OPENTELEMETRY_SAMPLE_PERCENT when full trace collection is not
required; see OpenTelemetry export.
Zebra’s RPC and metrics ports are not published to the host; Prometheus
scrapes zebra:9999 over the Compose network. Restrict the remaining non-P2P
published ports to host loopback by prefixing each mapping with 127.0.0.1:.
These include the monitoring interfaces, Jaeger’s receivers, and Jaeger’s
span-metrics endpoint. For example, replace 9094:9090 with
127.0.0.1:9094:9090. Keep RPC disabled altogether if no trusted client needs
it by removing ZEBRA_RPC__LISTEN_ADDR.
Export GF_SECURITY_ADMIN_PASSWORD with a strong, unique password in the
shell running Compose. An unset or empty value prevents startup. Then build
and start the stack:
docker compose -f docker/docker-compose.observability.yml up -d --build
| Service | Host endpoint after loopback restriction |
|---|---|
| Grafana | http://127.0.0.1:3000 |
| Prometheus | http://127.0.0.1:9094 (container port 9090) |
| Jaeger UI | http://127.0.0.1:16686 |
| Alertmanager | http://127.0.0.1:9093 |
| Jaeger trace receivers | gRPC on 127.0.0.1:4317; HTTP on 127.0.0.1:4318. |
| Jaeger span metrics | http://127.0.0.1:8889/metrics |
Grafana provisions the supplied dashboards and Prometheus/Jaeger data
sources. Log in as admin with the password supplied through
GF_SECURITY_ADMIN_PASSWORD. This initializes new Grafana volumes only;
change existing account passwords in Grafana. Prometheus’s admin and
lifecycle APIs are disabled. The supplied Jaeger configuration stores traces
in memory, capped at 100000 traces. Alertmanager defines receiver names but
no notification destinations; configure receivers before relying on alerts.
The configuration and dashboard files are under
docker/observability/.
Running CI tests locally
The test image contains the source tree and nextest. Configure test selection
in docker/test.env, or pass overrides when starting the container. Select
the ci profile to run the current unit and integration test set:
docker compose -f docker/docker-compose.test.yml build
docker compose -f docker/docker-compose.test.yml run --rm \
-e NEXTEST_PROFILE=ci zebra
The entrypoint runs nextest with --locked --release, the image’s FEATURES,
and --run-ignored=all. The ci profile excludes stateful and end-to-end
tests through its default filter. Select ci-stateful or ci-e2e with a
nonempty NEXTEST_FILTER expression for those suites, and supply their
required cached state. See the
nextest profiles
and continuous integration.
Without NEXTEST_PROFILE, the test image executes its command, which defaults
to cargo test; this does not use the prebuilt release-test configuration.
The runtime image does not contain Cargo or test binaries.
Mining example limitations
The mining example requires a reward address in docker/mining/.env.
S-NOMP waits for Zebra’s internal /healthy endpoint before starting, but
mining still requires a synchronized node. See
Mining with Docker for configuration and limitations.
Tracing Zebra
Zebra records structured events and spans through Rust’s tracing library.
Events are individual log records; spans attach context to an operation and
its nested work. Zebra writes formatted logs to standard output unless a
log file is configured. Optional exporters send this instrumentation to
journald, Sentry, or an OpenTelemetry collector.
Configuration
To write plain-text logs to a file:
[tracing]
filter = "info,zebra_network=warn"
log_file = "/var/log/zebra/zebrad.log"
use_color = false
buffer_limit = 128000
Create a writable log directory owned by the Zebra account. If Zebra runs
with elevated permissions, prevent other users from modifying that directory
or its parents. Zebra appends to the file; it does not configure log rotation.
Set use_color = false for plain-text files: formatted-log color detection
uses standard output, even when log_file redirects the logs.
For progress-bar log paths, buffer limits, and color overrides, see the configuration reference.
Filters and compiled log levels
The filter uses tracing-subscriber directives. A bare level
sets the global threshold; target=level sets a threshold for a Rust target,
usually a module path. Separate directives with commas. Levels are error,
warn, info, debug, and trace; off disables a target.
For example, warn,zebrad=info,zebra_network=debug selects warnings globally,
information from zebrad, and debug events from zebra_network. Directives
can also select spans and their fields; see the syntax reference before using
field filters.
Override the configured filter with any of these command forms:
zebrad -c zebrad.toml --filters 'info,zebra_network=debug'
zebrad -c zebrad.toml --filters 'info,zebra_network=debug' start
zebrad -c zebrad.toml start 'info,zebra_network=debug'
The top-level --filters option applies with or without an explicit start.
Positional filters after start take precedence over --filters and are
joined with commas. An explicit filter takes precedence over the default
selected by -v.
The utility commands generate and tip-height use
fixed filters and disable flamegraph collection.
Default release builds compile out debug and trace instrumentation through
release_max_level_info; neither -v nor runtime filters can restore it.
Build from the repository with default features disabled to retain all levels:
cargo install --locked --path zebrad --no-default-features \
--features 'progress-bar,prometheus,sentry,opentelemetry'
For other compile-time ceilings, see Cargo features.
Dynamic tracing
Build with the filter-reload feature and configure a loopback listener:
[tracing]
endpoint_addr = "127.0.0.1:3000"
Read or replace the running filter:
curl --fail http://127.0.0.1:3000/filter
curl --fail --data 'info,zebra_network=debug' http://127.0.0.1:3000/filter
The server accepts GET /filter and POST /filter; the POST body is the new
UTF-8 filter string. A reload changes the running subscriber, not the
configuration file. The endpoint has no authentication. Keep it on loopback
or a restricted management network.
Filter reloading is disabled in builds using tokio-console with
tokio_unstable, even if the HTTP endpoint is enabled. See
Tokio console.
Journald logging
Build with the journald Cargo feature and set [tracing] use_journald = true
on Linux systems running systemd. Zebra adds a journald subscriber for events
and spans; formatted output to standard output or log_file remains enabled.
Failure to initialize the journald subscriber fails tracing initialization.
Setting the field without the Cargo feature logs a warning instead of
activating journald.
Flamegraphs
A flamegraph visualizes time spent in nested tracing spans, not samples of
every function on the CPU stack. Build with the flamegraph Cargo feature
and set a writable output path:
[tracing]
flamegraph = "/var/log/zebra/spans"
Zebra records folded span stacks in spans.folded and generates spans.svg
during its shutdown hook. An extension in the configured path is replaced by
these extensions. An abrupt process termination can prevent SVG generation.
Use a directory protected against modification by other users, and compile
in the span levels needed for the investigation.
OpenTelemetry export
OpenTelemetry
collects spans as traces for operation-level analysis. With the
opentelemetry feature, configure an endpoint:
[tracing]
opentelemetry_endpoint = "http://127.0.0.1:4318"
opentelemetry_service_name = "zebra"
opentelemetry_sample_percent = 10
Zebra exports batches using OTLP, the OpenTelemetry Protocol, over HTTP with
binary Protocol Buffers encoding. It appends /v1/traces unless that suffix
is already present. Use an OTLP HTTP receiver, not a gRPC endpoint. The
observability stack supplies Jaeger as a
receiver and trace viewer.
Set [tracing] opentelemetry_sample_percent to an integer from 0 through 100:
Zebra samples no traces at 0, all at 100, and uses trace-ID ratio sampling
between them. See the configuration reference for
environment fallbacks and their nonstandard integer-percentage syntax.
Sampling cannot recover instrumentation removed by compile-time level features or excluded by the tracing filter. Zebra continues without trace export if exporter initialization fails.
Sentry production monitoring
Sentry receives error reports, diagnostic logs, and
release health sessions. With the sentry feature,
set SENTRY_DSN to a valid project ingestion address to enable reporting.
Leave it unset to disable reporting.
With Sentry enabled, Zebra sends:
- Error events as issues and Sentry Logs; warnings as Logs and breadcrumbs. Breadcrumbs are contextual records attached to later issues.
- Sync progress logs with a 60-second reporting interval, and a log when the best chain tip changes network upgrade. Other information-level events become breadcrumbs rather than standalone Logs. Runtime filters still apply.
- Panics as fatal, unhandled exceptions, with the full panic report and source location as context.
- Release health sessions, rotated every 24 hours. Graceful shutdown ends a session as exited; a panic, crash, or forced shutdown ends it as crashed. Sessions share a random identifier per run, so crash-free users measures runs, not people. A process killed before shutdown handling cannot send its final session update.
Events carry chain.network and, once the progress task observes a chain tip,
chain.height and chain.network_upgrade. Logs carry the same information
as attributes; the height is numeric for aggregation.
| Environment variable | Effect |
|---|---|
SENTRY_ENVIRONMENT | Sets the environment attached to reports and sessions. |
SENTRY_RELEASE | Overrides the release name; otherwise Zebra uses the build version printed by zebrad --version. |
Review the destination and access controls before enabling reporting: error context and panic reports leave the node. For deployment metadata, see GCP deployment operations; for event selection, see the Sentry integration.
Zebra metrics
Zebra exposes counters, gauges, and timing distributions through a Prometheus HTTP exporter. Prometheus stores the samples; Grafana queries Prometheus to render dashboards. Metrics complement health checks: a successful scrape does not establish that the node is ready to serve clients.
Enable the exporter
Use a build with the prometheus feature.
If no configuration file exists, create one:
zebrad generate -o zebrad.toml
In zebrad.toml, set [metrics] endpoint_addr:
[metrics]
endpoint_addr = "127.0.0.1:9999"
Start Zebra with that configuration and retrieve the metrics:
zebrad -c zebrad.toml start
From another terminal:
curl --fail http://127.0.0.1:9999/metrics
The exporter provides no authentication or TLS configuration. Bind it to loopback when Prometheus runs on the same host. For remote scrapes, bind to a private interface and restrict access with a firewall or authenticated proxy. Failure to bind the configured address terminates startup.
For Prometheus running on the same host, use:
global:
scrape_interval: 15s
scrape_configs:
- job_name: zebra
metrics_path: /metrics
static_configs:
- targets: ["127.0.0.1:9999"]
For container deployments, use an address reachable from Prometheus; the
supplied Compose stack uses zebra:9999.
Configure Grafana’s Prometheus data source with the Prometheus server URL,
not Zebra’s exporter URL. For a same-host deployment without containers, the
usual Prometheus URL is http://127.0.0.1:9090. The Compose stack provisions
its own data source automatically.
Metric names and types
Zebra’s source uses dot-separated names, such as
sync.estimated_distance_to_tip. The exporter replaces dots with underscores,
so Prometheus queries use sync_estimated_distance_to_tip. Counter names are
not automatically given a _total suffix: use the exported name as written.
Counters accumulate during a process run; use rate() or increase() for
activity over time. Gauges represent a current or most recently measured
value. A metric can be absent until its recording path executes; absence is
not equivalent to zero.
Zebra exports timing distributions as Prometheus summaries, with a
quantile label and _sum and _count series, not _bucket series:
rpc_request_duration_seconds{quantile="0.99"}
Do not apply histogram_quantile() to these Zebra summaries or average their
quantiles across nodes. For a mean duration over five minutes, divide the rate
of _sum by the rate of _count. Metrics generated separately by Jaeger’s
span-metrics connector have their own types and naming.
Principal operator metrics
The table uses source names; replace dots with underscores in Prometheus. Suggested alert conditions are operational choices, not Zebra defaults. Choose sustained thresholds that distinguish initial sync from steady-state operation and accommodate the network’s block production intervals.
| Metric | Exported type | Meaning and alert use |
|---|---|---|
zcash.chain.verified.block.height | Gauge | Best verified chain height, updated as blocks commit. Alert on sustained lack of change when blocks are expected. |
sync.estimated_distance_to_tip | Gauge | Estimated blocks behind the network tip, clamped to zero. Alert on sustained lag after initial sync; this is an estimate, not a peer-agreed height. |
sync.estimated_network_tip_height | Gauge | Estimated network tip height used to interpret the lag gauge. |
zcash.net.peers | Gauge, network label | Peer-set size, including ready and busy peers. Alert on zero peers or sustained low connectivity. |
candidate_set.recently_live | Gauge, network label | Address-book peers that have responded recently. Use with the peer-set size to diagnose connectivity. |
zcash.net.connection.outcomes.total | Counter | TCP connection and handshake outcomes by network, direction, address_family, stage, and outcome. Alert on sustained increases in failed outcomes relative to successes. |
zebra.net.in.errors | Counter, error label | Incoming message decoding errors. Investigate sustained increases by error category. |
sync.verified.block.count | Counter | Successfully verified blocks in the sync download pipeline. Use its rate to track sync throughput. |
state.checkpoint.error.block.count | Counter | Failed checkpoint-block commits. Investigate increases with the node logs. |
rpc.requests.total | Counter, method and status labels | Completed RPC requests, separated into success and error. |
rpc.errors.total | Counter, method and error_code labels | RPC errors; compare their rate with the request rate for the same method. |
rpc.request.duration_seconds | Summary, method label | RPC latency in seconds. Alert on sustained latency for methods whose response-time budget is known. |
rpc.active_requests | Gauge | RPC calls currently being processed. Track sustained concurrency alongside latency. |
mempool.failed.verify.tasks.total | Counter, reason label | Mempool download or verification task failures. Distinguish invalid transactions from state or download failures. |
zcash.mempool.cost.bytes | Gauge | Aggregate mempool eviction cost, in cost bytes rather than process memory bytes. Compare with [mempool] tx_cost_limit. |
zebra.state.rocksdb.total_disk_size_bytes | Gauge | Sum of RocksDB’s total-sst-files-size properties across column families, in bytes. Track growth, but monitor filesystem free space separately. |
zebra.state.rocksdb.batch_commit.duration_seconds | Summary | RocksDB batch-write latency in seconds. Use sustained increases to diagnose storage bottlenecks. |
Prometheus also generates up{job="zebra"} for scrape success. Alert on zero
or missing scrape data separately from chain progress. This metric comes
from Prometheus, not Zebra.
Network metric aggregation
Network metrics omit per-peer labels to bound the number of time series. Update dashboards, recording rules, and joins that depend on peer identities:
- Connection-state, request, response, message, and byte metrics have no
addrlabel. Thezebra.net.connection.stategauge counts live connections bycommand; transitions and disconnects decrement the old state. Request, response, and message counters retaincommandlabels. - The counters
zcash.net.peers.connected,zcash.net.peers.obsolete, andzcash.net.peers.missing_serviceshave noremote_iporuser_agentlabels. They count handshake activity, not currently connected peers. - The gauges
zcash.net.peers.version.connectedandzcash.net.peers.version.obsoletecontain the remote protocol version from the most recent corresponding handshake, withoutremote_iplabels. They are not peer counts or a distribution of connected versions. - The
zebra.net.in.errorscounter has noaddrlabel.
The counter zcash.net.peer.version.outcomes.total provides a software
breakdown through the bounded implementation values zebra, zakura,
legacy_zcashd, and other. Select outcome="success" for successful
handshakes. These remain cumulative handshake counts, not current peer counts.
Other bounded labels include:
| Metric or label | Values |
|---|---|
mempool.failed.verify.tasks.total, reason | in_state, state_error, download_failed, cancelled, invalid, timeout |
zebra.net.in.errors, error | io, parse, utf8_error, try_from_slice_error, try_from_int_error, from_hex_error, amount, bad_transaction_balance, script, opcode, num |
Handshake remote_version and negotiated_version | Protocol versions defined by Zebra, or other; arbitrary received numbers do not create new labels. |
RPC method | Registered RPC method names, or unknown for unregistered names. |
Mining metrics
Prometheus replaces dots in metric names with underscores. The RPC dashboard includes mining
outcomes and mean build and announcement durations.
Zebra exports duration metrics as Prometheus summaries, with quantiles plus _sum and _count series.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
rpc.submitblock.outcomes.total | Counter | outcome | Submission results: accepted, duplicate, duplicate-inconclusive, inconclusive, or rejected. |
rpc.getblocktemplate.build.duration_seconds | Summary | None | Completed template construction: transaction selection, coinbase construction, and block roots, in seconds. |
rpc.getblocktemplate.first_template.duration_seconds | Summary | None | Seconds from best-tip publication to the first template served with that tip as its parent. |
sync.mined.block.announcement.outcomes.total | Counter | outcome | Mined-block announcement attempts: completed or failed. |
sync.mined.block.announcement.duration_seconds | Summary | None | Seconds from gossip receiving an accepted mined block to successful completion of its announcement’s local writes. |
Submission labels follow the zcashd-compatible result, with accepted representing JSON null.
Operational JSON-RPC errors remain in rpc.errors.total; they are not consensus rejections.
Some result variants are reserved by the response type and are not currently returned.
Build timing covers the shared constructor used by background and fallback builders, not state or mempool reads, executor scheduling, long-poll waits, or cache hits. It records each completed build, including work superseded before a miner requests it.
First-template latency uses the state’s monotonic publication time, not the block header timestamp or the time the RPC notices a change. It records once per tip transition across RPC clones; repeated requests do not add samples. Tips with no served template and templates on uncommitted candidates are excluded. Returning to an earlier tip after a reorg starts a new interval. At startup, timing begins when the state publishes its loaded tip. The measurement ends when the RPC returns the template, before network delivery or pool job distribution; it also includes time until a miner requests work.
Announcement completion means every selected connection completed its local write, not that peers received, validated, or relayed the block. Failed attempts include write errors and timeouts; a retry records another outcome. Successful retry durations retain the original gossip receipt time, including queueing, peer-readiness waits, and retry delays. Ordinary tip and early-relay adverts are excluded. No peer count is exported because the broadcast response does not carry one.
Watch increases in rejected submissions and failed announcements alongside rising announcement durations. A successful local write is not proof against stale blocks, and a stalled attempt has no completion sample yet. These metrics do not measure network-wide propagation or stale-block rates.
Observability stack
The Compose stack builds Zebra and starts Prometheus,
Grafana, Jaeger, and Alertmanager using configuration in
docker/observability/. It persists Zebra’s state, Prometheus,
Grafana, and Alertmanager in volumes.
The supplied stack is a development deployment, not a hardened public service. Zebra’s RPC and metrics ports stay on the Compose network, and RPC requires cookie authentication. The remaining port mappings bind on all host interfaces; restrict published management ports to loopback or a private network before starting it. Prometheus’s admin and lifecycle APIs are disabled.
Export GF_SECURITY_ADMIN_PASSWORD with a strong, unique password; Compose
rejects an unset or empty value. Run from the repository root:
docker compose -f docker/docker-compose.observability.yml up -d
docker compose -f docker/docker-compose.observability.yml logs -f zebra
| Service | Host URL |
|---|---|
| Grafana | http://localhost:3000 |
| Prometheus | http://localhost:9094 |
| Jaeger | http://localhost:16686 |
| Alertmanager | http://localhost:9093 |
Log in to Grafana as admin with the password supplied through
GF_SECURITY_ADMIN_PASSWORD. It initializes new Grafana volumes only; change
existing account passwords in Grafana. The stack provisions the Prometheus
and Jaeger data sources and the dashboards in
docker/observability/grafana/dashboards/. The home dashboard is
zebra_overview.json; other dashboards include peers.json,
network_health.json, syncer.json, mempool.json, rocksdb.json,
rpc_metrics.json, rpc_tracing.json, and value_pools.json. For a separate
Grafana deployment, import the JSON files and select its Prometheus data
source.
The file docker/observability/prometheus/prometheus.yaml scrapes
zebra:9999 and Jaeger’s span metrics at jaeger:8889, every 15 seconds.
Zebra exports traces to http://jaeger:4318 with 100% sampling in this stack.
Reduce [tracing] opentelemetry_sample_percent through the Compose environment
when lower trace volume is required; see OpenTelemetry export.
Review docker/observability/prometheus/rules/zebra_alerts.yml before relying
on its alerts. Its latency rules use exported summary quantiles: RPC p99 above
two seconds and successful-handshake p95 above two seconds, each sustained for
five minutes. Zebra abandons handshakes after three seconds, so slower ones
never reach the handshake series. The RPC latency panels show per-method
quantiles, not a combined percentile across methods or nodes.
The receivers in docker/observability/alertmanager/alertmanager.yml have no
notification destinations; configure them before expecting notifications.
Stop the stack without deleting its volumes:
docker compose -f docker/docker-compose.observability.yml down
Zebra health endpoints
Zebra’s HTTP/1.1 health listener provides liveness and readiness probes. Liveness checks for recently responsive peers; readiness also checks sync progress before a load balancer routes client traffic to the node.
Enable it with [health] listen_addr; no optional Cargo feature is needed:
[health]
listen_addr = "127.0.0.1:8080"
min_connected_peers = 1
ready_max_blocks_behind = 2
ready_max_tip_age = "5m"
enforce_on_test_networks = false
Configuration
See the configuration reference for fields and defaults.
The field [health] ready_max_tip_age measures local progress, not the latest
block header’s timestamp. The progress task samples the chain about every five
seconds; only a height above its previous maximum resets the timer.
The health server refreshes its peer count from the address book every five seconds. Peers count only while marked as having responded within the network’s response recency window; failed or pending connections do not count. This is not a count of open TCP sockets.
Endpoints
All responses use Content-Type: text/plain; charset=utf-8.
Liveness: GET /healthy
| HTTP status | Body | Condition |
|---|---|---|
200 OK | ok | Recently live peer count is at least min_connected_peers. |
503 Service Unavailable | insufficient peers | Recently live peer count is below the threshold. |
This endpoint does not require a chain tip or a completed initial sync.
It applies the same peer check on every network, including Regtest. Setting
min_connected_peers = 0 makes it succeed whenever the listener responds,
but also removes the peer requirement from readiness.
Readiness: GET /ready
On Mainnet, or when enforce_on_test_networks = true, Zebra evaluates these
checks in order and returns the first failure:
| HTTP status | Body | Condition |
|---|---|---|
503 Service Unavailable | insufficient peers | Recently live peer count is below min_connected_peers. |
503 Service Unavailable | syncing | The synchronizer’s recent sync lengths do not indicate proximity to the tip. |
503 Service Unavailable | no tip | The progress task has no estimated remaining block count. |
503 Service Unavailable | tip_age=<seconds>s | Time since the last observed height increase exceeds ready_max_tip_age. |
503 Service Unavailable | lag=<blocks> blocks | Estimated remaining blocks exceed ready_max_blocks_behind. |
200 OK | ok | All checks pass. |
The synchronizer’s near-tip signal uses the moving average of recent sync lengths; it is not the same as the estimated block lag. Readiness requires both checks. Negative lag estimates are clamped to zero by the progress task. Equality with either configured maximum passes the corresponding check.
On test networks, the default enforce_on_test_networks = false bypasses
all readiness checks: /ready returns 200 OK even with no peers or
chain tip. Set it to true before using readiness to gate test-network client
traffic. Regtest’s synchronizer always reports near-tip status, but enabling
enforcement still applies the peer, tip, age, and lag checks. In a Regtest
network that mines only on demand, the tip-age check can fail between blocks.
Other requests
Requests with a method other than GET, including HEAD, return
405 Method Not Allowed. The body is method not allowed, except that
HTTP responses to HEAD have no body. Unknown paths requested with GET
return 404 Not Found with not found.
Kubernetes and load balancers
Bind [health] listen_addr = "0.0.0.0:8080" inside a pod when the kubelet
must reach the listener through the pod IP. Restrict that port with network
policies rather than publishing it as a public service.
Use /ready as the readiness probe. Keep the pod out of service during
initial sync instead of restarting it for lack of readiness. A TCP liveness
probe avoids restarting a functioning node during a peer outage:
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
Use httpGet with path: /healthy for liveness only when insufficient peers
should trigger a restart. Allow enough startup and failure time for peer
discovery and temporary network outages; restarting cannot repair an
unreachable external network.
Route load-balancer traffic only after /ready returns 200; remove the node
on 503. Probe RPC separately: neither health endpoint checks the JSON-RPC
listener or executes a client-facing method.
Security
The health listener has no authentication or TLS. It returns coarse node
state, including lag or time since progress on a failed readiness check.
Bind to loopback or an internal interface, and restrict access with firewall
rules or network policies. Failure to bind listen_addr terminates startup.
JSON-RPC server
Zebra serves chain data, transaction submission, and mining operations over
HTTP JSON-RPC. Zebra does not hold wallet keys or implement wallet accounting;
Zallet provides the wallet API. The
zcashd API migration guide records differences
from zcashd, including missing fields and unsupported methods.
Enabling the server
JSON-RPC is disabled until [rpc] listen_addr is set to an
explicit IP address and port. See Ports for network conventions.
Generate a configuration file:
zebrad generate -o zebrad.toml
Add the listener to its existing [rpc] section:
[rpc]
listen_addr = "127.0.0.1:8232"
Start Zebra with that file:
zebrad -c zebrad.toml start
Keep the listener on loopback. Zebra provides neither TLS encryption nor
per-method access control. A client with RPC access can submit transactions
and blocks and call state-changing methods such as invalidateblock.
Cookie authentication authenticates the caller; it does not encrypt traffic.
For remote access, use an authenticated, TLS-terminating reverse proxy or an
encrypted tunnel, and prevent clients from reaching the plaintext listener
directly. Do not publish an unauthenticated RPC port to the internet.
Authentication
The [rpc] enable_cookie_auth field defaults to true. At JSON-RPC server
startup, Zebra generates a random credential and writes
<cookie_dir>/.cookie in this format:
__cookie__:<base64-encoded random secret>
The username is __cookie__; the secret encodes 32 random bytes. Zebra creates
new cookie files with mode 0600 on Unix and refuses to write through a
symlink. Restrict access to the cookie directory to the node operator and
trusted clients. Treat the file as a password, not as an HTTP browser cookie.
Clients send its contents using HTTP Basic authentication and must read the
new credential after a Zebra restart.
On clean RPC shutdown, Zebra removes the cookie file; removal failures are
logged.
The [rpc] cookie_dir setting is independent of
[state] cache_dir: changing the state directory does not move the cookie.
Give each Zebra instance a separate cookie directory, even on different networks.
For the Linux default cookie directory, call the server with:
cookie="${XDG_CACHE_HOME:-$HOME/.cache}/zebra/.cookie"
curl --user "$(cat "$cookie")" \
--header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":1,"method":"getblockchaininfo","params":[]}' \
http://127.0.0.1:8232/
Set cookie to the configured directory’s .cookie file when using a custom
[rpc] cookie_dir.
Zebra has no configuration for static RPC usernames and passwords. Setting
[rpc] enable_cookie_auth = false disables credential checks; credentials
sent by clients then have no effect. This is required by
lightwalletd, which does not read Zebra’s cookie.
Authentication settings apply only to JSON-RPC, not to either gRPC listener.
Configuration
See the configuration reference for [rpc] fields and defaults.
The experimental lightwalletd-compatible gRPC service is covered in
Serving light clients directly from Zebra.
Zebra has no [rpc] CORS, allowed-origin, client-IP allowlist, or batch-count
configuration. Cross-origin browser restrictions are not a replacement for
RPC authentication or network access controls.
Indexer gRPC service
The indexer service is separate from JSON-RPC and from the wallet-facing
CompactTxStreamer protocol. Its
protocol definition
provides four methods:
| Method | Purpose |
|---|---|
ChainTipChange | Streams best-chain tip hashes and heights. |
NonFinalizedStateChange | Streams non-finalized blocks after the caller’s known tips. |
MempoolChange | Streams transaction additions, invalidations, and mining events. |
GetBlock | Returns a best-chain block by hash or height. |
Indexers use these streams alongside read-only access to Zebra’s finalized
state database. Build with the indexer feature
for consumers that require the spending-transaction index, including
Zallet’s Zebra backend. The listener does not require this feature:
successful startup does not establish that the index exists.
Set [rpc] indexer_listen_addr = "127.0.0.1:8230" to enable the service on an
example port. It uses plaintext HTTP/2 without authentication; JSON-RPC cookie
settings do not protect it. Keep it on loopback or a restricted private network.
Transport and compatibility
Send HTTP POST requests with Content-Type: application/json. Zebra accepts
JSON-RPC 2.0 and the legacy request format used by zcashd. For individual
legacy requests, include method, params, and id, and omit jsonrpc.
Zebra also accepts lightwalletd’s "jsonrpc":"1.0" field. The compatibility
middleware converts the response back to the corresponding legacy format,
including both result and error. Missing content types and text/plain
are treated as JSON. The server does not offer WebSocket transport.
Use JSON-RPC 2.0 for batches. Send an array of request objects, each with its
own "jsonrpc":"2.0" and id. The legacy compatibility conversion applies
to individual objects, not to an entire batch. Batches have no separate
request-count limit, but the request body is limited to 4,001,024 bytes and
the response body to [rpc] max_response_body_size.
For example, using the cookie variable from Authentication:
curl --user "$(cat "$cookie")" \
--header 'Content-Type: application/json' \
--data '[{"jsonrpc":"2.0","id":1,"method":"getblockcount","params":[]},{"jsonrpc":"2.0","id":2,"method":"getbestblockhash","params":[]}]' \
http://127.0.0.1:8232/
Zebra uses zcashd-style application error codes, including -5 for an
unknown block hash and -8 for an invalid block height. The RPC middleware
rewrites invalid-parameter code -32602 to -1 while preserving the error
message, data, and request ID. Other error codes pass through unchanged;
this is not a guarantee that every failure matches zcashd byte for byte.
Methods
Zebra 7.0.0 registers the following 42 JSON-RPC method names. The
RPC trait and implementation
define their parameters and results. The rpc.discover method returns an
OpenRPC 1.3.2 document with machine-readable method and type schemas.
Control and discovery
| Method | Semantics |
|---|---|
getinfo | Returns node software, chain, connection, and status information. |
getdeprecationinfo | Returns end-of-support information, with an estimated halt height on Mainnet. |
rpc.discover | Returns the server’s OpenRPC schema. |
stop | Requests process shutdown on Regtest; unavailable on other networks and on Windows. |
Blockchain
| Method | Semantics |
|---|---|
getblockchaininfo | Returns chain height, network upgrades, value pools, and synchronization estimates. |
getbestblockhash | Returns the best-chain tip hash. |
getbestblockheightandhash | Returns the best-chain tip height and hash together. |
getblockcount | Returns the best-chain tip height. |
getblockhash | Returns the block hash at a height; negative indices count back from the tip. |
getblock | Returns a block by hash or height as hexadecimal data or a structured response with verbosity 1, 2, or 3. |
getblockheader | Returns a block header by hash or height as hexadecimal data or a structured response. |
getdifficulty | Returns proof-of-work difficulty relative to the minimum difficulty. |
gettxout | Returns an unspent transparent output, optionally accounting for the mempool. |
z_gettreestate | Returns Sapling, Orchard, and Ironwood shielded tree state at a block, according to pool activation. |
z_getsubtreesbyindex | Returns indexed subtree roots and completion heights for Sapling, Orchard, or Ironwood. |
invalidateblock | Invalidates a block in non-finalized state; cannot invalidate finalized blocks. |
reconsiderblock | Reconsiders a cached invalidated block and returns the reconsidered block hashes. |
preciousblock | Prefers a non-finalized tip over competing tips with equal cumulative work; the last call wins and the preference is lost on restart. |
Transactions and mempool
The mempool holds verified transactions that have not yet been mined.
| Method | Semantics |
|---|---|
getmempoolinfo | Returns aggregate mempool size and memory information. |
getrawmempool | Returns mempool transaction IDs, or transaction metadata with verbose output. |
getrawtransaction | Returns a transaction as hexadecimal data or a structured response, optionally restricted to a block. |
sendrawtransaction | Submits a signed transaction to the mempool and returns its transaction ID; allowhighfees is accepted but ignored. |
getstandardfee | Returns the recommended fee per logical action in zatoshis, with version: 0. For the next block on Mainnet, reports 5,000 before height 3,590,000 and 1,000 from that height; test networks report 1,000 at every height. |
Transparent address index
| Method | Semantics |
|---|---|
getaddressbalance | Returns confirmed balance and received amount for transparent addresses. |
getaddresstxids | Returns transaction IDs involving transparent addresses within an optional height range. |
getaddressutxos | Returns unspent outputs for transparent addresses, with optional height, count, and chain-information fields. |
Mining
See Mining for configuration and block-template use.
| Method | Semantics |
|---|---|
getblocktemplate | Takes an optional request object; returns a mining template, supports long polling, or validates a proposed block. |
submitblock | Validates and submits a hexadecimal block; an optional work-ID object is ignored. Returns null, "duplicate", or "rejected", or an RPC error. See the mining guide for result caveats. |
getmininginfo | Takes no parameters; returns tip height, block size and non-coinbase transaction count when available, network name, and estimated network solution rate. |
getblocksubsidy | Returns miner subsidy, funding-stream and lockbox allocations excluding transaction fees at an optional height (default: tip); reserve-dependent future rewards require the preceding block. |
getnetworksolps | Estimates network solutions per second with optional block count and height (defaults: 120 blocks, tip). A nonpositive count selects the difficulty averaging window at the effective query height: 17 blocks before NU7, 102 from NU7. |
getnetworkhashps | Deprecated alias for getnetworksolps. |
generate | Generates blocks paying the configured miner address when proof of work is disabled, including Regtest. |
generatetoaddress | Generates blocks paying a supplied address when proof of work is disabled. |
Peer networking
| Method | Semantics |
|---|---|
getnetworkinfo | Returns network protocol, connection, and relay information. |
getpeerinfo | Returns the peer information tracked by Zebra’s address book. |
ping | Returns success without initiating a peer ping. |
addnode | Adds a peer address on Regtest; only the add command is supported. |
Address utilities
A Unified Address contains one or more payment receivers. A TEX address encodes a transparent receiver with a restriction on the source of its funds.
| Method | Semantics |
|---|---|
validateaddress | Validates a transparent address and returns its supported metadata. |
z_validateaddress | Validates a supported Zcash address and reports its address type and metadata. |
z_converttex | Converts a transparent P2PKH address to a TEX address. |
z_listunifiedreceivers | Extracts supported payment receivers from a Unified Address. |
Running Zallet with Zebra
Zallet provides wallet functions; Zebra validates the chain and serves node RPCs. Use the API migration guide to route calls and Zallet’s setup guide to initialize the wallet, encryption, and wallet RPC server.
Keep the versions in step
Zallet 0.1.0-beta.3 is the latest release. Its zebra backend reads Zebra’s
database directly and requires format 28
(backend changelog);
Zebra 7.0.0 writes format 29. No released zebra backend supports it.
Do not deploy the direct-reader configuration below until a compatible
backend is released. If its required format is absent, startup fails with
no zebra-state v… database found; changing endpoints does not fix it.
The beta.3 zaino backend is unaffected when
[indexer.read_state_service] is absent: it
fetches chain data over JSON-RPC
without reading Zebra’s database. That topology is untested here. The
pool payout RPC gaps apply to
either backend. For the tested alternative, see
pool payouts through the sidecar.
Check the backend’s release notes and zebra-state dependency before
upgrading Zebra. Zallet opens state read-only and cannot
upgrade it. Do not rename or symlink a versioned directory to bypass the
compatibility check.
What Zallet needs from Zebra
The zebra backend runs on Linux and needs three interfaces simultaneously:
| Interface | Purpose | Requirement |
|---|---|---|
| Read-only state database access | Reads finalized blocks, transactions, shielded trees, unspent outputs, and spending-transaction indexes. | Run on the same host, with a compatible zebra-state version and access to Zebra’s persistent state cache. |
| Indexer gRPC | Follows tip changes and non-finalized blocks, and fetches missing blocks. | Set [rpc] indexer_listen_addr; keep the endpoint private. |
| JSON-RPC | Calls getrawmempool, getrawtransaction, sendrawtransaction, and getblockchaininfo for mempool access, transactions, and the network-upgrade table. | Set [rpc] listen_addr and give Zallet the authentication cookie. |
The backend does not connect to Zebra’s wallet-facing
[rpc] lightwalletd_listen_addr service. That endpoint implements a different
protocol for light wallets.
Configuring Zebra
For a compatible direct-reader backend, build Zebra with the
indexer feature:
cargo install --locked --version 7.0.0 --features indexer zebrad
For a custom Docker build, pass
--build-arg FEATURES="default-release-binaries indexer".
Set the state path and both endpoints in zebrad.toml. Choose directories
that Zebra can write and Zallet can access:
[state]
cache_dir = "/srv/zebra/state"
[rpc]
listen_addr = "127.0.0.1:8232"
indexer_listen_addr = "127.0.0.1:8230"
cookie_dir = "/srv/zebra/rpc"
enable_cookie_auth = true
Start Zebra with that configuration:
zebrad -c zebrad.toml start
Give Zallet read access to /srv/zebra/rpc/.cookie without making the
credential public. Keep indexer gRPC on loopback:
it has no authentication or TLS, and JSON-RPC authentication does not protect
it. Port 8230 is an example, not a default.
Configuring Zallet
Set these fields in the Zallet configuration produced by its setup procedure.
The root backend key belongs before the first TOML section.
backend = "zebra"
[consensus]
network = "main"
[indexer]
validator_address = "127.0.0.1:8232"
validator_cookie_path = "/srv/zebra/rpc/.cookie"
[indexer.read_state_service]
grpc_address = "127.0.0.1:8230"
zebra_state_path = "/srv/zebra/state"
The [indexer] validator_address field is a host and port, without an
http:// prefix. The [indexer] validator_cookie_path field selects cookie
authentication; the backend reads the file again for each request, so it
picks up a new credential after Zebra restarts. Do not substitute static
validator_user and validator_password values for Zebra’s generated cookie.
The [indexer.read_state_service] zebra_state_path field points to the cache
root configured by Zebra’s [state] cache_dir, not to a network-specific
versioned subdirectory. Use an absolute path to avoid resolving a relative
path against Zallet’s data directory. Match Zallet’s consensus network to
Zebra’s network.
Zallet’s wallet RPC listener is separate from both of these backend connections. Assign it a different port; applications that need wallet methods must connect to Zallet, not to Zebra.
Index availability and restart behavior
Zebra starts [rpc] indexer_listen_addr even without the indexer feature.
A read-only consumer then receives empty results for missing indexes rather
than an initialization error
(#11155).
Keep the indexer feature enabled on the Zebra process that writes the state.
Starting a non-indexer build against an indexed database removes the
spending-transaction indexes. Restarting an indexer build rebuilds them from
the existing chain state; a wallet reader does not perform that rebuild.
Wait for Zebra’s finished checking/adding indexes for spending tx ids log
before relying on the indexes.
Every indexed startup checks or builds these indexes, scanning existing blocks even without a format change. Allow for the resulting CPU load when scheduling restarts (#11090).
Running lightwalletd with Zebra
lightwalletd serves compact blocks and transaction queries to light wallets over gRPC. Zebra validates the chain; lightwalletd reads Zebra’s JSON-RPC API and maintains its own compact-block cache. Wallets connect to lightwalletd, not to Zebra’s JSON-RPC port.
Zaino, a Rust indexer, serves the same
light-wallet CompactTxStreamer gRPC protocol but uses different backend and
server configuration. Follow its deployment instructions.
The Docker deployment runs Zebra and lightwalletd as separate services. Zebra also has an experimental direct gRPC service.
Configure Zebra for lightwalletd
Generate a Zebra configuration file:
zebrad generate -o zebrad.toml
Edit the existing sections rather than adding duplicate TOML tables.
JSON-RPC
Set these fields in zebrad.toml for a Mainnet backend:
[rpc]
listen_addr = "127.0.0.1:8232"
enable_cookie_auth = false
For Testnet, set [network] network = "Testnet" and use the conventional
RPC port, 127.0.0.1:18232.
Lightwalletd does not read Zebra’s authentication cookie, so this setup disables authentication. Keep the listener on loopback or an isolated container network; do not expose it to wallet users.
Lightwalletd does not require Zebra’s indexer Cargo feature or the
[rpc] indexer_listen_addr endpoint. It uses JSON-RPC rather than direct
access to Zebra’s database.
Sync Zebra
Start Zebra with the configured listener:
zebrad -c zebrad.toml start
Keep Zebra running while lightwalletd serves clients. Monitor synchronization, or start from a state snapshot. A successful RPC connection does not establish that Zebra has reached the tip.
Configure lightwalletd
Install the zcash/lightwalletd implementation using its
installation instructions.
Create its backend connection file, zcash.conf:
rpcbind=127.0.0.1
rpcport=8232
rpcuser=lightwalletd
rpcpassword=unused
Zebra does not read this file. Lightwalletd’s parser requires a nonempty
rpcpassword even though Zebra ignores it with authentication disabled;
an empty file is insufficient, and the sample password is not access control.
For Testnet, change rpcport to 18232. The parser uses testnet=1 only to
choose that port when rpcport is absent; it does not change Zebra’s network.
Restart lightwalletd when switching the backend network.
For containers, use the backend service hostname instead of loopback. The
repository’s Compose setup
connects lightwalletd to zebra:8232 without publishing RPC to the host. It
still publishes plaintext gRPC and HTTP; restrict access before using it
outside a trusted test environment.
Current upstream lightwalletd also accepts a Zebra .toml file through
--zcash-conf-path, reading [rpc] listen_addr, but not Zebra’s cookie.
Sync lightwalletd
For local testing, start lightwalletd with an explicit loopback gRPC listener:
mkdir -p "$HOME/.cache/lightwalletd"
lightwalletd \
--zcash-conf-path ./zcash.conf \
--data-dir "$HOME/.cache/lightwalletd" \
--grpc-bind-addr 127.0.0.1:9067 \
--http-bind-addr 127.0.0.1:9068 \
--no-tls-very-insecure \
--log-file /dev/stdout
Lightwalletd serves requests while filling its compact-block cache; uncached requests read from Zebra. A quiet log does not prove either service is synced.
Query the service with grpcurl:
grpcurl -plaintext 127.0.0.1:9067 \
cash.z.wallet.sdk.rpc.CompactTxStreamer/GetLightdInfo
At its default log level, lightwalletd enables gRPC reflection, so this command needs no local protocol files. The response reports the backend network, height, and version. Compare the height with Zebra’s sync status before connecting wallets.
Connect a wallet to lightwalletd
Configure the wallet’s server address to point at lightwalletd’s gRPC
listener. The local test example uses 127.0.0.1:9067; loopback on another
device refers to that device, not to the server.
For a network-facing deployment, remove --no-tls-very-insecure and supply
--tls-cert and --tls-key with a certificate and private key trusted by the
wallet clients. Keep the backend JSON-RPC listener and lightwalletd’s HTTP
listener private. Follow lightwalletd’s
production deployment instructions
for certificate configuration.
Run tests
Zebra’s lightwalletd integration tests
launch a separate lightwalletd process. Install lightwalletd and protoc,
enable the lightwalletd-grpc-tests Cargo feature, and set
TEST_LIGHTWALLETD=1. These tests are ignored by default; select them using
the stateful test instructions.
Sync and wallet tests also need cached Zebra state. Tests that reuse
lightwalletd state accept LWD_CACHE_DIR; without it, the harness looks for
the platform cache directory, such as $HOME/.cache/lwd on Linux. Use
disposable test copies, not the directories of running production services.
Experimental: serving light clients directly from Zebra
Zebra’s experimental CompactTxStreamer service runs without lightwalletd.
Its behavior and configuration can change incompatibly in any release.
Enable it in Zebra’s [rpc] section:
[rpc]
lightwalletd_listen_addr = "127.0.0.1:9067"
The listener calls Zebra’s RPC handlers in-process and reads state and mempool
services directly. It needs no additional Cargo feature, JSON-RPC listener,
zcash.conf, or lightwalletd cache. Do not bind lightwalletd to the same
address and port.
This is not Zallet’s indexer gRPC protocol:
[rpc] indexer_listen_addr enables a different service.
The port is not protected
The direct gRPC listener serves plaintext HTTP/2 without TLS or authentication. JSON-RPC cookie authentication does not apply. Anyone who can reach it can query chain data and submit transactions; network observers can read or modify unencrypted traffic, including transparent-address queries.
Bind it to loopback or a restricted private interface. For remote wallets, put a reverse proxy with gRPC support in front of it, terminate TLS at the proxy, and block direct access to Zebra’s plaintext port. Configure proxy stream timeouts for long-lived block-range and mempool requests.
Do not run it on a user’s machine
Run this unauthenticated service on infrastructure you administer, not on end users’ computers. A loopback bind limits network reachability but does not isolate the service from untrusted local processes.
Compatibility limits
The direct service differs from lightwalletd:
- The
Pingmethod returns gRPCunimplemented. - The
GetLightdInforesponse leavesgit_commit,branch,build_date,build_user, anddonation_addressempty. - Compact blocks have an empty
headerfield.
Test wallet synchronization, transaction submission, and recovery across chain reorganizations with the intended client before deployment; a successful connection does not establish interoperability.
zcashd API migration guide
Route node, chain, mempool, and mining calls to Zebra; wallet calls to Zallet. Changing the RPC URL alone does not preserve parameters or responses.
The tables cover zcashd 6.20.0, Zebra 7.0.0, and
Zallet 0.1.0-beta.3. Zallet’s reference follows development;
the release-specific restrictions here take precedence. Its direct zebra
backend is incompatible with Zebra 7.0.0; see backend requirements
for the unaffected RPC-based zaino backend.
For pool integrations, see the payout RPC compatibility table and tested sidecar
procedure; ownership does not imply that
an unmodified payout processor works with Zallet.
Where the boundary falls today
| API requirement | Migration target |
|---|---|
| Chain, network, mempool, mining | Zebra, with the differences below |
| Wallet keys, balances, transaction construction | Zallet, with its altered RPC semantics |
| Existing wallet integration during migration | The restricted zcashd-compat sidecar |
How to read the table
Method
Method links point to the zcashd RPC reference.
The inventory includes hidden and wallet-encryption methods omitted there.
Owner
| Owner | Meaning |
|---|---|
| Zebra | Implements the node-level method or the named replacement. |
| Zallet | Implements the wallet method or records its disposition in the wallet method status. |
| — | No implemented target RPC; Notes records proposals or non-RPC alternatives. |
Rows name Zebra when wallet state is unnecessary, even if Zallet also serves
the method (getrawtransaction, validateaddress, z_listunifiedreceivers,
and z_converttex).
The sidecar is temporary, not an API owner. Its network and mempool methods describe its own state. Deprecation flags, index requirements, sync gates, and wallet restrictions still apply:
| Marker | Sidecar behavior |
|---|---|
| (none) | No special registration marker; other sidecar restrictions still apply. |
| † | Wallet transaction construction is subject to the shielded-pool restrictions. |
| * | The sidecar does not register the method; calls return -32601. |
Status
Status describes the named target, not the temporary sidecar.
| Status | Meaning |
|---|---|
| Done | Implemented; responses need not be byte-identical. |
| Partial | Implemented with restrictions or semantic differences in Notes. |
| Stub | Returns success without performing the operation. |
| Replaced | An available replacement requires client changes. |
| Planned | Recorded implementation decision; not implemented. |
| Deferred | Waits on the named work. |
| Unavailable | Not implemented; no delivery commitment. |
| Undecided | Disposition remains open. |
| Retired | Not part of the target API. |
Decisions and open questions
Node-method dispositions follow #11022. Issue #11033 tracks remaining shared-RPC differences.
RPC methods
Control
| Method | Owner | Status | Notes |
|---|---|---|---|
getexperimentalfeatures | — | Retired | |
getinfo | Zebra | Partial | No timeoffset or wallet balances. The unset proxy field is omitted; paytxfee is always 0. |
getmemoryinfo | — | Retired | |
help | Zallet | Done | Zallet serves help for its own RPC; Zebra does not. |
setlogfilter | — | Replaced | Use the tracing /filter endpoint; requires the filter-reload build feature. |
stop | Zebra | Partial | Regtest on non-Windows targets only. Returns Zebra server stopping, not Zcash server stopping. #11033 |
Blockchain
| Method | Owner | Status | Notes |
|---|---|---|---|
getbestblockhash | Zebra | Done | |
getblock | Zebra | Partial | No authdataroot, chainhistoryroot, chainwork, or anchor. The verbosity argument is numeric, not boolean. Adds nTx and Ironwood tree data. Verbosity 3 adds transparent-input prevout data and non-coinbase fee where spent outputs are available. |
getblockchaininfo | Zebra | Partial | No initial_block_download_complete, softforks, or pruneheight. The chainwork value is numeric 0, not a hex string; commitments is 0, pruned is false, and headers equals the committed tip. #11033 |
getblockcount | Zebra | Done | |
getblockdeltas | — | Retired | Build explorer indexes on indexer gRPC. |
getblockhash | Zebra | Done | |
getblockhashes | — | Retired | Build explorer indexes on indexer gRPC. #8436 |
getblockheader | Zebra | Partial | Omits zcashd’s undocumented chainwork; adds blockcommitments. |
getchaintips | — | Planned | Tracked in #10666. |
getdifficulty | Zebra | Partial | Computes the ratio from the high 128 bits of expanded targets. Returns an error when state cannot supply chain information, where zcashd can return 1.0 for an empty chain. |
getmempoolinfo | Zebra | Partial | The regtest-only key is spelled fully_notified; zcashd spells it fullyNotified. |
getrawmempool | Zebra | Partial | In verbose output, modifiedfee equals fee; Zebra has no prioritisetransaction fee deltas. |
getspentinfo | — | Planned | Intended to use Zebra’s indexer spending-transaction index. |
gettxout | Zebra | Partial | P2PK outputs do not receive the synthetic P2PKH address reported by zcashd; the script address list is empty. |
gettxoutproof | — | Planned | Decision. |
gettxoutsetinfo | — | Retired | Build explorer indexes on indexer gRPC. |
verifychain | — | Retired | |
verifytxoutproof | — | Planned | Decision. |
z_getsubtreesbyindex | Zebra | Done | Accepts sapling, orchard, and ironwood. During subtree-index rebuilding, a not-yet-rebuilt range can return an empty list. |
z_gettreestate | Zebra | Partial | Returns finalRoot and finalState for available Sapling, Orchard, and Ironwood trees. Omits Sprout entirely and does not return skipHash. Ironwood appears from NU6.3. #11033 |
Address index
| Method | Owner | Status | Notes |
|---|---|---|---|
getaddressbalance | Zebra | Done | Accepts a single transparent address or an object containing addresses; no separate address-index flag is required. |
getaddressdeltas | — | Retired | Build explorer indexes on indexer gRPC. #8440 |
getaddressmempool | — | Retired | Build explorer indexes on indexer gRPC. |
getaddresstxids | Zebra | Partial | Accepts a single address or an addresses object with optional inclusive start/end heights. Limits requests to 10,000 addresses and 100,000 matching transactions; narrow oversized ranges. |
getaddressutxos | Zebra | Partial | Supports chainInfo and optional startHeight/maxEntries result limits. Limits requests to 10,000 addresses and results to 100,000 outputs. |
Mining
| Method | Owner | Status | Notes |
|---|---|---|---|
getblocksubsidy | Zebra | Partial | Returns historical subsidies, including founders’ rewards before the first halving. After reserve-dependent reissuance begins, the preceding block must be available; arbitrary future subsidies cannot be forecast. |
getblocktemplate | Zebra* | Done | Implements long polling and proposal mode. Adds maxtime and submitold; advertises capabilities: ["proposal"] and mutable: ["time"]. |
getlocalsolps | — | Retired | |
getmininginfo | Zebra | Partial | No errors, errorstimestamp, genproclimit, localsolps, pooledtx, or generate. |
getnetworkhashps | Zebra | Done | Deprecated alias for getnetworksolps, in both zcashd and Zebra. |
getnetworksolps | Zebra | Done | |
prioritisetransaction | — | Planned | Decision. |
submitblock | Zebra* | Partial | Returns only null, duplicate, or rejected. zcashd also returns duplicate-invalid, duplicate-inconclusive, inconclusive, and the specific BIP-22 reject reason in place of a bare rejected. |
Generating
| Method | Owner | Status | Notes |
|---|---|---|---|
generate | Zebra* | Partial | Requires a network with proof-of-work validation disabled, including Regtest. Pays the configured [mining] miner_address. |
getgenerate | —* | Retired | Controls zcashd’s built-in CPU miner. Mine with getblocktemplate and submitblock. |
setgenerate | —* | Retired | Controls zcashd’s built-in CPU miner. Mine with getblocktemplate and submitblock. |
Network
| Method | Owner | Status | Notes |
|---|---|---|---|
addnode | Zebra* | Partial | Regtest only, and only the add command. #10329 |
clearbanned | — | Deferred | Waits on v2 P2P (#11281). |
disconnectnode | — | Deferred | Waits on v2 P2P (#11281). |
getaddednodeinfo | — | Deferred | Waits on v2 P2P (#11281). |
getconnectioncount | Zebra | Replaced | Use connections from getnetworkinfo or getinfo. |
getdeprecationinfo | Zebra | Partial | Returns end_of_service on Mainnet only. Omits version, subversion, deprecated_features, and disabled_features. |
getnettotals | — | Deferred | Waits on v2 P2P (#11281). |
getnetworkinfo | Zebra | Partial | No warningstimestamp. localaddresses and warnings are empty. #11030 |
getpeerinfo | Zebra | Partial | Returns addr, services, lastrecv, inbound, banscore, subver, version, startingheight, and the added connection_state; pingtime and pingwait are optional. Other zcashd per-peer fields are absent. |
listbanned | — | Deferred | Waits on v2 P2P (#11281). |
ping | Zebra | Stub | Does not request a ping; peers use automatic heartbeats (#11028). |
setban | — | Deferred | Waits on v2 P2P (#11281). |
Raw transactions
| Method | Owner | Status | Notes |
|---|---|---|---|
createrawtransaction | Zallet | Replaced | Use Zallet’s partially created Zcash transaction (PCZT) methods: pczt_create, pczt_prove, pczt_sign, and pczt_extract. #8952 |
decoderawtransaction | Zallet | Done | #8645 |
decodescript | Zallet | Done | |
fundrawtransaction | Zallet | Replaced | Use Zallet’s PCZT methods instead. |
getrawtransaction | Zebra | Partial | Uses numeric verbosity. Outputs expose valueZat, not valueSat, and omit spentTxId, spentIndex, and spentHeight. P2PK outputs have no synthetic address (#9671). Supports Ironwood transaction data. |
sendrawtransaction | Zebra | Partial | allowhighfees is accepted and ignored. |
signrawtransaction | Zallet | Replaced | Use Zallet’s PCZT methods instead. |
Util
| Method | Owner | Status | Notes |
|---|---|---|---|
createmultisig | Zallet | Unavailable | |
validateaddress | Zebra | Partial | No scriptPubKey. Adds isscript. |
verifymessage | Zallet | Done | #8947 |
z_validateaddress | Zebra | Partial | Emits address_type, not the deprecated type alias. Omits Sapling key details, rejects Sprout addresses, and reports ismine: false. |
Wallet
Build Zallet with zcashd-import
for its wallet-migration CLI. This also enables transparent-key-import;
without it, the registered z_importaddress RPC returns an error.
| Method | Owner | Status | Notes |
|---|---|---|---|
addmultisigaddress | Zallet | Unavailable | |
backupwallet | — | Retired | No Zallet RPC equivalent; zallet backup is unreleased, not in beta.3. |
dumpprivkey | — | Retired | Not planned by Zallet. |
encryptwallet | Zallet | Replaced | Key material is always encrypted from wallet setup; use walletpassphrase/walletlock. |
getbalance | Zallet | Replaced | Use z_getbalanceforaccount instead. |
getnewaddress | Zallet | Replaced | Use z_getnewaccount + z_getaddressforaccount. |
getrawchangeaddress | — | Retired | Not needed: Zallet derives change addresses internally. |
getreceivedbyaddress | Zallet | Unavailable | |
gettransaction | Zallet | Replaced | Use z_viewtransaction instead. |
getunconfirmedbalance | Zallet | Unavailable | |
getwalletinfo | Zallet | Partial | Balances and counters are placeholders; mnemonic_seedfp is "TODO" and mnemonic_seedfps is absent. Only unlocked_until is live. Use dedicated balance methods. |
importaddress | Zallet | Replaced | Use zallet import-address for a bare address, public key, or redeem script. The z_importaddress RPC covers public keys and redeem scripts, not bare addresses. |
importprivkey | Zallet | Unavailable | |
importpubkey | Zallet | Replaced | Use z_importaddress. |
importwallet | Zallet | Replaced | Use z_importkey per key, or zallet migrate-zcashd-wallet. |
keypoolrefill | — | Retired | Not needed: Zallet has no key pool. |
listaddresses | Zallet | Partial | Adds derived_transparent, expands watch-only coverage, and uses bech32m seed fingerprints. See altered semantics. |
listaddressgroupings | — | Retired | Not planned by Zallet. |
listlockunspent | Zallet | Unavailable | |
listreceivedbyaddress | Zallet | Unavailable | |
listsinceblock | Zallet | Unavailable | |
listtransactions | Zallet | Unavailable | Use z_listtransactions, optionally filtered by account. |
listunspent | Zallet | Replaced | Use z_listunspent instead. |
lockunspent | Zallet | Unavailable | |
sendmany | Zallet | Replaced | Use z_sendmany or z_sendfromaccount. |
sendtoaddress | Zallet | Replaced | Use z_sendfromaccount; see z_sendmany for its restrictions. |
settxfee | — | Retired | Zallet uses ZIP 317 fees rather than a wallet-wide fixed fee. |
signmessage | Zallet | Done | Signs with an unlocked wallet’s transparent P2PKH key; ephemeral addresses are refused. |
walletconfirmbackup | Zallet | Replaced | Internal to zcashd; use the zallet confirm-backup command. |
walletlock | Zallet | Done | |
walletpassphrase | Zallet | Done | |
walletpassphrasechange | Zallet | Undecided | |
z_converttex | Zebra | Partial | Converts P2PKH to ZIP 320 TEX. Returns -5 for Sprout or unsupported unified receivers where zcashd returns -8. |
z_exportkey | Zallet | Partial | Sapling extended spending keys only. |
z_exportviewingkey | Zallet | Partial | Supports Sapling and unified viewing keys, adds ivk, and rejects Sprout. See altered semantics. |
z_exportwallet | Zallet | Unavailable | Zallet plans a ZeWIF export workflow. |
z_getaddressforaccount | Zallet | Partial | Accepts account UUIDs; derives time-based addresses when neither a transparent receiver nor an explicit index is requested. See response changes. |
z_getbalance | Zallet | Replaced | Use z_getbalanceforaccount instead. |
z_getbalanceforaccount | Zallet | Done | |
z_getbalanceforviewingkey | Zallet | Replaced | Imported viewing keys get accounts, so z_getbalanceforaccount covers them. |
z_getmigrationstatus | — | Retired | Zallet does not support Sprout migration. |
z_getnewaccount | Zallet | Partial | Requires account_name; seedfp is required when the wallet contains multiple seeds. |
z_getnewaddress | Zallet | Replaced | Use z_getnewaccount + z_getaddressforaccount. |
z_getnotescount | Zallet | Done | |
z_getoperationresult | Zallet | Done | |
z_getoperationstatus | Zallet | Done | |
z_gettotalbalance | Zallet | Partial | Deprecated; include_watchonly = false is not honored. Prefer z_getbalanceforaccount. |
z_importkey | Zallet | Partial | Sapling extended spending keys only. |
z_importviewingkey | Zallet | Partial | Sapling extended full viewing keys only. |
z_importwallet | Zallet | Replaced | Use z_importkey per key, or zallet migrate-zcashd-wallet. |
z_listaccounts | Zallet | Partial | Adds UUIDs, names, seed fingerprints, and include_addresses. Omits account and diversifier_index when derivation is unknown. See response changes. |
z_listaddresses | Zallet | Replaced | Use listaddresses instead. |
z_listoperationids | Zallet | Done | |
z_listreceivedbyaddress | Zallet | Unavailable | |
z_listunifiedreceivers | Zebra | Done | |
z_listunspent | Zallet | Partial | Includes transparent outputs; uses value, account_uuid, and is_watch_only instead of amount, account, and spendable. Ignores include_watchonly. See response changes. |
z_mergetoaddress | Zallet† | Unavailable | |
z_sendmany | Zallet† | Partial | Requires a null or omitted fee; defaults to the ZIP 315 confirmation policy. Operation results add txids and broadcast. See altered semantics. |
z_setmigration | — | Retired | Zallet does not support Sprout migration. |
z_shieldcoinbase | Zallet† | Partial | Accepts one address or account UUID, not *; rejects non-null fees. Allows only AllowRevealedSenders or AllowLinkingAccountAddresses. Check the added broadcast result before assuming submission. |
z_viewtransaction | Zallet | Partial | Includes transparent activity, account effects, and additional top-level fields. See altered semantics. |
zcbenchmark | — | Retired | |
zcsamplejoinsplit | — | Retired | Sprout-only. |
Disclosure
| Method | Owner | Status | Notes |
|---|---|---|---|
z_getpaymentdisclosure | — | Retired | Sprout-only. |
z_validatepaymentdisclosure | — | Retired | Sprout-only. #8443 |
Hidden
| Method | Owner | Status | Notes |
|---|---|---|---|
dumpwallet | — | Retired | The legacy handler returns Method not found and directs callers to z_exportwallet. Zallet records a planned ZeWIF export instead. |
invalidateblock | Zebra | Partial | Only removes blocks present in non-finalized state; it is not a general finalized-chain rollback mechanism. |
reconsiderblock | Zebra | Partial | Uses the cache of previously invalidated blocks and returns reconsidered hashes rather than null. #11033 |
resendwallettransactions | Zallet | Undecided | |
setmocktime | — | Retired | Test-only. |
Zebra-only methods
| Method | Owner | Status | Notes |
|---|---|---|---|
getbestblockheightandhash | Zebra | Done | Tip height and hash in one call, so a caller cannot read a torn pair across two requests. |
getstandardfee | Zebra | Done | Returns {standard_fee, version}: the next block’s marginal fee per logical action, in zatoshis. Version is 0; the fee is height-dependent (#11557). |
generatetoaddress | Zebra | Done | Requires proof-of-work validation to be disabled, including Regtest. Pays the supplied address instead of [mining] miner_address. |
preciousblock | Zebra | Done | Returns null; see the tip-selection rules below. |
rpc.discover | Zebra | Done | OpenRPC schema for every Zebra RPC method. |
preciousblock prefers a non-finalized chain tip only over competing tips with
equal cumulative work; higher-work chains always take precedence. The last call
for competing equal-work tips wins. Preferences are not persisted and are lost
on restart. Finalized blocks and known non-tip blocks succeed without changing
the preference; an unknown hash returns Block not found.
Notification transports
Zebra publishes zcashd-compatible ZMQ hashblock notifications when built with
the optional zmq feature and configured with
[notify] zmqpubhashblock, for example
"tcp://127.0.0.1:28332". The publisher is off by default and has no
authentication; expose it only to trusted clients. Like block notify, it is
suppressed during initial sync. See
ZMQ block notifications for framing and
best-effort delivery semantics. The other ZMQ topics remain proposed in
#11034 and #11036. Zebra’s indexer gRPC streams cover
them but require client changes:
Set [rpc] indexer_listen_addr; the streams need no indexer
build feature. Keep the unauthenticated, plaintext listener on loopback or a
protected private network; JSON-RPC cookie authentication does not apply.
zcashd topic | Owner | Status | Available alternative and limitation |
|---|---|---|---|
zmqpubhashblock | Zebra | Done | Optional zmq feature and [notify] zmqpubhashblock (#10881). Indexer ChainTipChange, block-notify, and mining long polling are alternatives. |
zmqpubrawblock | Zebra | Replaced | Indexer NonFinalizedStateChange streams serialized non-finalized blocks, including forks; use GetBlock for a best-chain block by hash or height. This is not identical to a best-block ZMQ topic (#11034). |
zmqpubhashtx | Zebra | Replaced | Indexer MempoolChange reports added, invalidated, and mined transaction IDs. Its events are not the complete zcashd ZMQ transaction contract (#11036). |
zmqpubrawtx | Zebra | Replaced | Combine MempoolChange with getrawtransaction; a transaction removed before the lookup can be unavailable. The stream does not contain transaction bytes (#11036). |
Streams can close when a consumer falls behind. Reconcile state through RPC after reconnecting rather than treating notifications as a durable event log. The non-finalized stream accepts known chain-tip hashes; an empty request streams the current non-finalized state before subsequent updates.
Other integration points
| Integration point | Owner | Status | Notes |
|---|---|---|---|
REST interface (-rest, /rest/*) | — | Undecided | Use JSON-RPC; REST disposition remains open in #11022. |
-blocknotify hook | Zebra | Done | Set [notify] block_notify_command; notifications run only near the tip and can coalesce intermediate changes. #10726 |
-txexpirynotify hook | — | Undecided | Indexer MempoolChange reports invalidation without an expiry reason. #11022 |
-walletnotify hook | Zallet | Unavailable | Zallet beta.3 accepts [external] notify, and migrate-zcash-conf maps walletnotify to it, but startup warns that the hook is not implemented. See its release notes. |
-alertnotify hook | — | Retired | No Zebra alert-system hook. |
Configuration mappings
Zebra reads TOML, not zcash.conf. These are behavioral mappings, not accepted
aliases. See the configuration reference for defaults and
environment-variable overrides.
zcashd setting | Zebra setting or migration action |
|---|---|
conf | Select Zebra’s TOML with zebrad -c zebrad.toml start. |
datadir | Set [state] cache_dir for Zebra’s own database. This does not import a zcashd directory. |
testnet, regtest | Set [network] network = "Testnet" or "Regtest". |
bind, port | Set one [network] listen_addr = "IP:port"; multiple zcashd binds are not a direct mapping. |
externalip | Set [network] external_addr to the advertised socket address. |
addnode, seednode | Use [network] initial_mainnet_peers or initial_testnet_peers as discovery seeds, not a persistent-peer guarantee. |
connect, discover, proxy and relay-policy controls | No equivalent connect-only policy. Initial peers do not constrain later discovery; decisions remain in #11027. |
server, rpcbind, rpcport | Set [rpc] listen_addr; the listener is disabled when unset. |
rpcuser, rpcpassword, rpcauth | No static-credential mapping. Use Zebra’s cookie authentication and an authenticated deployment boundary for remote access. |
rpccookiefile | Set [rpc] cookie_dir; the filename is .cookie, not configurable separately. |
rpcallowip | No source-address allowlist setting. Bind to loopback or enforce access at the deployment boundary. |
rpcthreads | No equivalent thread-count control; [rpc] parallel_cpu_threads is deprecated and has no effect. |
mineraddress | Set [mining] miner_address on Zebra, not the sidecar. |
blocknotify | Set [notify] block_notify_command. |
| Wallet and ZMQ settings retained during migration | Keep them in the sidecar’s zcash.conf; Zebra does not translate or forward them. |
When reusing sidecar data, preserve its transaction-index setting.
zcashd-compat mode
Zebra’s zcashd-compat mode keeps legacy wallet integrations running during
migration to Zallet and Zebra’s JSON-RPC API. It uses a
separate, restricted zcashd sidecar with
shielded-pool limitations.
Zebra neither proxies wallet RPCs nor reads wallet.dat.
Process and network boundaries
Zebra validates blocks from public peers; the sidecar receives them over one outbound peer-to-peer (P2P) connection to Zebra. Each node validates and stores its own chain. Both binaries must support the current network upgrade.
public Zcash peers <--- P2P ---> zebrad <--- one P2P connection --- zcashd
^ ^
node/mining RPC wallet RPC, ZMQ
| |
miners legacy integration
| Responsibility | Zebra | Sidecar |
|---|---|---|
| Public P2P connections and peer discovery | Yes | No |
| Chain validation and storage | Zebra consensus and state | Separate validation, block files, and chainstate |
| Wallet keys, balances, and transaction construction | No wallet | Existing zcashd wallet |
| Mining templates and block submission | getblocktemplate, submitblock | Methods not registered |
| Legacy wallet RPC and configured ZeroMQ (ZMQ) topics | Not forwarded | Served directly |
Configure both nodes for the same network and use distinct RPC listeners. Compat mode does not move or secure RPC credentials or ZMQ endpoints. The migration configuration keeps the legacy wallet endpoint for existing clients; see ports for network defaults.
For mining pools, see the tested reward-to-payout procedure and its wallet RPC, fee, and privacy-policy limits.
Quick start: the installer
Download and inspect the installer, then run it:
curl -fsSL https://raw.githubusercontent.com/ZcashFoundation/zebra/main/scripts/install-zebra.sh -o install-zebra.sh
less install-zebra.sh
bash install-zebra.sh --install-profile zcashd-compat
| Installer mode | Behavior |
|---|---|
split-binary (default) | Downloads Zebra and the sidecar; prints separate start commands. |
supervised | Downloads Zebra; prints a command that selects Zebra’s embedded sidecar download and supervision. |
build-from-source | Checks source trees and toolchains; prints build and start commands. |
The installer checks platform, hardware, filesystem capacity, and permissions; it offers discovered data directories and verifies downloaded archives against SHA-256 checksums. It does not start the nodes. For unattended installation, supply the mode and paths explicitly:
bash install-zebra.sh --install-profile zcashd-compat --mode supervised \
--network Mainnet --zebra-state-dir /var/lib/zebra \
--zcashd-datadir /var/lib/zcashd --non-interactive
The installer has no compat Docker option, and Zebra publishes no sidecar image. To use a regular Zebra container, provide a compatible sidecar binary or run it separately, mount persistent data, and configure container-reachable addresses.
The sidecar zcashd build
Use the Zcash Foundation sidecar release, not stock
zcashd. Zebra 7.0.0 embeds this release manifest:
| Property | Pinned value |
|---|---|
| Release | zebra-compat-v1.3.0 |
| Download target | x86_64-pc-linux-gnu only |
| Archive | zcashd-zebra-compat-v1.3.0-linux-x86_64.tar.gz |
| Archive SHA-256 | a419147c49b08a15be2f1f1f5fea2801b5228491f62835bfa8caae8b06350c13 |
| Executable archive member | ./bin/zcashd |
The pinned build supports NU7; zebra-compat-v1.2.0 and earlier cannot follow
it. See activation heights.
The sidecar enforces these restrictions independently of Zebra:
- Exactly one nonempty
-connectpeer is required. Additional peers through-addnodeor-seednode, P2P binds, and-listen=1are rejected. TheaddnodeRPC is not registered. - The
getblocktemplate,submitblock,getgenerate,setgenerate, andgeneratemethods are not registered; calls return-32601. - The upstream scheduled end-of-support halt and warning are absent. This does not remove the need to update the sidecar for consensus changes.
The sidecar’s network RPCs describe its connection to Zebra, not Zebra’s
public peer set. Its prioritisetransaction method affects only its own
mempool, not Zebra’s mining templates.
Running externally managed
Set the compat configuration in Zebra’s TOML file:
[zcashd_compat]
enabled = true
manage_zcashd = false
zcashd_datadir = "/var/lib/zcashd"
Start Zebra with that file, then start the sidecar separately:
zebrad -c zebrad.toml start --zcashd-compat
zcashd -datadir=/var/lib/zcashd -connect=127.0.0.1:8233 \
-listen=0 -dnsseed=0 -listenonion=0 -discover=0 -printtoconsole
For Testnet, add -testnet and use Zebra’s Testnet P2P address. For Regtest,
add -regtest -regtestacceptunvalidatedpow: Zebra’s generated Regtest blocks
omit proof of work, so the sidecar must accept them.
The sidecar needs an existing zcash.conf; an empty file is sufficient for
peer selection when the arguments above supply it. Preserve existing wallet,
index, and RPC settings. Remove conflicting peer-selection settings. The
installer prepares a missing config for its externally managed modes; Zebra
does not bootstrap or supervise an externally managed process.
Running supervised
Use the supervised configuration
and start command.
The CLI flag enables compat mode, not supervision; explicitly set
[zcashd_compat] manage_zcashd = true.
With zcashd_source = "embedded", Zebra downloads the pinned archive over
HTTPS, verifies its digest, and caches the executable under
<state.cache_dir>/zcashd-compat/bin/<release>/<target>/zcashd. Cached binaries
are checked against their recorded digest before reuse; an invalid cache
triggers a fresh download. Downloads have a 10-minute timeout and a 1 GiB
size limit. Unsupported targets, failed downloads, and archive digest
mismatches fail startup. To supply a verified local build, set
zcashd_source = "path" and zcashd_path; an explicit path takes precedence
over zcashd_source and bypasses the embedded download and checksum policy.
Zebra creates a missing sidecar data directory and minimal zcash.conf but
never overwrites an existing config. Prefer [zcashd_compat] zcashd_datadir
to a path in zcashd_extra_args; see the
path and extra-argument rules.
The supervisor passes the selected data directory, network flags, and
-printtoconsole, followed by extra arguments. It then appends the forced
peer settings and -daemon=0, keeping the child in the foreground. Testnet
adds -testnet; Regtest adds -regtest -regtestacceptunvalidatedpow.
Peer-selection arguments (-connect, -addnode, -seednode, including
negated forms) are forbidden in zcashd_extra_args.
The sidecar’s stdout and stderr enter Zebra’s logs under
zcashd_compat.zcashd. With the normal -printtoconsole setting, the sidecar
writes there instead of debug.log.
Sidecar peer privileges
Set [zcashd_compat] p2p_connect_addr if the sidecar needs a destination other
than Zebra’s bound listener. The block_gossip_peer_ips field instead matches
the source IP Zebra sees. For a separate host or container, configure
that source:
[zcashd_compat]
enabled = true
manage_zcashd = false
block_gossip_peer_ips = ["10.0.0.2"]
Listed IPs receive block inventory broadcasts without random peer sampling.
They share a reserved inbound pool with one slot per distinct source IP,
capped to preserve at least one public inbound slot. The pool is shared,
not a per-IP quota. Reserved connections bypass the recent-IP reconnection
limit; overflow connections compete for public slots with normal rate
limiting. All listed IPs are exempt from the FindBlocks/FindHeaders
sync-stall detector.
Treat this list as a trust boundary. List only addresses whose originating
processes are all trusted. Do not grant a shared Docker proxy or NAT address
these privileges if untrusted public connections also arrive from it. Use
networking that distinguishes the sidecar’s source. Merely raising
[network] max_connections_per_ip does not establish that distinction.
Supervision and shutdown
See supervisor timing fields for delays and defaults.
The supervisor retries spawn failures and every pre-shutdown child exit, including success, indefinitely with exponential backoff. Persistent config errors therefore cause restart loops. If supervision itself fails, Zebra logs the failure and keeps running; monitor the wallet process separately.
Stop Zebra rather than stopping its supervised child independently. Normal shutdown sends the child SIGTERM, waits for the grace period, then uses SIGKILL if necessary. An abrupt kill of Zebra cannot perform this cleanup; check for a surviving child before starting another process on the same data directory.
Verify the integration
Query the sidecar with its own credentials:
zcash-cli -datadir=/var/lib/zcashd getpeerinfo
zcash-cli -datadir=/var/lib/zcashd getconnectioncount
zcash-cli -datadir=/var/lib/zcashd getblocktemplate
Check the sidecar’s single-peer and disabled-mining restrictions above.
Inspect listeners with ss -tlnp: only Zebra should listen for P2P.
Then compare fixed-height block hashes and wallet state.
Mining: Zebra is canonical
Send mining requests to Zebra, not the sidecar; configure its payout address as described in Mining.
Wallet shielded-pool support (Orchard & Ironwood)
The sidecar transaction builder enforces these restrictions:
| Pool | Sidecar wallet behavior |
|---|---|
| Ironwood | Unsupported by the wallet. |
| Orchard | From NU6.3 for the next block, excludes Orchard notes from selection and rejects transactions requiring Orchard spends, payments, or change. |
| Sapling | Supported; from NU7, transaction construction also enforces the ZIP 218 Sapling action limit. |
| Transparent | Supported. |
| Sprout | From NU7, spending is rejected because ZIP 2003 removes version 4 transactions. |
The Orchard restriction can produce RPC_INVALID_PARAMETER (-8) with the
message beginning zcashd does not support the Ironwood pool. An unsupported
recipient can instead fail address resolution. A unified address with a
supported receiver does not require an Orchard payment: selection also
respects the requested privacy policy and available funds.
Migrate affected wallet workflows to Zallet, not an indexer or wallet library.
Initial sync and existing datadirs
Follow Migrating an existing zcashd node for data reuse, sync gates, backup, cutover, troubleshooting, and rollback.
Configuration reference
See the configuration reference for fields, defaults, and array parsing. To override extra arguments through the environment:
export ZEBRA_ZCASHD_COMPAT__ZCASHD_EXTRA_ARGS='["-rpcbind=127.0.0.1","-rpcallowip=127.0.0.1"]'
Hardware preflight (Linux)
Enabled compat mode runs Linux startup checks even without supervision. Preflight checks available CPUs, cgroup-limited RAM, directory permissions, and filesystem capacity, not free space. Check free space separately and allow for growth.
Minimum shortfalls and check errors fail startup. The --unsafe-low-specs
flag downgrades errors to warnings; it does not repair permissions, provide
missing libraries, bypass checksum validation, or prevent later failures.
Preflight checks executable permissions, not dynamic-library loading.
Non-Linux compat startup also needs this override; embedded downloads remain
Linux x86-64 only.
Migrating an existing zcashd node
Use compat mode while migrating legacy wallet RPC callers;
without wallet dependencies, run Zebra directly. Zebra needs its own
state database and cannot import zcashd block files.
Zallet beta.3’s direct zebra backend is incompatible with Zebra 7.0.0;
the RPC-based zaino backend is unaffected.
Review the sidecar’s wallet pool restrictions
before moving production traffic.
What you need
- A cleanly stopped
zcashddata directory, its configuration, and a verified wallet backup. Record the network, wallet filename, RPC credentials, and index settings. - Separate storage for Zebra’s state and enough time for its own sync, or a Zebra state snapshot.
- Zebra and a compatible sidecar build.
- Capacity and permissions satisfying compat preflight.
The sidecar reuses the existing chain and wallet state from a compatible
zcashd data directory; Zebra still needs its own synchronization.
The procedure below uses Mainnet, /var/lib/zcashd for the existing sidecar
root, and /var/lib/zebra for Zebra. Substitute the actual paths. On Testnet or
Regtest, preserve the network-specific wallet subdirectory and configure both
processes for the same network.
The migration
1. Back up the wallet
Stop the existing node cleanly and wait for it to exit before copying its
files. Preserve the entire data directory when rollback must also restore
chainstate and indexes. At minimum, back up the actual wallet file and
zcash.conf to restricted storage:
install -d -m 700 /secure/zcashd-backup
cp -p /var/lib/zcashd/wallet.dat /secure/zcashd-backup/wallet.dat
cp -p /var/lib/zcashd/zcash.conf /secure/zcashd-backup/zcash.conf
Use the configured wallet filename if it is not wallet.dat. Do not run the
old daemon and sidecar against the same directory, and do not overwrite the
backup during the migration.
2. Write the config
Create a Zebra config with separate RPC and state paths:
[network]
network = "Mainnet"
[state]
cache_dir = "/var/lib/zebra"
[rpc]
listen_addr = "127.0.0.1:8234"
cookie_dir = "/var/lib/zebra"
[zcashd_compat]
enabled = true
manage_zcashd = true
zcashd_source = "embedded"
zcashd_datadir = "/var/lib/zcashd"
zcashd_extra_args = ["-rpcbind=127.0.0.1", "-rpcallowip=127.0.0.1"]
This Mainnet layout keeps sidecar RPC on port 8232 for existing clients and assigns Zebra port 8234. Keep both on loopback unless an authenticated, encrypted deployment boundary protects them.
For a supplied binary, set zcashd_source = "path" and
zcashd_path = "/usr/local/bin/zcashd"; verify it separately because this
bypasses embedded archive verification.
Review the existing zcash.conf before startup:
- Preserve the transaction-index setting. The supervisor does not add
-txindex; an existingtxindex=1inzcash.confis sufficient. Add"-txindex=1"tozcashd_extra_argsonly when needed to preserve the directory’s existing setting. Changing it triggers a reindex requirement. - Remove
connect,addnode,seednode,bind, andwhitebindentries that conflict with the supervisor’s sole peer. Multi-valued peer options accumulate; a command-line setting does not erase existing peers. - Preserve wallet, authentication, and ZMQ settings. The supervisor does not translate them or choose a nonconflicting RPC port.
Give the Zebra service account access to both data directories and the wallet’s existing authentication files.
3. Start
Start the supervised pair:
zebrad -c zebrad.toml start --zcashd-compat
Zebra runs preflight, resolves the binary, initializes its services, and
starts the sidecar after the configured delay, without waiting for the tip.
Honor reported index-upgrade, wallet-scan, or recovery requirements.
Read sidecar output in Zebra’s logs,
not the old debug.log.
4. Wait for both nodes to reach the tip
Query the sidecar:
zcash-cli -datadir=/var/lib/zcashd getblockchaininfo
zcash-cli -datadir=/var/lib/zcashd getpeerinfo
The sidecar must have one outbound peer pointing to Zebra. Its
initial_block_download_complete field must become true before
chain-dependent wallet operations are available. Those methods return -28
with This wallet operation is disabled while reindexing. during initial
block download even when no reindex is occurring. Key and address methods
such as z_exportkey and listaddresses are not subject to that gate.
Monitor Zebra’s synchronization separately. If both heights stall, inspect Zebra’s sync and peers; if only the sidecar stalls, inspect its connection, consensus version, and logs. Equal heights do not prove either node has reached the network tip.
5. Verify
Check peer and listener isolation, then compare block hashes at a fixed height available on both nodes:
height=$(zcash-cli -datadir=/var/lib/zcashd getblockcount)
zcash-cli -datadir=/var/lib/zcashd getblockhash "$height"
curl --silent --show-error --user "$(cat /var/lib/zebra/.cookie)" \
--header 'Content-Type: application/json' \
--data-binary "{\"jsonrpc\":\"2.0\",\"id\":\"migration\",\"method\":\"getblockhash\",\"params\":[$height]}" \
http://127.0.0.1:8234/
Compare the sidecar’s hash with the JSON result; recheck at a stable height
if a reorganization occurs. Protect the RPC cookie;
do not put it in a URL.
Query wallet state through the sidecar and compare it with the inventory recorded before migration:
zcash-cli -datadir=/var/lib/zcashd listaddresses
zcash-cli -datadir=/var/lib/zcashd getwalletinfo
zcash-cli -datadir=/var/lib/zcashd z_gettotalbalance
Check addresses, balances, application RPCs, and notification consumers before restoring wallet traffic. A matching tip does not prove the correct wallet was loaded.
After the migration
The sidecar disables deprecated methods such as z_listaddresses by default,
but keeps z_gettotalbalance enabled. Migrate callers using the
API table. For a required legacy method,
append -allowdeprecated=<method> to zcashd_extra_args; do not replace
the existing arguments.
Troubleshooting
| Symptom | Diagnosis and action |
|---|---|
| Hardware or capacity preflight fails | Resolve the reported problem; see preflight and override limits. |
| Dynamic-loader or missing-library error | Use a compatible runtime or verified native build; preflight does not load libraries. See container options. |
reindex to change -txindex | Restore the existing index setting. Reindex only for an intentional setting change or diagnosed damage. |
| Sidecar RPC cannot bind | Separate the RPC addresses/ports; preserve the sidecar’s port if clients depend on it. |
| Zebra cannot write its auth cookie | Check permissions and symlinks against the authentication requirements. |
Wallet RPC returns -28 | Wait for sidecar initial block download; inspect logs if height stalls. |
| Wallet operation rejects Orchard or Ironwood | Migrate the restricted wallet workflow; changing peers or RPC routing does not help. |
| Sidecar exits or cannot spawn | Read the child error before the supervisor warning; retries are indefinite. |
| Zebra runs without a managed sidecar | Check manage_zcashd, startup logs, and supervisor errors; supervisor failure does not stop Zebra. |
| Sidecar cannot connect across hosts or containers | Check destination and firewall, then configure the trusted source IP. |
| Both nodes stop at an upgrade boundary | Check both versions; upgrading Zebra does not update an explicit zcashd_path binary. |
Rolling back
Stop wallet traffic, then stop the supervised pair. Check for a surviving child before changing binaries or supervision.
To undo supervision without discarding wallet progress:
- Set
[zcashd_compat] manage_zcashd = false; keep compat mode enabled for the sidecar’s trusted-peer handling. - Restart Zebra with the same state and RPC configuration.
- Start the same compatible sidecar externally against its existing data
directory, with exactly one
-connectpeer pointing to Zebra. See Running externally managed. - Repeat the fixed-height hash and wallet checks before restoring traffic.
For a binary or data rollback, preserve the failed directory and restore a
clean backup separately. Use binaries compatible with current consensus
rules and the backup’s format; an obsolete stock zcashd is not a supported
replacement. Do not assume on-disk updates are reversible by downgrading.
Reconcile wallet activity since the backup before spending: it can omit later keys and transactions. Retain the original wallet and subsequent backups until migration is complete. Disabling supervision does not convert wallet formats.
Mining Zcash with Zebra
Zebra supplies block templates and validates submitted blocks over JSON-RPC. A mining pool distributes those templates to external Equihash solvers; Zebra does not start a solver when an operator configures a reward address.
Download Zebra
Install Zebra or use the Docker instructions. Miners must follow the rules at the candidate block’s height, including the changes in NU7; see the public activation schedule.
Configure Zebra for mining
Generate a configuration file:
zebrad generate -o zebrad.toml
Set the following fields in the existing sections. Replace the address placeholder with an address controlled by the reward recipient.
[mining]
miner_address = "YOUR_MAINNET_MINING_ADDRESS"
internal_miner = false
[rpc]
listen_addr = "127.0.0.1:8232"
enable_cookie_auth = true
[network]
experimental_early_block_relay = true
See the mining configuration reference for field defaults
and limits, and Early block relay and compact blocks
for the [network] setting.
Miner address
The coinbase builder supports these address kinds on the selected network:
- Transparent pay-to-public-key-hash (P2PKH) and pay-to-script-hash (P2SH).
- Sapling payment addresses.
- Unified Addresses with a supported receiver. Zebra tries the Orchard receiver first, then Sapling, then transparent.
From NU6.3 onward, Zebra pays an Orchard receiver through the Ironwood shielded pool, not the Orchard pool. Ironwood uses the same receiver address format. Coinbase transactions cannot contain an Orchard component from NU6.3 onward; see ZIP 213 and ZIP 229.
Sprout addresses and transparent-source-only TEX addresses are not supported coinbase destinations. A syntactically valid address is not sufficient: it must match the network and contain a supported receiver. Zebra has no default reward recipient. Non-mining nodes need no address. A missing address, a network mismatch, or an invalid memo prevents template generation.
Extra coinbase data
The coinbase transaction is the first transaction in a block. Its input
encodes the block height and arbitrary miner data under the
transaction consensus rules.
Zebra adds the Unicode U+1F993 marker, encoded as bytes f0 9f a6 93, to this
input. A nonempty [mining] extra_coinbase_data follows the marker and a
two-byte separator, a colon and a space (3a 20):
extra_coinbase_data = "/MyPoolName/"
The 86-byte limit reserves space for the height, marker, separator, and script push encoding within the 100-byte coinbase input limit. Zebra rejects an oversized value while loading the configuration. An unset or empty value leaves only the marker after the height.
Miner memo
Set [mining] miner_memo to attach a memo to a shielded miner output:
miner_memo = "Mined by MyPoolName"
A coinbase memo is not private. Shielded coinbase outputs must be recoverable with the all-zero outgoing viewing key, so anyone can recover their recipient, value, and memo. Do not include secrets. This differs from ordinary shielded payments; see ZIP 213.
RPC section
Configure the pool to read Zebra’s RPC cookie and reload it after each restart. Bind RPC to loopback for a local pool; use a protected tunnel or TLS proxy for remote connections. Disable authentication only when an isolated network or an authenticating proxy controls access.
A submitted block that exactly matches a successful local proposal verdict, except for the nonce and Equihash solution, reuses that verdict and is advertised before commit. Solved proof of work, size, current time, live parent eligibility and local invalidation are checked before the body becomes available to peers. Changed or uncached proposals undergo normal full validation; checkpoint-era blocks wait for commit. These advertisements use ordinary block gossip; all-peer advertisements are reserved for committed submissions. With early block relay, peers that also enable it can receive an uncached submitted block before its transaction verification finishes. The RPC response still waits for verification and commit.
Committed notifications belong to the state writer, so cancelling an RPC does not discard a later successful commit notification. The default notification channel holds at most 10,000 messages. A full or closed channel produces an operational error before committing a mined block, without blocking ordinary state writes. Notification insertion does not guarantee delivery to peers. Restrict RPC access to trusted miners.
ZMQ block notifications
Pools using zcashd’s hashblock subscription can receive the same multipart messages from Zebra.
Build or install Zebra with --features zmq, then add this to zebrad.toml:
[notify]
zmqpubhashblock = "tcp://127.0.0.1:28332"
This is equivalent to zcashd’s -zmqpubhashblock=tcp://127.0.0.1:28332. The Cargo feature is off by
default and uses a pure-Rust implementation; no system ZMQ package is needed. Only TCP is supported.
Official default binaries and Docker images do not include it. Without the feature, configuring the
endpoint is a startup error.
With no endpoint configured, no ZMQ socket is opened.
Subscribe to hashblock. Each notification has three frames:
- The nine ASCII bytes
hashblock, with no null terminator. - The 32 hash bytes in display order: hex-encoding them gives the
getbestblockhashRPC result. - A four-byte little-endian sequence number, starting at zero, wrapping after
2^32 - 1, and resetting when Zebra restarts.
Like block notify, notifications are suppressed until Zebra is close to the network tip. Only best-chain tip changes are reported, including reorganizations; this is not a per-block archive or a same-tip transaction update feed. Intermediate tips may be coalesced. A bounded local queue holds up to 1,000 messages and drops on overflow, never delaying validation. Each subscriber has its own 64-message queue. A full queue or a write taking more than one second disconnects only that subscriber; other subscribers continue receiving notifications.
The server accepts at most 64 connections, including handshakes in progress. Handshakes and incomplete incoming frames have five-second deadlines. After the handshake, a client must send its first valid subscription within five seconds; other frames do not extend that deadline. Subscribed clients may remain silent indefinitely without heartbeats. Only ZMTP 3.x NULL connections with a SUB or XSUB socket type are supported. Inbound frames must use short lengths (at most 255 bytes), with at most 16 distinct subscription prefixes of up to 64 bytes each. Each connection may send at most 128 frames and 16 KiB, including frame headers, per one-second window after the handshake. Exceeding either budget disconnects the client. The budgets renew so periodic heartbeats do not exhaust a lifetime allowance. Invalid clients are disconnected. Heartbeat intervals shorter than about 10 ms can exceed the 128-frame-per-second inbound budget and disconnect the subscriber; ordinary intervals of 100 ms or more are unaffected. Heartbeat requests receive bounded PONG replies through the same per-subscriber queue, even when the chain tip is unchanged. Reply contexts are capped at 16 bytes.
Subscribers must fetch fresh work over RPC and reconcile after sequence gaps, reconnects, or restarts. Keep periodic RPC polling as a backstop: coalesced tips do not necessarily create sequence gaps.
ZMQ has no authentication or encryption. Restrict the endpoint to trusted clients using firewall rules or a private network, and account for untrusted local applications even on loopback. Never expose it directly to the public Internet. These limits do not provide fair admission: a misbehaving trusted client can still occupy all slots by subscribing, even to an unused topic or subsequently cancelling, or repeatedly reconnect to obtain fresh inbound budgets. Restrict access to well-behaved clients; the endpoint is not a defense against hostile clients.
Early block relay and compact blocks
Set [network] experimental_early_block_relay = true on every mining node. Between two nodes that
both enable it:
- A node advertises a block extending its best tip once the header, proof of work and transaction commitments check out, before transaction verification finishes. Each receiver still verifies the block fully before committing it.
- Up to three such peers per announcement receive a compact block: the header and short transaction IDs. Each rebuilds the block from its mempool and requests missing transactions once; if that fails, it fetches the full block.
Other peers, including zcashd, other node implementations and Zebra releases without this setting,
get ordinary full-block relay. Nodes negotiate the feature with experimental service bit 25 and a
sendcmpct exchange. Neither is an allocated Zcash protocol, and the encoding may change in later
releases; nodes without a common encoding version fall back to full blocks.
Experimental early block relay describes the limits and
failure handling.
Running Zebra
Start Zebra with the configuration file:
zebrad -c zebrad.toml start
Wait for synchronization before requesting mining work. On Mainnet,
getblocktemplate requires the sync service to be close to the tip and the
tip timestamp to be no more than 7,500 seconds old. A stale node or an
incorrect local clock can produce RPC error -10, indicating that Zebra
has not synced to the chain tip.
Testnet still requires synchronization, but it does not apply the Mainnet
tip-age limit. This permits mining to resume after a Testnet stall. Networks
with proof of work disabled, including default Regtest, bypass this sync
check. The [rpc] debug_force_finished_sync field changes only the estimate
in getblockchaininfo; it does not bypass the mining check.
Reusing proposal validation
Zebra validates transaction-bearing templates in the background. If you change the coinbase,
transactions, timestamp or another header field, submit the modified block with
getblocktemplate parameters {"mode":"proposal","data":"<hex-encoded-block>"} before mining it.
A successful proposal lets a later submitblock reuse verification when only the nonce and
Equihash solution differ. Proof of work, current time, size and state application are still checked.
The cache holds up to eight recent successful proposals within a 576 MiB retained-content budget,
including spent-output scripts. This uses a conservative retention charge, not measured heap usage.
It keeps eight representative full-size proposals with 19,215 transactions using 104-byte
one-input/one-P2PKH-output payments, charged at about 66.2 MB per proposal. Proposals packed with
smaller consensus-valid transactions or large spent-output scripts can retain fewer entries.
An evicted or changed proposal is verified normally: eviction loses cache hits, not correctness.
Template requests share the updater’s work during startup and refreshes, rather than starting a separate coinbase proof for each waiting miner. On-demand construction runs off the async executor and rechecks the committed parent, synchronization, timestamp range and difficulty before returning. Long polls also recheck synchronization after waking. Expanding a clock-capped timestamp range does not cancel still-valid work; a genuine difficulty-boundary transition still replaces it. New committed tips interrupt obsolete snapshot reads without abandoning a started proof. If the updater stops, cached work is refreshed at the normal mempool polling interval. Long polls continue observing shared template publications after switching to on-demand refresh. When no usable work exists, snapshot failures are returned to the RPC caller rather than hidden behind an indefinite cache wait.
Testing the setup
On Linux, with the default cookie directory, request a template:
cookie="${XDG_CACHE_HOME:-$HOME/.cache}/zebra/.cookie"
curl --fail-with-body --silent --show-error \
--user "$(cat "$cookie")" \
--header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":1,"method":"getblocktemplate","params":[]}' \
http://127.0.0.1:8232/ | jq
Use the configured cookie path if [rpc] cookie_dir differs. A successful
response contains a result object with these mining inputs:
| Field | Meaning |
|---|---|
height, previousblockhash | Candidate height and parent block. |
coinbasetxn.data | Serialized coinbase transaction, including the configured reward destination. |
transactions | Selected non-coinbase transactions; depends uses 1-based positions in this array. |
defaultroots | Header commitment roots for the supplied coinbase and transactions. |
bits, target | Compact and expanded difficulty targets. |
mintime, curtime, maxtime | Unix timestamps, in seconds, for the allowed range and suggested block time. |
longpollid | Opaque identifier for subsequent long-poll requests. |
Treat RPC errors as failures; an HTTP success status alone does not establish that the node can generate templates.
Mining RPCs
See the mining RPC reference for method parameters and results.
Proposal mode accepts a request object with "mode":"proposal" and
"data" containing the candidate block hex. It checks the candidate without
requiring valid proof of work and returns null for a valid proposal or a
rejection string. Submit a solved block with submitblock, not with a
getblocktemplate submit mode.
A submitblock result of null means Zebra committed the block, including
when it is on a side chain. Unlike zcashd’s distinction between accepted and
inconclusive submissions, this result does not prove that the block is on
Zebra’s best chain.
Long polling and template expiry
Long polling keeps an RPC request open until work changes. Pass the previous
template’s longpollid in the next request object:
{"jsonrpc":"2.0","id":2,"method":"getblocktemplate","params":[{"longpollid":"REPLACE_WITH_RETURNED_LONGPOLLID"}]}
Zebra wakes the request when the tip or relevant template state changes. It
also returns when the template’s inclusive maxtime expires during the
request, even if the tip and mempool do not change. The response can retain
the same longpollid for this expiry notification. Repeating a request for
that already-expired template waits for changed work rather than repeatedly
returning the same expiry notification.
The response’s submitold field is false when the tip changes or the time
range expires. Discard old work in that case. A mempool-only change can return
submitold: true. Allow long requests through the pool’s HTTP client and
any reverse proxy; do not interpret a pending long poll as an RPC failure.
Coinbase and consensus rules
Use the supplied coinbase and transaction set unless the pool implements the
current consensus rules. Modifying transactions changes fees and commitment
roots; modifying the coinbase also requires recomputing the affected roots.
The defaultroots values apply only to the template as returned.
The template marks the coinbase as required and advertises only "time" in
its mutable list. Request a fresh template when changing the parent or the
transaction set rather than reusing its coinbase.
Zebra constructs mandatory funding-stream outputs and scheduled lockbox disbursements. The lockbox is the deferred value pool: its allocation is accounted for without an ordinary recipient output. Under the active revision-2 funding streams, 12% of the subsidy is deferred and 8% goes to Zcash Community Grants through the Financial Privacy Fund; the miner receives the remainder and claimable fees. Amounts use consensus integer rounding. See ZIP 207, ZIP 214, and ZIP 271.
From NU6 onward, a coinbase must claim its full allowed value; miners cannot silently discard a portion of the subsidy or fees. Preserve the deferred allocation and any disbursement as well as the miner output. See ZIP 236.
Transparent coinbase outputs must mature for 100 blocks and must be shielded when spent on public networks. Shielded coinbase outputs do not have that 100-block maturity rule. See ZIP 213 and the transaction consensus rules.
From NU6.3 (Ironwood) activation, the template’s coinbase is a version 6 transaction, and the template can include version 6 transactions from the mempool. Pool software that parses, checks, or rebuilds template transactions must support the version 6 transaction format.
NU7 mining rules
Zebra 7.0.0 applies these rules from the NU7 activation height on each network:
- Timing and issuance: ZIP 218 changes the target spacing from 75 to 25 seconds and the difficulty averaging window from 17 to 102 blocks. Scheduled subsidy per block is divided by three, with integer rounding, and remaining halving intervals are tripled.
- Block contents: ZIP 218 limits a block to 330 Orchard actions, 330 Ironwood actions, and 300 Sapling inputs plus outputs, with a shared budget of 330 across those pools. The limits include coinbase outputs. Sprout JoinSplits are disallowed. ZIP 2003 disallows version 4 transactions; versions 5 and 6 remain valid.
- Fees: ZIP 235 removes
floor(total_transaction_fees * 6 / 10)zatoshis from circulation into the Network Sustainability Mechanism (NSM) reserve. Calculate this once over the block’s aggregate fees, not separately for each transaction. The coinbase claims only the remainder. The template’scoinbasetxn.feeis the negative of those claimable miner fees; non-coinbasefeefields remain gross. - Reserve reissuance: ZIP 237 adds a
rounded-up fraction of the parent block’s NSM reserve to scheduled subsidy
from height 8,939,664 on Mainnet or 7,305,222 on Testnet. This is not
immediate reissuance at NU7 activation. From that height,
getblocksubsidyneeds the parent block’s state and cannot calculate subsidies beyond the block after the best tip. - Funding periods: ZIPs 207 and 214 stretch the remaining funding periods for the faster spacing. The revision-2 streams end at the adjusted third halving: 6,132,390 on Mainnet and 4,497,948 on Testnet, exclusive. ZIP 2008 rotates the Mainnet grants recipient starting with the first funding period after the period containing the last pre-NU7 block. Do not hard-code a single recipient address in a pool.
The NSM reserve is separate from the lockbox. The former tracks funds removed from circulating supply for algorithmic reissuance; the latter holds deferred funding-stream value. Zebra’s generated coinbase applies both mechanisms.
Run a mining pool
Configure the pool’s daemon connection with Zebra’s RPC address and credentials. Confirm that the pool handles the supplied transaction formats, coinbase, commitment roots, long polling, and NU7 fee split. Pool accounting and payments to individual workers are the pool’s responsibility, not Zebra’s.
See Getting new work to mining pools for long polling, block notifications, separate payout wallets, and recovery planning. The s-nomp Testnet setup is experimental.
Testnet, Regtest, and the internal miner
For public Testnet, set [network] network = "Testnet", use a Testnet reward
address, and set [rpc] listen_addr = "127.0.0.1:18232". Public Testnet uses
real Equihash proof of work and the same upgrade-dependent coinbase rules.
For deterministic local block production, use the Regtest generation RPCs, not the experimental internal miner.
Getting new work to mining pools
First configure and sync Zebra for mining, including the miner address and authenticated RPC access. Then choose a new-work delivery method supported by your pool software, in this order:
- Long polling: hold a
getblocktemplaterequest until work changes. - Block notifications: trigger a template request when Zebra’s best chain tip changes.
- Polling: periodically request a template as a fallback or backstop.
With a polling interval of I, a new block arriving uniformly between polls leaves miners on stale work for I / 2 on average, before RPC and job-distribution delays. For example, 500 ms polling adds about 250 ms of stale work per block. Notifications remove that polling wait, not all delivery delay; this is a latency estimate, not a guaranteed revenue gain.
Prefer getblocktemplate long polling
The pool’s template provider must explicitly implement long polling; receiving a longpollid does
not enable it automatically.
-
Fetch an initial template without a long-poll ID:
{"jsonrpc":"2.0","id":1,"method":"getblocktemplate","params":[{}]} -
Publish the template to miners, then make another request with its returned
longpollid:{ "jsonrpc": "2.0", "id": 2, "method": "getblocktemplate", "params": [{"longpollid": "<exact ID from the previous response>"}] } -
Keep that request open. When it returns, process the template and repeat using its new ID.
Treat the ID as opaque: do not parse it, assume its length, or construct it yourself. Keep one outstanding long poll per backend, with a separate connection or connection-pool capacity for block submissions. Configure client and proxy timeouts for long-held requests, rather than a short ordinary RPC timeout.
A response can reflect a new chain tip, changed transactions, or an expired timestamp range. Ensure
the pool forwards updated work to miners even when previousblockhash is unchanged. Honor
submitold when deciding whether to retain previous work; do not discard it merely because the
transaction selection changed. After a timeout or connection failure, reconnect with bounded retry
backoff, fetch a fresh template without an ID, and restart the long-poll loop.
Use block notifications
s-nomp commands
For a pool that supports a block-notification command but not long polling, add this to Zebra’s configuration and restart the node:
[notify]
block_notify_command = "node /path/to/s-nomp/scripts/cli.js blocknotify <coin> %s"
Before using this example, replace the script path and <coin> with the pool’s configured coin name.
The CLI connects to s-nomp’s local command listener by default, so this example assumes it is
reachable from the Zebra host. Ensure the service account running Zebra can execute Node.js and read
the script. The command runs through the system shell; keep its configuration under operator control.
Zebra substitutes the new best-tip block hash for %s. The pool uses the notification to request a
fresh template and distribute new jobs; the notification itself is not a template. Commands run
asynchronously, only when Zebra is close to the network tip. Delivery is best-effort: intermediate
tips can be skipped during rapid changes, and notifications are not a replayable event log.
Keep periodic polling enabled as a backstop for missed notifications or a restarted pool. Block notifications only signal tip changes, so retain the pool’s periodic job refresh for transaction-only updates too. See the s-nomp testnet guide for its compatibility and payment limitations; this command does not make an unsupported pool version production-ready.
ZMQ subscribers
Pools that subscribe to zcashd’s ZMQ hashblock topic, such as Miningcore, can use Zebra’s ZMQ
publisher. Build Zebra with --features zmq, then configure:
[notify]
zmqpubhashblock = "tcp://127.0.0.1:28332"
Notifications start when Zebra is close to the network tip. Delivery is best-effort: use the sequence numbers to detect gaps and reconcile through RPC after gaps or reconnects. Keep periodic polling as a backstop; coalesced intermediate tips do not necessarily create sequence gaps.
The endpoint has no authentication; expose it only to trusted clients. See ZMQ block notifications for framing and resource limits.
Resolve equal-work tip races
When backends choose different tips with equal cumulative work, call preciousblock with the same
known, non-finalized tip hash on each backend to align the fleet:
{"jsonrpc":"2.0","id":1,"method":"preciousblock","params":["<preferred non-finalized tip hash>"]}
Among equal-work tips the last call wins, but more-work chains always take precedence, so a call on a
block with less work than the best tip is a no-op. The preference covers only that block while it is
a tip: once a block is added on top of it, the extended chain no longer carries it. If the race
continues at the next height with equal work, call preciousblock again on each backend with your
new tip. Known non-tip or finalized blocks also return null without changing the preference;
unknown hashes return Block not found. Preferences are in memory only and are forgotten on restart.
Keep mining and payouts separate
Zebra validates blocks and serves mining work; it is not a wallet. Point the pool’s mining RPC connection at Zebra, and its payout or payment-processor connection at a compatible wallet that controls the reward address. Configure the same intended reward address on each mining backend.
Use Zebra’s validateaddress or z_validateaddress to check address validity, not wallet ownership.
The transparent-address response omits ismine; a valid shielded-address response reports
ismine: false. Check ownership with the payout wallet instead.
Wallet RPC compatibility
As of 27 October 2026, no released Zallet direct-state backend reads this Zebra’s database; see the database compatibility table. The beta.3 Zaino backend can fetch over JSON-RPC without reading that database, but that topology is untested here. The payout RPC gaps below apply regardless of backend.
This inventory compares s-nomp at 88187c9 and
Miningcore at a553f62 with
Zallet 0.1.0-beta.3 at 987382f and the
zcashd-compat sidecar, release zebra-compat-v1.3.0.
| Processor | Wallet calls | Zallet | Sidecar |
|---|---|---|---|
| Both | gettransaction, z_getbalance | Absent; use z_viewtransaction, z_getbalanceforaccount | Available; enable allowdeprecated=z_getbalance |
| Both | z_shieldcoinbase, z_sendmany, z_getoperationresult | Available; sending has altered semantics | Available for transparent/Sapling flows |
| s-nomp | getbalance, listunspent, sendmany | Absent; account balances, z_listunspent, z_sendmany replace them | Available |
| s-nomp | validateaddress, z_validateaddress | Transparent validation omits required ismine; shielded method absent | Available with ownership information |
| s-nomp | getaccount | Absent | Absent; this processor ignores its failed result |
| s-nomp | z_getoperationstatus | Available | Available |
| Miningcore | listunspent (fallback shielding), walletpassphrase, walletlock | First absent; lock/unlock available | Available; lock/unlock need an encrypted wallet, whose creation is experimental and disabled by default |
The processors also query chain RPCs: s-nomp uses getblock, getmininginfo, and
getnetworkinfo; Miningcore uses getblockchaininfo. Those are not Zallet wallet methods.
The sidecar serves them. Miningcore constructs its payout client from the first daemon entry;
it does not provide s-nomp’s separate paymentProcessing.daemon connection.
Miningcore remains inspected-only. Its hard-coded 0.0001 ZEC fee failed a separate shielding
probe, and its sends need an explicit privacy policy for transparent recipients. Adapting its fee
selection and privacy policy remains untested. The tested s-nomp changes below do not enable
legacy privacy behavior.
Tested sidecar payout path
An isolated Regtest run mined through s-nomp, recorded real miner shares in Redis, and let
libs/paymentProcessor.js shield rewards, unshield into its payout address, and pay the miner.
The test used:
| Component | Version or revision |
|---|---|
| Zebra | 7.0.0+210.gf94b0b5, revision f94b0b5e5819, archive mode |
| Sidecar | Unmodified zebra-compat-v1.3.0, revision 5c11db4f4fbc |
| s-nomp | 0.0.9, base 88187c9, with the changes below |
| node-stratum-pool | 0.2.0, revision 2a0572caa96b, preserving Zebra’s coinbase |
| nheqminer | Revision 72e320aacaa4, two CPU threads, Equihash 200,9 |
| Runtime | Node.js 8.17.0, Redis 7.2.6 |
Apply these changes to libs/paymentProcessor.js. They track the shielding operation ID, reserve
the configured unshielding fee and permit the transparent staging recipient:
@@ -275 +275 @@
- var opid = (result.response || result[0].response);
+ var opid = (result.response || result[0].response).opid;
@@ -294 +294 @@
- if ((zBalance - 10000) <= 0)
+ if ((zBalance - coinsToSatoshies(fee)) <= 0)
@@ -303 +303 @@
- var amount = satoshisToCoins(zBalance - 10000);
+ var amount = satoshisToCoins(zBalance - coinsToSatoshies(fee));
@@ -305 +305,2 @@
- var params = [poolOptions.zAddress, [{'address': poolOptions.tAddress, 'amount': amount}]];
+ var params = [poolOptions.zAddress, [{'address': poolOptions.tAddress, 'amount': amount}],
+ minConfShield, fee, 'AllowRevealedRecipients'];
- Run Zebra and an externally managed sidecar.
Pin the sidecar’s only peer with
connect=<Zebra-P2P-address>andlisten=0; keep both RPC endpoints on loopback with separate credentials. Set[zcashd_compat] enabled = trueandmanage_zcashd = falseon Zebra. The sidecar synchronizes over P2P and does not read Zebra’s database, so the database-format mismatch does not affect it. - Create two distinct wallet-controlled transparent addresses with
getnewaddressand a Sapling address withz_getnewaddress sapling. Enableallowdeprecated=getnewaddress,allowdeprecated=z_getnewaddress, andallowdeprecated=z_getbalancein the sidecar’szcash.conf. Verify ownership with the wallet’svalidateaddressandz_validateaddress. Set the reward address as[mining] miner_addresson Zebra andaddressin s-nomp; set the other transparent address astAddressand the Sapling address aszAddress. - Point s-nomp’s
daemonsat Zebra andpaymentProcessing.daemonat the sidecar’s authenticated RPC endpoint. EnablepaymentProcessing.enabledand usepaymentMode: "prop",minConf: 10,paymentInterval: 30,minimumPayment: 0.05, andwalletInterval: 1. In the coin profile, retainrequireShielding: trueandtxfee: 0.0004. KeeprewardRecipients: {}andpoolHex: "": this Stratum adapter preserves Zebra’s coinbase rather than inserting pool-fee outputs. The test used a share difficulty of1 / 1024, disabled the website and clustering, and enabledcleanJobsOnTemplateUpdatefor nheqminer. - Connect the miner with its transparent payout address as the username. Let the processor
observe maturity through the sidecar, complete shielding and unshielding, and call
sendmany. Check its Redis payment record, confirm that transaction on Zebra, and verify the recipient’s output amount. The configured fee covers the tested small withdrawal, not arbitrary batches; size fees under ZIP 317. The privacy policy reveals recipient addresses and amounts.
For reproduction, the run activated Overwinter through NU5 at height 1 on both nodes, left later
upgrades inactive, and used fresh state with no funding streams. Zebra used network = "Regtest",
rpc.debug_force_finished_sync = true, rpc.enable_cookie_auth = false on loopback only, and
mempool.debug_enable_at_height = 0. The sidecar used regtest=1 and six nuparams entries:
5ba81b19:1, 76b809bb:1, 2bb40e60:1, f5b9230b:1, e9ff75a6:1, and c2d6d0b4:1.
It also set regtestshieldcoinbase=1, regtestacceptunvalidatedpow=1, and maxtipage=2000000000.
These test-only auth, sync, and PoW settings must not be copied to public networks.
The pool mined height 1. After stopping the miner, Zebra generated 101 blocks to mature its reward,
then ten blocks for each processor-created transaction. The test waited for matching node tips and
the sidecar’s getblockchaininfo.fullyNotified after each batch. Shielding spent the pool coinbase
at height 103; unshielding confirmed at 113; the processor’s 6.2496 ZEC miner payment confirmed
at 123 and had ten confirmations at 132. The Redis payment record credited the real height-1
round, and the recipient UTXO matched its amount. Additional generated rewards shared the wallet;
the test did not fabricate shares or issue shielding, unshielding, or payment calls itself.
This proves one worker’s proportional accounting and scheduled payout, not production readiness. It does not test multiple miners, pool fees, restart or reorg recovery, encrypted wallets, production proof of work, or upgrade transitions. The legacy Node.js version is unsupported. Keep payouts transparent/Sapling: the sidecar never supports Ironwood and rejects Orchard from NU6.3 onward.
Plan for restarts and replacement
- Keep a separately synced standby node with the same network and mining settings, its own state directory, and its own RPC endpoint and credentials. Test the pool’s backend failover before an outage; a reachable RPC port alone does not establish that the node can provide current work.
- Before a planned restart, move mining traffic to the standby. After startup, wait for sync and a fresh successful template before returning traffic. Re-read RPC cookie credentials after a restart when cookie authentication is enabled, and restart long polling with a fresh template.
- After a pool restart, request current work immediately rather than waiting for the next block notification. On backend changes, discard the old backend’s long-poll ID and start a new loop.
- Use the published state snapshots to bootstrap a replacement faster than syncing from genesis. Follow that guide’s network, database-version, checksum, and extraction instructions, then let the node catch up before serving miners. Snapshots require trusting their producer’s validation; checksums detect transfer corruption, not a malicious snapshot. Downloading, extracting, and catching up still take time, so snapshots complement a warm standby rather than replace one.
Monitor mining
Use the mining metrics to watch template build time, new-tip to first-template latency, submission outcomes, and mined-block announcement failures and latency for stale-block risk. Announcement completion measures local writes, not peer receipt or network-wide propagation.
Mining with Zebra in Docker
Configure a reward address in the Zebra image, synchronize the node, then connect a pool and external solver as described in Mining.
Mainnet node
Set MINER_ADDRESS in the shell to a supported
Mainnet reward address controlled by the recipient.
docker run -d --name zebra-mainnet \
-e ZEBRA_NETWORK__NETWORK=Mainnet \
-e "ZEBRA_MINING__MINER_ADDRESS=${MINER_ADDRESS:?Set MINER_ADDRESS}" \
-e ZEBRA_RPC__LISTEN_ADDR=0.0.0.0:8232 \
-p 8233:8233 \
-p 127.0.0.1:8232:8232 \
-v zebra-mainnet-cache:/home/zebra/.cache/zebra \
zfnd/zebra:7.0.0
Keep RPC published only on host loopback unless a protected network or proxy controls access. Never share writable state between running nodes; see Docker paths and permissions.
Inspect node output with:
docker logs --follow zebra-mainnet
Wait for a mining template before starting a pool.
RPC authentication
Request a template from the host using the container’s RPC cookie:
cookie=$(docker exec zebra-mainnet cat /home/zebra/.cache/zebra/.cookie)
curl --fail-with-body --silent --show-error \
--user "$cookie" \
--header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":1,"method":"getblocktemplate","params":[]}' \
http://127.0.0.1:8232/ | jq
unset cookie
Configure the pool to reload these credentials when Zebra restarts. Treat them as a password. For container-to-container RPC and isolated deployments without authentication, follow Docker RPC.
Testnet node
Set MINER_ADDRESS to a Testnet address, then use separate container and
volume names:
docker run -d --name zebra-testnet \
-e ZEBRA_NETWORK__NETWORK=Testnet \
-e "ZEBRA_MINING__MINER_ADDRESS=${MINER_ADDRESS:?Set a Testnet MINER_ADDRESS}" \
-e ZEBRA_RPC__LISTEN_ADDR=0.0.0.0:18232 \
-p 18233:18233 \
-p 127.0.0.1:18232:18232 \
-v zebra-testnet-cache:/home/zebra/.cache/zebra \
zfnd/zebra:7.0.0
Use zebra-testnet and port 18232 in the authentication commands. Public
Testnet requires real proof of work; use Regtest for local
block-generation tests.
Configuration overrides
Use environment overrides for the
mining fields. Docker’s --env-file option replaces
individual -e options when maintaining these values in a file.
To use a TOML file, mount it read-only and set CONFIG_FILE_PATH to its path
inside the container:
docker run -d --name zebra-configured \
--mount "type=bind,source=$PWD/zebrad.toml,target=/etc/zebra/zebrad.toml,readonly" \
-e CONFIG_FILE_PATH=/etc/zebra/zebrad.toml \
-v zebra-configured-cache:/home/zebra/.cache/zebra \
zfnd/zebra:7.0.0
Configure the file’s network, mining address, and listeners, and publish only the ports required for that deployment. See mounted configuration for file loading.
Repository mining Compose stack
The repository’s
docker/mining/docker-compose.yml
builds Zebra and the s-nomp pool, starts Redis 7, and defines an optional
nheqminer container under the miner profile. Its defaults are Testnet,
internal RPC port 18232, P2P port 18233, Stratum port 3333, and pool web port
8080. Stratum is the protocol between the pool and its mining workers.
The stack does not publish Zebra’s RPC or Redis ports to the host.
Copy docker/mining/.env.example to docker/mining/.env and set
MINER_ADDRESS to a reward address for the selected network. Optional
EXTRA_COINBASE_DATA supplies literal UTF-8 data, at most 86 bytes, through
ZEBRA_MINING__EXTRA_COINBASE_DATA.
The Zebra service enables its health listener at 127.0.0.1:8080 without
publishing it. S-NOMP waits for /healthy to succeed before starting.
A successful health check does not replace the mining sync check.
The stack disables RPC authentication inside its Docker network. Do not publish that unauthenticated RPC port. The pool and miner build contexts use moving upstream branches rather than pinned commits. Its s-nomp payment limitations also apply.
Use the s-nomp Testnet procedure for an isolated deployment with compatibility workarounds, independent of this Compose file. Do not use this experimental stack as a production pool.
Mining on Testnet with s-nomp
The Zcash Foundation’s s-nomp fork connects Equihash miners to Zebra through
Stratum, a protocol for distributing mining jobs and collecting solutions.
This is an experimental Testnet setup, not a production pool: custom
coinbase construction and pool-fee outputs are disabled, and this procedure
disables worker payments. All miner rewards go to [mining] miner_address
in Zebra. A worker’s Stratum username does not select a payment recipient.
Compatibility and limitations
The public fork’s default branch is zebra-mining. Its package version is
0.0.9; the source revision used below is
49fd89bf.
Use Node.js 10.24.1 and Python 3.10, selected by the fork’s container recipe,
not the README’s Node.js 8.11. Node.js 12 and later have incompatible Equihash
native-module APIs.
The pool’s stratum-pool dependency passes Zebra’s
coinbasetxn.data and coinbasetxn.hash
through unchanged. Its
block assembly
forwards the supplied transactions and uses the legacy finalsaplingroothash
field, which Zebra fills with the current block commitment. This bypasses the
obsolete custom coinbase builder. It provides neither a current transaction
parser nor a payment implementation.
The fork supplies pool_configs/zcash_testnet.json for native use. Its
container entrypoint generates a separate configuration but omits two
settings required for this deployment:
- The missing
switchingobject causesObject.keys(portalConfig.switching)to fail at startup. - Its generated Stratum port omits
tls: false. The dependency selects TLS unless that field is explicitly false, so a plain-TCP miner cannot use the generated default.
Use the fork’s container build with Zebra’s
s-nomp-entrypoint.sh.
The wrapper supplies both settings. The
repository Compose stack
has separate Zebra startup defects.
Prepare the sources
Use a Linux x86-64 host with Docker, Git, and jq. Keep this old pool runtime
on a dedicated Docker network. Do not expose its Redis, RPC, web interface,
or Stratum listener to untrusted clients.
Clone the sources into a working directory:
git clone --branch v7.0.0 --depth 1 https://github.com/ZcashFoundation/zebra.git zebra
git clone --branch zebra-mining https://github.com/ZcashFoundation/s-nomp.git s-nomp
git -C s-nomp checkout --detach 49fd89bf32fac6fe2440143d16622650e061e5a0
In s-nomp/package.json, pin the stratum-pool dependency:
"stratum-pool": "git+https://github.com/upbqdn/node-stratum-pool.git#7fbb0685f8895abf1b746d6682de30dd814b9382"
Build the pool image:
docker build -f s-nomp/Dockerfile.arch -t zebra-s-nomp s-nomp
The Dockerfile installs Boost, libsodium, GMP, Redis, and native build tools,
sets CXXFLAGS=-std=gnu++17, and patches the RPC transport. Other npm
dependencies and the Arch base image remain unpinned; this is not a
reproducible dependency build.
The transport patch adds "jsonrpc":"2.0" to the startup batch, as required by
Zebra’s batch protocol. A plain
npm install leaves that batch incompatible.
Install, run, and sync Zebra
Set MINER_ADDRESS in the shell to a transparent Testnet address that you
control. This recipe also uses it for s-nomp’s transparent-address startup
check, even though Zebra itself supports additional
miner address kinds.
Create a dedicated network and start Zebra:
docker network create zebra-mining
docker run -d --name zebra-testnet \
--network zebra-mining \
-e ZEBRA_NETWORK__NETWORK=Testnet \
-e "ZEBRA_MINING__MINER_ADDRESS=${MINER_ADDRESS:?Set a Testnet MINER_ADDRESS}" \
-e ZEBRA_RPC__LISTEN_ADDR=0.0.0.0:18232 \
-e ZEBRA_RPC__ENABLE_COOKIE_AUTH=false \
-p 18233:18233 \
-v zebra-testnet-mining-cache:/home/zebra/.cache/zebra \
zfnd/zebra:7.0.0
The pool’s generated daemon credentials are the fixed strings x and x;
it does not load Zebra’s cookie. This example therefore disables RPC
authentication but does not publish port 18232. Attach only trusted
containers to this network. The authenticated setup in
Mining with Docker is preferable for
pool software that supports it.
Inspect synchronization:
docker logs --follow zebra-testnet
Before starting s-nomp, request a template from inside the node container:
docker exec zebra-testnet curl --fail-with-body --silent --show-error \
--header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":1,"method":"getblocktemplate","params":[]}' \
http://127.0.0.1:18232/ | jq
Wait for a result containing a template, not an RPC error; see the
sync requirements. The supplied coinbase handles
NU7 funding-stream and NSM fee accounting.
Run s-nomp
Run these commands from the directory containing the zebra and s-nomp
checkouts:
docker run -d --name s-nomp-testnet \
--network zebra-mining \
-e NETWORK=Testnet \
-e ZEBRA_HOST=zebra-testnet \
-e ZEBRA_RPC_PORT=18232 \
-e STRATUM_PORT=3333 \
-e "POOL_ADDRESS=${MINER_ADDRESS:?Set a Testnet MINER_ADDRESS}" \
-p 127.0.0.1:3333:3333 \
-p 127.0.0.1:8080:8080 \
--mount "type=bind,source=$PWD/zebra/docker/mining/s-nomp-entrypoint.sh,target=/app/s-nomp-entrypoint.sh,readonly" \
--entrypoint /bin/bash \
zebra-s-nomp /app/s-nomp-entrypoint.sh
docker logs --follow s-nomp-testnet
The entrypoint generates /app/config.json and
/app/pool_configs/zcash.json, selects coins/testnet/zec.json, disables
payment processing, starts Redis on local port 6379, and runs node init.js.
The pool polls for block templates every 500 milliseconds. Its website is
available on the host at http://127.0.0.1:8080. The pool’s startup message
Stratum Pool Server Started indicates that it has created a job and started
the listener; it does not establish that the pool has mined an accepted block.
The entrypoint overwrites generated configuration on each start. Do not edit those files expecting the changes to survive a container restart.
Install and run a CPU miner
The Zcash Foundation’s nheqminer fork supplies a Dockerfile for a CPU-only
build with the Xenoncat solver. This solver requires an x86-64 CPU with AVX
support; it selects AVX2 when available. The following revision is from the
zebra-mining branch:
git clone --branch zebra-mining https://github.com/ZcashFoundation/nheqminer.git nheqminer
git -C nheqminer checkout --detach 72e320aacaa49c9e170be62d28a4189957767d3c
docker build -t zebra-nheqminer nheqminer
docker run -d --name nheqminer-testnet \
--network zebra-mining \
zebra-nheqminer \
-l s-nomp-testnet:3333 \
-u "${MINER_ADDRESS:?Set a Testnet MINER_ADDRESS}.worker1" \
-t 1
docker logs --follow nheqminer-testnet
The miner’s -t 1 option selects one CPU thread. A native Equihash miner on
the same host can instead connect to 127.0.0.1:3333. Do not use port 18232
as a Stratum endpoint; it belongs to Zebra’s JSON-RPC server.
A share accepted by the pool is not necessarily a block: it can meet the
pool’s easier share target without meeting the network target. Check Zebra’s
submitblock outcome and best chain when evaluating a mined block.
Diagnose failures
| Symptom | Check |
|---|---|
Missing config.json or no enabled pools | Start through the entrypoint; npm start alone does not generate configuration. |
| Unauthorized RPC access | The generated x credentials do not authenticate against Zebra’s default cookie authentication. Do not solve this by exposing unauthenticated RPC. |
Error -10 from getblocktemplate | Wait for Zebra to synchronize. |
Mining-parameters error from getblocktemplate | Check Zebra’s reward address, selected network, and memo length. |
| Block rejected after changing coinbase or transactions | Restore the supplied template or implement current coinbase, commitment, and NU7 rules. |
Website logs Could not dumpprivkey | The website calls this unsupported wallet RPC even with payments disabled. Zebra holds no wallet keys; this error concerns the website’s mining-key page, not block-template generation. |
Stop the experiment
Stop the miner before the pool and node:
docker stop --timeout -1 nheqminer-testnet s-nomp-testnet zebra-testnet
The Zebra state volume remains available for reuse. The pool recipe does not mount a persistent Redis volume; do not treat its share statistics as durable payment records.
Custom Testnets
A custom Testnet changes Zebra’s network identity or consensus parameters without changing public Testnet. Use Regtest for a new local chain, or fork public Testnet to reuse existing blocks and balances.
Configuration
Set [network] network = "Testnet" and place overrides in
[network.testnet_parameters]; omit the parameter table for public Testnet.
With the public activation schedule, supply checkpoints through Canopy minus
one: the configured network’s genesis-only default fails the coverage check.
Use checkpoints = true when retaining the public chain, or a
bounded checkpoint list for a fork.
The following configuration defines an accelerated, proof-of-work-disabled
network. Save it as custom-testnet.toml:
[network]
network = "Testnet"
listen_addr = "127.0.0.1:18233"
initial_testnet_peers = []
cache_dir = false
[network.testnet_parameters]
network_name = "ConfiguredTestnet_1"
network_magic = [0, 1, 0, 255]
slow_start_interval = 0
disable_pow = true
funding_streams = []
lockbox_disbursements = []
[network.testnet_parameters.activation_heights]
Canopy = 1
NU5 = 2
NU7 = 9
[mining]
miner_address = "t27eWDgjFYJGVXmzrXeVjnb5J3uXDM9xH9v"
[mempool]
debug_enable_at_height = 0
[rpc]
listen_addr = "127.0.0.1:18232"
cookie_dir = "./custom-testnet-rpc"
[state]
cache_dir = "./custom-testnet-state"
Replace the example miner address with an address you control when testing spends. Start Zebra with:
zebrad -c custom-testnet.toml start
Unlike Regtest, a custom Testnet does not insert a genesis block at startup.
Provision the matching genesis block before requesting a block template: submit
its hex-encoded serialized bytes with submitblock, obtain it from a configured
peer, or copy compatible state. This example retains public Testnet’s genesis
hash but has no peers. It therefore needs an externally supplied genesis block.
Zebra does not construct new genesis blocks from a configured hash.
After genesis is present, use the
generation RPCs to produce blocks.
The example’s cookie file is ./custom-testnet-rpc/.cookie. Disabling proof
of work skips Equihash and block-hash target checks, not target validity or
other consensus rules.
Parameter reference
See the network configuration reference for parameter names, types, defaults, input formats, and selector layouts. The rules below govern combinations of parameters.
Activation heights
Use the case-sensitive keys in the activation field reference, quoting names containing dots. Heights must be nondecreasing in upgrade order; equal heights are allowed. An omitted earlier upgrade activates with the next explicitly configured upgrade. Upgrades after the last configured upgrade remain inactive. Height zero is reserved for genesis.
A present activation table replaces all public activation heights. For example,
NU6 = 1 activates every upgrade through NU6 at height 1 and leaves later
upgrades inactive. Do not supply an empty schedule: Zebra requires Canopy and
its predecessors for state verification and block-template generation.
For a fork, retain every historical public Testnet activation explicitly.
Regtest fills omitted early upgrades through Canopy from its defaults:
Overwinter defaults to BeforeOverwinter or height 1, and each upgrade
through Canopy defaults to the preceding one. Later omitted upgrades activate
with the next configured upgrade, or remain inactive if none follows.
Subsidy schedule and NU7
The subsidy slow start and halvings follow the
protocol subsidy rules.
The block target spacing is 150 seconds before Blossom, 75 seconds from
Blossom, and 25 seconds from NU7 under
ZIP 218. Spacing is not independently
configurable. The post-Blossom halving interval is twice
pre_blossom_halving_interval; the post-NU7 interval is six times it. A halving
height is not necessarily a multiple of the interval because the schedule
accounts for upgrade boundaries and the slow-start shift.
For custom Testnets, Zebra validates that:
- The first halving height is representable and Blossom is scheduled.
- The halving index at the slow-start shift is nonnegative.
- Scheduled issuance through height
2^31 - 1, excluding the unspendable genesis subsidy, does not exceed 21 million coins. - A nonempty funding-stream schedule has a nonzero address-change interval, calculated as the post-Blossom halving interval divided by 48.
- With NU7 scheduled, the post-NU7 halving interval is positive and no greater
than
6_931_680_000, so the integer NSM reissuance coefficient is nonzero.
Early spacing upgrades during slow start can violate these bounds. Set
slow_start_interval = 0 for accelerated chains such as the example above.
The configuration accepts activation heights through 2^31 - 1, but the
current on-disk height encoding supports only 2^24 - 1.
The Network Sustainability Mechanism (NSM) reserve is separate from issued
supply. Zebra seeds it at NU7 and applies the
mining fee split. Without NU7, no NSM reissuance
height is calculated. Unless overridden, reissuance starts at the
ZIP 237 crossover, not automatically at NU7;
accelerated schedules can have no crossover. An explicit
nsm_reissuance_height requires NU7 and must not precede it.
An explicit initial reserve equal to the public default preserves public Testnet compatibility; a different reserve requires distinct magic and private peers. A private chain copied from public Testnet must explicitly retain the public historical seed when changing its magic. See Testing NU7 reissuance.
Funding streams and lockbox
Funding streams allocate percentages of the block subsidy to named recipients; the deferred recipient accrues value in the lockbox rather than a coinbase output. Their protocol definitions are in ZIP 207, ZIP 214, and ZIP 1015. The one-time NU6.1 lockbox payout is specified by ZIP 271.
Use complete funding-stream entries for private chains:
- Nonempty height ranges must not overlap.
- Use each receiver once per range; Zebra stores recipients in a map keyed by receiver. The effective numerators must sum to at most 100.
- Non-deferred recipients need Testnet P2PKH or P2SH addresses covering every period in the range. Address extension repeats an existing list; it cannot fill a required empty list.
Missing height_range or recipients inherits the public Testnet entry at
that list index: pre-NU6, post-NU6, then post-NU6.1. Additional entries must
specify both fields. Zebra prepends pre_nu6_funding_streams, then
post_nu6_funding_streams, before applying positional defaults. The legacy
post-NU6 table always inherits post-NU6 defaults, even without a pre-NU6 table.
Neither legacy table can accompany funding_streams = [].
Inherited post-NU6.1 funding end heights adjust to NU7; explicit height ranges do not. Public funding addresses are consensus data, not operator payouts.
To test a deferred allocation, remove funding_streams = [] from the example
and append this complete entry after its existing sections. Keep
lockbox_disbursements = [] in the parent parameter table:
[[network.testnet_parameters.funding_streams]]
height_range = { start = 9, end = 30 }
recipients = [{ receiver = "Deferred", numerator = 12 }]
Zebra rejects malformed funding addresses, reversed ranges, excessive
numerator sums, missing positional defaults, and insufficient or wrong-network
addresses during configuration loading, on both Testnet and Regtest. It also
checks lockbox addresses and rejects a total above 2100000000000000 zatoshis.
The chain must still have enough lockbox value for the configured payout.
Checkpoints
A checkpoint binds a height to an expected block hash. Zebra requires checkpoint verification for the blocks before Canopy because it does not fully validate all pre-Canopy rules. A custom checkpoint list must:
- Contain height zero with the configured genesis hash.
- Use unique heights and unique, nonzero block hashes.
- Keep heights at or below
2^31 - 1. - Reach at least Canopy activation minus one.
Use the checkpoint input formats in the field reference. Relative file paths resolve from Zebra’s working directory. With Canopy at height 1, the default genesis-only list is sufficient. For a fork, use checkpoints only on the shared prefix; never include public hashes at or after divergence. Do not edit Zebra’s bundled checkpoint file.
Regtest does not run the full custom-Testnet checkpoint-coverage validation. Acceptance of its configuration is not proof that the resulting blocks can be mined or verified.
Peer isolation and storage
Use distinct magic for every incompatible network and configure only its peers
in [network] initial_testnet_peers. Zebra rejects public DNS seed hostnames
for incompatible custom Testnets regardless of case, trailing dots, or port.
An empty peer list does not waive the distinct-magic requirement. Checkpoints
and the display name are not consensus-compatibility tests.
Network magic is not authentication, and initial peers are not an allowlist. Restrict inbound P2P access with a firewall when the network is private. Bind JSON-RPC to loopback or a protected interface and retain cookie authentication.
Zebra names custom storage namespaces
<ascii-lowercased-name>-<eight-digit-hex-magic>. For the example, the name is
configuredtestnet_1-000100ff. State, non-finalized backups, and peer caches use
this namespace; Mainnet, public Testnet, and Regtest retain mainnet, testnet,
and regtest. Old unsuffixed custom directories are not loaded automatically.
Use a compatible prefix or resynchronize; never reuse blocks already validated
under different consensus rules. See
Forking the Zcash Testnet with Zebra for copying state.
Genesis transparent outputs are unspendable and excluded from issued supply. Rebuild custom state created by an implementation that counted nonzero genesis transparent outputs, including experimental v29 state. A database-format upgrade does not recalculate those balances.
Differences from Mainnet and Regtest
Custom Testnets keep Testnet address encodings and coin type, the Testnet minimum-difficulty rule, and the Testnet maximum-time-rule activation height. These are not TOML parameters. In particular, the minimum-difficulty rule changes at NU7 as specified in ZIP 218, and the maximum-time rule applies from height 653,606 as specified in the block-header rules. Changing the genesis hash and activation schedule alone does not reproduce Mainnet consensus.
See Regtest for its fixed parameters and supported overrides.
Regtest with Zebra
Regtest provides a local chain for testing block production, transactions, and network upgrades. Zebra inserts the Regtest genesis block automatically, skips Equihash and block-hash proof-of-work checks, and enables the mempool from genesis. Other consensus checks, including target validity, still apply. Unlike Zebra, stock zcashd Regtest validates proof of work; do not assume the two nodes accept the same generated blocks.
Usage
Save the following configuration as regtest.toml:
[network]
network = "Regtest"
listen_addr = "127.0.0.1:18233"
initial_testnet_peers = []
cache_dir = false
[mining]
miner_address = "t27eWDgjFYJGVXmzrXeVjnb5J3uXDM9xH9v"
[state]
ephemeral = true
[rpc]
listen_addr = "127.0.0.1:18232"
cookie_dir = "./regtest-rpc"
Replace the example miner address with an address controlled by your test wallet if you need to spend the rewards. Start Zebra:
zebrad -c regtest.toml start
The example’s ephemeral state is deleted on a clean exit.
For a persistent chain, remove ephemeral = true and set [state] cache_dir
to a dedicated directory. If consensus changes affect existing blocks, start
again from a compatible earlier state, not the existing tip.
The RPC examples read ./regtest-rpc/.cookie; keep it private. See
RPC authentication.
Defaults and overrides
See the network field reference for Regtest defaults and
supported overrides, and Ports for listener addresses.
Zebra rejects conflicting overrides of fixed Regtest consensus and networking
parameters and names the field in the configuration error. For example,
disable_pow = false is an error; disable_pow = true is accepted. Matching
fixed values can be omitted. Regtest ignores network_name; its name remains
Regtest, so older saved configurations still load.
Use a custom Testnet to change those parameters.
Set should_allow_unshielded_coinbase_spends = false to require shielding
coinbase spends. The option is the inverse of zcashd’s
-regtestshieldcoinbase; it does not remove coinbase maturity checks.
Activation heights
Add an activation table to enable later upgrades. For example:
[network.testnet_parameters.activation_heights]
NU5 = 1
Follow the activation and inheritance rules. Keep Canopy at height 1 for a chain mined entirely by Zebra: it cannot construct pre-Canopy templates. Regtest validates funding addresses and lockbox disbursements during configuration loading, as on custom Testnets. Its reduced checkpoint validation does not guarantee that a loaded configuration produces valid blocks.
Using RPC methods directly
The generate RPC mines the requested number of blocks immediately using
null proof-of-work solutions and returns their hashes. It uses
[mining] miner_address and requires neither the internal-miner feature
nor an external miner:
curl --silent --show-error \
--user "$(cat ./regtest-rpc/.cookie)" \
--header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":1,"method":"generate","params":[1]}' \
http://127.0.0.1:18232/
The generatetoaddress RPC uses an address supplied with the call, without
changing the configured miner address:
curl --silent --show-error \
--user "$(cat ./regtest-rpc/.cookie)" \
--header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":2,"method":"generatetoaddress","params":[1,"t27eWDgjFYJGVXmzrXeVjnb5J3uXDM9xH9v"]}' \
http://127.0.0.1:18232/
Both methods also accept custom Testnets with disable_pow = true; the
implementation rejects networks with proof of work enabled.
For tests that modify block contents, obtain a template with
getblocktemplate, construct and serialize the block, and pass its hexadecimal
encoding to submitblock. A successful submission returns a JSON-RPC result
of null. The public
zebra_rpc::proposal_block_from_template
helper takes the template, a block-template time source, and a reference to the
actual configured network. Do not pass Network::Mainnet for Regtest.
Submit signed transactions with sendrawtransaction; Zebra verifies them
before including them in templates.
Testing NU7 reissuance
Use the NSM parameters to test reserve reissuance independently of NU7 activation.
Use a fresh chain and add these sections to the base configuration, replacing any existing activation-height section:
[network.testnet_parameters]
initial_nsm_value_balance = 100_000_000
nsm_reissuance_height = 12
[network.testnet_parameters.activation_heights]
Canopy = 1
NU5 = 2
NU7 = 9
The example activates NU6 through NU6.3 together with NU7 at height 9 and seeds the reserve with one coin. At height 12, the additional block subsidy begins drawing from the parent block’s reserve.
Mine through these boundaries with generate. Inspect getblocktemplate,
getblocksubsidy, and getblockchaininfo; subsidy calculation after reissuance
requires the parent block’s reserve, so getblocksubsidy cannot answer an
arbitrary future height whose parent is absent.
Follow Upgrading for an existing installation. Database-format compatibility does not make changed consensus parameters compatible.
Using Zebra’s internal miner
The experimental internal miner runs continuously. Compile Zebra with the
internal-miner feature and add
internal_miner = true to the existing [mining] section, retaining
miner_address:
cargo build --release --locked --bin zebrad --features internal-miner
./target/release/zebrad -c regtest.toml start
The miner calls the block-template implementation internally, without an HTTP
RPC listener. Its single solver computes Equihash (200, 9) solutions and
checks their hash targets even on Regtest, unlike zcashd’s easier Regtest
Equihash parameters. Initial-template retries wait 20 seconds. Use generate
for controlled block production rather than waiting for a solution.
Connecting local nodes
Regtest skips public DNS seeders, ignores the peer cache when loading initial
peers, and retains only initial addresses equal to 127.0.0.1 or ::1.
Explicit localhost peers are supported; Regtest is not a no-network mode.
For a second node, use a different P2P port, RPC port, cookie directory, and
state directory. Set its [network] initial_testnet_peers to
["127.0.0.1:18233"] to reach the example node. Use identical consensus
parameters on both nodes. Check connections with getpeerinfo and chain
agreement with getbestblockhash.
Keep the P2P listener on loopback for local tests. Regtest still creates an inbound listener; its network magic is not an access-control mechanism.
Experimental early block relay
For private relay experiments, set experimental_early_block_relay = true in
[network] on each participating Zebra node. It defaults to false and requires
both ends of a connection to advertise the experimental capability during the
handshake. Peers that have not opted in retain ordinary full-block relay.
After NU5, opted-in nodes can advertise and serve an authenticated child of the committed best tip before transaction verification finishes. Full validation is still required before committing it. The cache retains one body for at most 15 seconds, with a 2 MB wire limit and a conservative 64 MiB decoded-content charge. Rejection, cancellation, or expiry withdraws the body; after withdrawal, the admission slot stays reserved until its deadline or the next committed tip to bound replayed advertisements. Locally invalidated hashes are ineligible. At most eight authenticated, deterministic rejection hashes are remembered for 60 seconds to suppress repeated expensive verification. Unauthenticated variants, timeouts, and infrastructure failures do not populate this set. Ordinary committed announcements are unchanged. Compatible peers do not punish one another for later transaction validation failures after authenticating the body against a current committed parent. If that context is unavailable, including historical or pipelined downloads, verification continues. Before assigning a punitive later failure, authentication retries for at most 30 seconds without publication; unavailable context remains non-punitive, while a failed authentication is scored. Withdrawn bodies are rechecked before their local wire write begins, falling back to committed state if necessary. Bytes already sent cannot be recalled.
These connections also negotiate provisional compact encoding version 1 with
sendcmpct. At most three recipients per announcement receive cmpctblock;
other connections receive ordinary inventory and full blocks. Up to three opted-in
peers are selected as compact receive sources, preferring first announcers only
after their block is committed locally. Unvalidated inventory cannot change this
selection, and a stalled demotion write cannot hold a source slot.
The compact encoding includes the complete Zcash header, a nonce, six-byte SipHash-2-4 short
IDs, and a prefilled coinbase. Short IDs use the complete witnessed identity for
v5/v6 transactions and the transaction ID for earlier versions.
Reconstruction uses the verified mempool, then one getblocktxn/blocktxn
exchange for missing transactions. Collisions, commitment mismatches, unknown
parents, and a two-second recovery timeout fall back to a full block without
punishing the announcing peer. Repeated recovery responses share a block-size
payload budget for 15 seconds after delivery, even if publication permission is withdrawn.
The deadline also bounds backpressured recovery writes, releasing the globally
bounded snapshot before connection cleanup. The normal consensus verifier still
checks every reconstructed block. Coinbase-only blocks use full relay because
compact encoding would not save bytes.
This uses experimental service bit 25 from the temporary-experiment range inherited from Bitcoin. Bit 25 avoids bit 24, which another implementation already uses for a different capability. This is not a production protocol allocation; Mining Zcash with Zebra covers its use on mining nodes. Regtest’s disabled proof of work does not measure the cost of authenticated admission on a public network.
Forking the Zcash Testnet with Zebra
A private fork reuses a public Testnet prefix, then applies different consensus rules without connecting to public peers. Zebra can configure the activation heights of implemented upgrades; adding a new consensus rule still requires a modified implementation. Changing a network name or branch identifier alone does not implement an upgrade.
Use Regtest instead when the test does not require public history. For a coordinated multi-implementation test, also follow Private Testnet testing.
Choose the shared prefix
Choose the first height at which the private rules differ, called the fork height. Keep the genesis block and all consensus parameters through the preceding block compatible with public Testnet. Retain historical activation heights, subsidy and funding schedules, and lockbox disbursements. No copied block may be at or beyond the first incompatible height.
The example replays NU7 at height 4,200,000 using public Testnet blocks through 4,199,999, before public NU7 activation. For a different cutoff, update the state, checkpoints, and activation table together. NU7 requires a height divisible by three under ZIP 259.
Record the prefix’s final height and block hash and share them with every participant. A different hash at the same height means the participants are not starting from the same chain.
Prepare public state
Use a dedicated public Testnet configuration, for example public-testnet.toml:
[network]
network = "Testnet"
[state]
cache_dir = "/srv/zebra-public"
delete_old_database = false
Start Zebra and synchronize until the chosen prefix is finalized:
zebrad -c public-testnet.toml start
Stop Zebra and other processes accessing this state before preparing the copy. Keep an untouched backup. The paths in this procedure must be writable by the Zebra user and must not contain another test’s state.
Prepare the snapshot with a compatible writer; follow Upgrading before copying. Do not rename version directories to bypass an upgrade.
Copy a bounded prefix
The expert copy-state command copies finalized blocks through a specified
source height into another database. It does not read the source’s
non-finalized backup. It uses the source network for both databases; only
[state] options from its target configuration take effect. It does not
convert public Testnet state directly into a custom network namespace or
verify a new consensus implementation.
Save prefix-target.toml with a new, persistent destination:
[state]
cache_dir = "/srv/zebra-fork-prefix"
delete_old_database = false
Copy the example prefix:
zebrad -c public-testnet.toml copy-state \
--max-source-height 4199999 \
--target-config-path prefix-target.toml
With --target-config-path, ZEBRA_TARGET_STATE__CACHE_DIR overrides the
target cache directory without changing the source’s configuration.
The height is inclusive. The command stops at the source’s finalized tip if it is lower than the requested height, so check the resulting prefix rather than assuming it reached the cutoff:
zebrad tip-height --network Testnet --cache-dir /srv/zebra-fork-prefix
For this example the result must be 4199999. The copy log also reports
final_target_tip, including its hash; compare that hash across participants.
The command commits copied blocks as checkpoint-verified blocks, rebuilding
state under the source network. It is not a replacement for verifying the
source chain.
The prefix is stored at /srv/zebra-fork-prefix/state/v29/testnet. Copy it
into an empty private namespace:
mkdir -p /srv/zebra-nu7-private/state/v29
test ! -e /srv/zebra-nu7-private/state/v29/nu7private-00010007 && \
cp -a /srv/zebra-fork-prefix/state/v29/testnet \
/srv/zebra-nu7-private/state/v29/nu7private-00010007
This bounded copy finalizes every copied block, so it does not need a public non-finalized backup. Never add a backup containing public blocks beyond the cutoff.
Copy an already bounded snapshot
If the stopped source already ends below divergence, an offline directory
copy avoids replaying every block. Copy its finalized database and its matching
non_finalized_state/testnet directory, when present, into empty destinations
with the private namespace:
| Source below the public cache root | Destination below the private cache root |
|---|---|
state/v29/testnet | state/v29/nu7private-00010007 |
non_finalized_state/testnet | non_finalized_state/nu7private-00010007 |
A non-finalized backup preserves recent blocks that are not yet in the finalized database. It can lag the live tip, so verify the restored tip before testing. Do not recover missing prefix blocks by reconnecting the private network to public seeds. Obtain a complete agreed prefix before isolation. Never copy a public peer cache into the private namespace or copy private state back into public Testnet.
Prepare private checkpoints
The checkpoint coverage rules require this prefix to reach at least height 1,028,499. A checkpoint at or after divergence would require the private chain to match the public block it replaces.
From the Zebra source checkout, extract only checkpoints below the example fork height:
awk '$1 < 4200000' \
zebra-chain/src/parameters/checkpoint/test-checkpoints.txt \
> /srv/zebra-nu7-private/checkpoints.txt
Share the exact file with all participants. Do not use checkpoints = true:
a later Zebra release could bundle public checkpoints beyond the cutoff.
Configure the isolated network
Save the following as private-testnet.toml:
[network]
network = "Testnet"
listen_addr = "127.0.0.1:18233"
initial_testnet_peers = []
cache_dir = false
[network.testnet_parameters]
network_name = "Nu7Private"
network_magic = [0, 1, 0, 7]
checkpoints = "/srv/zebra-nu7-private/checkpoints.txt"
initial_nsm_value_balance = 55_768_414_957
[network.testnet_parameters.activation_heights]
BeforeOverwinter = 1
Overwinter = 207_500
Sapling = 280_000
Blossom = 584_000
Heartwood = 903_800
Canopy = 1_028_500
NU5 = 1_842_420
NU6 = 2_976_000
"NU6.1" = 3_536_500
"NU6.2" = 4_052_000
"NU6.3" = 4_134_000
NU7 = 4_200_000
[mining]
miner_address = "t27eWDgjFYJGVXmzrXeVjnb5J3uXDM9xH9v"
[mempool]
debug_enable_at_height = 0
[rpc]
listen_addr = "127.0.0.1:18232"
cookie_dir = "/srv/zebra-nu7-private/rpc"
[state]
cache_dir = "/srv/zebra-nu7-private"
delete_old_database = false
The custom namespace is
nu7private-00010007, not the public testnet directory.
The explicit NSM seed preserves
the public history after changing network magic.
For multiple hosts, bind P2P to a private interface and replace
initial_testnet_peers = [] with participant addresses and ports. Use the same
magic and consensus settings everywhere, with the
peer isolation rules.
The [mempool] debug_enable_at_height = 0 setting enables the mempool on the
old copied tip, not mining. RPC remains authenticated and bound to loopback.
Mine and inspect the fork
Start the private node:
zebrad -c private-testnet.toml start
Confirm the restored tip and connected peers before producing blocks:
curl --silent --show-error \
--user "$(cat /srv/zebra-nu7-private/rpc/.cookie)" \
--header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":1,"method":"getblockchaininfo","params":[]}' \
http://127.0.0.1:18232/
curl --silent --show-error \
--user "$(cat /srv/zebra-nu7-private/rpc/.cookie)" \
--header 'Content-Type: application/json' \
--data '{"jsonrpc":"2.0","id":2,"method":"getpeerinfo","params":[]}' \
http://127.0.0.1:18232/
The sample retains proof of work. Use a mining pool or the experimental internal miner. Meet the mining synchronization requirement before requesting templates; an old Testnet tip alone does not fail that check.
For tests excluding proof of work, set
[network.testnet_parameters] disable_pow = true on every participant and use
the generation RPCs. This does not
test difficulty or Equihash validation.
After activation, compare the height and best-block hash across nodes. Inspect
the upgrade status and consensus branch in getblockchaininfo, then test the
new rules with transactions and blocks. Use the implemented upgrade’s branch
identifier, not an arbitrary replacement. For NU7, the identifier is
0x77190ad9, specified in ZIP 259.
Zero-knowledge proof parameters
A proving key creates zero-knowledge proofs; a verifying key checks them. The protocol specification defines the proof systems used by each shielded protocol.
Zebra 7.0.0 includes or derives the parameters it needs. Running Zebra does not require a parameter-download command or a separate parameter directory. This does not change the parameter requirements of a separately installed wallet or zcashd sidecar.
Groth16: Sapling and Sprout
The Groth16 proof system uses three circuit-specific parameter sets: Sapling Spend, Sapling Output, and Sprout JoinSplit after Sapling activation. A JoinSplit is Sprout’s shielded transfer description. The protocol specifies the parameter-file hashes in Groth16 zk-SNARK parameters. These parameters come from a multiparty trusted setup; operators do not create a new setup for their nodes.
Zebra obtains its Groth16 parameters as follows:
| Circuit | Source in Zebra | Use |
|---|---|---|
| Sapling Spend and Output | The zcash_proofs crate’s bundled-prover feature embeds parameter data through wagyu-zcash-parameters. Zebra constructs a shared LocalTxProver::bundled() and obtains its verifying keys. | Verification of Sapling bundles; the same prover constructs Sapling outputs for coinbase transactions. |
| Sprout-on-Groth16 JoinSplit | The file zebra-consensus/src/primitives/groth16/sprout-groth16.vk is compiled into the binary with include_bytes!. | Verification of version 4 Sprout JoinSplit proofs; Zebra does not load the full Sprout proving-parameter file. |
The Sapling prover and Sprout verifying key initialize independently on first use and remain shared for the process lifetime.
Zebra uses the sapling-crypto batch validator for Sapling bundles and the
groth16 crate for Sprout proof verification. Sprout JoinSplit proofs are
verified individually, not in batches. The presence of Groth16 proofs depends
on the transaction’s shielded components, not merely its version number.
Historical Sprout BCTV14 proofs
Sprout transactions before Sapling activation use
BCTV14 proofs, not Groth16.
Zebra does not implement BCTV14 proof verification or load the corresponding
Sprout setup parameters. It verifies this chain history against mandatory
checkpoint hashes instead. These checkpoints remain active even when
[consensus] checkpoint_sync = false.
The protocol explicitly permits omitting BCTV14 verification when checkpointing after Sapling activation. Zebra’s transaction verifier rejects version 1–3 transactions outside the checkpoint path. A historical block accepted by checkpoint verification is not evidence that Zebra checked each proof within it.
Halo 2: Orchard and Ironwood
Orchard uses Halo 2, which does not require a trusted setup. ZIP 224 explains the distinction from Groth16. The Ironwood shielded pool also uses the Halo 2 Action circuit, as specified by ZIP 229.
Zebra builds verifying keys from the circuit definitions in the orchard
crate rather than downloading parameter files. The
Halo 2 verifier
keeps separate keys for the pre-NU6.2, NU6.2, and NU6.3-onward circuits. It
selects the Orchard key by the block’s network upgrade, not by transaction
version alone. Historical keys remain necessary for verifying older blocks;
Ironwood uses the NU6.3-onward key.
Elasticsearch block export
The optional elasticsearch Cargo feature exports newly finalized blocks
as JSON documents to Elasticsearch. Kibana can query those documents through
an Elasticsearch data view. This is separate from Prometheus metrics
and does not replace Zebra’s RocksDB chain state.
This experimental exporter has failure modes that lose batches or interrupt node operation. Use a separate node rather than making production depend on it.
Compatibility and security
Zebra pins the Rust Elasticsearch client to 8.17.0-alpha.1. That is the
client library version, not a verified server-version requirement. The
repository does not specify a tested Elasticsearch/Kibana server-version
matrix. Do not assume compatibility with an arbitrary current server release.
Install and configure the services using the
Elasticsearch documentation
and Kibana documentation.
Zebra connects to one configured URL using HTTP Basic authentication. HTTPS validates the server certificate and hostname by default. Configure Elasticsearch with a certificate signed by a CA trusted by the client’s bundled root store. Zebra has no configuration field for a custom CA or certificate pinning; adding a CA to the operating system’s trust store is not sufficient.
For a trusted local cluster with a self-signed certificate, set
[state] elasticsearch_skip_cert_validation = true. This disables certificate
validation and exposes credentials and indexed data to interception. Leave it
false for network-facing clusters, and use HTTPS: HTTP does not protect
credentials regardless of this setting.
The password remains plaintext in TOML despite being redacted from debug output.
Store the configuration file where only the Zebra account can read it, or mount
it through the deployment’s secret mechanism. The environment override
ZEBRA_STATE__ELASTICSEARCH_PASSWORD is rejected.
Build and configure Zebra
From a source checkout, install with the
elasticsearch feature:
cargo install --locked --path zebrad --features elasticsearch
zebrad generate -o zebrad-elasticsearch.toml
Set the [state] Elasticsearch fields for an account
authorized to create and write the target index. A build with this feature
exports automatically during normal startup; there is no runtime enable switch.
For a Testnet experiment, edit the generated file’s sections:
[network]
network = "Testnet"
listen_addr = "[::]:18233"
[state]
cache_dir = "./zebra-elasticsearch-state"
elasticsearch_url = "https://localhost:9200"
elasticsearch_username = "elastic"
# Only for a local cluster with a self-signed certificate.
elasticsearch_skip_cert_validation = true
Set [state] elasticsearch_password in the configuration file before starting
Zebra. Use a dedicated empty [state] cache_dir if the index must include
blocks from genesis. Enabling the feature on an existing state database
exports only blocks finalized afterwards; it does not replay stored blocks.
Do not delete an existing production state to populate an experimental index.
Start the node:
zebrad -c zebrad-elasticsearch.toml start
Indexed data and failure behavior
The exporter runs after a block commits to RocksDB and serializes its Rust representation, not the JSON-RPC response format. It excludes non-finalized blocks.
Zebra constructs the index name as zcash_ followed by the lowercase network
name: zcash_mainnet on Mainnet and zcash_testnet on the default Testnet.
Each document’s _id is the decimal block height. Indexing that height again
replaces the existing document.
During historical sync, Zebra sends batches of 24 blocks through the Elasticsearch bulk API. When the current block’s header timestamp is less than four hours behind the local clock, it sends the accumulated batch on each finalized block instead. This batching heuristic is independent of the node’s readiness checks.
The export path has these limitations:
- A failed ping request logs
Elasticsearch is not available, skipping block indexingand discards the queued batch. There is no retry queue. - A failed bulk request, an unreadable response, or a response reporting bulk errors triggers a panic rather than a recoverable indexing error.
- Export runs synchronously in the finalized-block commit path. Elasticsearch delays therefore delay that commit path.
- A partially filled batch has no shutdown flush or persistent storage. A restart does not reconstruct it from blocks already in RocksDB.
An Elasticsearch index can therefore have gaps even when Zebra’s own state is complete. Monitor exporter errors and verify index completeness before using it for analysis that requires every block.
Query with Kibana
After Elasticsearch accepts the first batch, create a Kibana data view for
the generated index, such as zcash_testnet. The documents include the
block header’s header.time field. Select it as the time field if the
Elasticsearch mapping exposes it as a date, and use a time range that includes
the historical blocks being indexed.
Zebra does not provision Kibana, data views, index mappings, or dashboards.
Troubleshooting
Use the first error and the startup context, not only the process exit code. Record the Zebra version, network, effective configuration, selected state path, and the last height that advanced. The startup log includes build and state-format metadata; see Startup and sync.
Configuration does not load or take effect
| Symptom | Cause | Remedy |
|---|---|---|
Zebra could not load the provided configuration file and/or environment variables | Unknown fields or invalid TOML/environment values | Compare the reported field with the configuration reference and a generated reference file. |
| A file change has no effect | Zebra loaded a different file | Check Using config file at:; pass --config /path/to/zebrad.toml before start to select it explicitly. |
| A loaded file’s value has no effect | An environment override replaces it | Check environment variables, including service-manager and container settings. |
| An environment variable is rejected as sensitive | The field forbids environment overrides | Set it in the configuration file. |
| The P2P port stays unchanged after switching networks | An explicit [network] listen_addr retains its port | Remove that setting or change its port. |
Generate the reference without overwriting the active file:
zebrad generate --output-file zebrad-defaults.toml
A listener cannot bind
The bind error names the endpoint and requested address; the P2P message starts
with Opening Zcash network protocol listener.
For an address-in-use error, identify the conflicting listener on Linux:
ss -ltnp
Stop the conflicting process or choose a distinct address for that endpoint; each listener has its own port setting.
For an unavailable-address error, choose an address assigned to the host or container. Without IPv6 support, use an IPv4 listener:
[network]
listen_addr = "0.0.0.0:8233"
This example listens on all IPv4 interfaces for Mainnet; use the network’s P2P port or an explicit alternate port. Do not expose administrative endpoints to work around a bind error.
The database is locked or cannot open
Database likely already open: another writer or an I/O error prevents opening state. Check services, containers, and manually started nodes. Stop the existing writer or give the second node a distinct[state] cache_dirand listeners. If no writer exists, check permissions and storage health. Do not delete RocksDB’sLOCKfile; wait for the owner to exit before modifying state.Permission denied creatingorOpening database: check access to the parent directories and write permission on the cache root. In containers, check the mounted volume and effective UID/GID; see Docker ownership handling.
Disk space is exhausted
The error No space left on device means that a write or directory creation
cannot complete. On Linux, check both capacity and available inodes on the
filesystem containing the cache. Replace the path below with an existing
cache directory:
df -h /path/to/zebra-cache
df -i /path/to/zebra-cache
Stop Zebra before moving its database. Free unrelated files or move the cache
to storage with sufficient free space, then update [state] cache_dir and any
environment or mount overrides. Do not delete individual RocksDB files to
make room: they form one database.
Keep room for growth and maintenance; snapshot restoration also needs space for both the archive and extracted state.
Sync is slow or stalled
The syncer retries downloads and sync rounds automatically; waiting to restart sync does not request a process restart. Check current_height and
time_since_last_state_block for continued progress.
For initial sync can't download and verify the genesis block or
chain updates have stalled, match the accompanying evidence:
| Symptom | Cause | Remedy |
|---|---|---|
| No successful peer activity | Peer connectivity is failing | Check DNS errors, resolved seed peer IP addresses, configured seed addresses, and outbound connectivity. |
| Block-header times are rejected | A header is more than two hours ahead of the node’s clock, violating the protocol rule | Correct the time source if the clock is skewed; do not disable validation. Clock skew also distorts progress estimates. |
| Storage errors or resource saturation accompany the stall | The node cannot sustain downloads, verification, or state writes | Check capacity and permissions; compare system requirements and limit contention. |
Zebra refuses to run | The Mainnet release has reached end of support | Upgrade; restarting the same binary does not help. |
| A restored node starts from genesis | Zebra did not open the intended snapshot | Follow snapshot troubleshooting before allowing another full download. |
Limit resource contention
For CPU or logging contention, use a release build and remove temporary debug filters. In builds that retain debug instrumentation, restrict it to the component under investigation:
[tracing]
filter = "info,zebra_network=debug"
Default release builds compile out debug and trace events; a filter cannot restore them. See compiled log levels.
Queued blocks consume memory, and concurrent verification competes for CPU. Choose sync concurrency limits for measured capacity rather than raising every limit.
For a bandwidth-constrained node, reduce download concurrency before changing verification limits:
[sync]
download_concurrency_limit = 25
Measure height advancement and resource use before and after a change; use metrics. Zebra raises limits below the minimum bounds and logs the adjustment.
Linux TCP tuning for block propagation
Linux enables net.ipv4.tcp_slow_start_after_idle by default. It reduces a TCP
connection’s congestion window after an idle period defined by the current
retransmission timeout. When peers are idle between block requests, this
restart can limit subsequent block-transfer throughput. Zebra checks the
setting on Linux and warns at startup when it is nonzero.
Inspect the effective setting:
sysctl net.ipv4.tcp_slow_start_after_idle
To retain the congestion window across idle periods, set:
sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0
This setting affects all TCP users in the network namespace, not only Zebra.
Review other workloads before changing it. To persist it for the host, put
this line in /etc/sysctl.d/99-zebra-network.conf:
net.ipv4.tcp_slow_start_after_idle=0
Apply persistent host settings with sudo sysctl --system. See the
Linux TCP sysctl reference for the kernel behavior.
Running Zebra in Docker
Apply the setting in the network namespace containing Zebra. With Docker’s
separate container networking, add
--sysctl net.ipv4.tcp_slow_start_after_idle=0 to the docker run options.
With --network=host, configure the host instead; Docker does not allow
network sysctl overrides in host-network mode. Changing only the host setting
is not a substitute for configuring an existing separate container network
namespace. See Docker’s sysctl rules.
Known limitation: private peer addresses
Zebra does not reject private IP addresses merely because they are private. It can connect to them and distribute peer addresses; filtering them by default remains tracked in issue #3117.
Use network-level egress controls if the node must not reach internal services. Do not rely on the public Zcash network configuration to isolate a node from private address ranges.
Build and test resource failures
For compilation memory exhaustion, reduce Cargo build parallelism:
export CARGO_BUILD_JOBS=2
This limits build jobs, not node threads. For test resource contention, limit nextest’s test concurrency separately:
cargo nextest run --locked --test-threads=2
For CI’s nextest profiles, follow continuous integration.
For unreliable external connectivity, set SKIP_NETWORK_TESTS=1 to skip tests
that use Zebra’s network-test guard. It does not skip
every network-dependent test or disable IPv6 tests. Unset it before claiming
network-test coverage.
Reporting a persistent failure
Include the recorded version, network, configuration, first error, and surrounding logs in a bug report. Redact RPC cookies, credentials, wallet information, and private infrastructure details before publishing logs. For suspected vulnerabilities, follow the security policy instead of opening a public issue.
Developer documentation
These pages are for contributors; the API reference covers individual
Rust APIs. The site root documents the latest release; the book of the main
branch is published under /main/.
- Design: architecture (crates, services, block and transaction flow), mempool specification, state database format. The RFCs are the original design records, annotated with where the code has diverged.
- Contributing: the contribution guide; property testing, profiling, tokio-console, private testnet tests.
- Maintenance: consensus refactors, mass renames, ECC dependency updates, checkpoints.
- Process: CI, delivery, releases, audits, changelogs, labels, crates.
Contributing
Running and debugging
Use the user documentation to build, configure, run, and instrument Zebra. Use the developer documentation for architecture and testing.
Bug reports
Submit reproducible defects through the bug report form. Include
expected and observed behavior, reproduction commands, relevant logs, operating
system details, and the output of zebrad --version.
Report vulnerabilities through the security disclosure process, not
public issues. Test against the latest release or current main, and identify
the exact version or commit.
Pull requests
External contributors must discuss changes before opening a pull request (PR). Maintainers do not need prior issue approval for their own work.
- Find an existing issue, or open one explaining the problem and proposed change. Wait for a Zebra team member’s acknowledgment before coding.
- Agree on the approach for changes beyond small bug fixes. Discuss the overall plan before splitting work across related PRs.
- Keep each PR to one logical change. Link the agreed issue and distinguish a complete fix from partial work or a dependency on another PR.
- Follow Conventional Commits for the PR title and branch commits. Zebra uses merge commits, so both remain in history.
- Add tests for changed behavior and the required change fragments. Review every changed line and run the applicable local checks before opening the PR.
Zebra validates and synchronizes the Zcash chain. Wallet functionality, block explorers, and mining-pool management belong in downstream projects, not in the node. The help wanted and good first issue labels identify work open to contributors; they do not waive prior discussion.
PR description
Follow the PR template:
- Summary: what changes and why, in at most three sentences or four short bullets.
- Reviewer Notes: design choices, migration requirements, or specifications absent from the diff. Omit when unnecessary.
- Evidence: before-and-after behavior, a regression test, real output, or a screenshot. Do not repeat checks CI reports. Omit for nonbehavioral changes.
- Merge Danger: identify the door and blast radius. A change is one-way when a revert cannot undo external effects: database upgrades, publication, deployment, or deleted infrastructure. Consensus, peer protocol, public crate API, configuration, RPC, and metrics changes are two-way only until release.
- AI Disclosure: follow AI-assisted contributions, or state that no AI tools were used.
Breaking changes
Put ! after the conventional-commit type and optional scope in both the PR
title and the commit that introduces a break, for example
feat(zebra-chain)!: change the public API. Follow the
changelog requirements for affected operators and library consumers.
The PR gate fails api-diff for undeclared API breaks and requires
successful analysis for declared breaks. Its api-diff and changelog-gate
jobs are advisory, not merge-blocking; fragment parsing and release readiness
are separate checks. Reviewers must check compatibility and changelog coverage.
The release planner reads commits on main, including branch commits and the
merge commit. A breaking marker only in the PR description is insufficient.
AI-assisted contributions
Disclose AI-generated or AI-revised code, tests, and prose in the PR description. Specify what the tool did and what you reviewed. Tab completion, spell checking, and syntax highlighting do not require disclosure.
Explain the logic and trade-offs of every change, including generated work. Review it and run the same checks required for human-written changes.
When we close PRs
Maintainers can close PRs that lack the required prior discussion, introduce unrequested features or refactors, make unsolicited minor formatting changes, lack evidence for changed behavior, or fall outside Zebra’s scope. Inability to explain a submitted change is also grounds for closure. Discuss the need in an issue before resubmitting.
Code standards
AGENTS.md states the full architecture, testing, and contribution
rules; in short:
- Keep crate dependencies consistent with the existing architecture. Keep
zebra-chainsynchronous. - Use structured errors. Explain the invariant behind each
expect(). - Move CPU-heavy work out of async executor threads; bound external waits.
- Bound allocations and work driven by untrusted input.
- Document public APIs and changed operator behavior.
Create change fragments with the source change, not by editing generated changelogs. For operator-visible changes:
changie new -j zebrad
For library-consumer-visible changes, select each affected project:
changie new -j zebra-chain -j zebra-state -k breaking
Use separate bodies when audiences or migration requirements differ.
Local checks
Install the native build dependencies, Rust toolchains,
cargo-nextest, and cargo-hack required by the jobs you reproduce. From the
repository root, run the main formatting, lint, and unit-test checks:
cargo +1.91.0 fmt --all -- --check
cargo +1.97.0 clippy --workspace --all-targets \
--features default-release-binaries -- -D warnings
cargo +1.97.0 clippy --workspace --all-targets \
--features 'default-release-binaries proptest-impl lightwalletd-grpc-tests zebra-checkpoints' \
-- -D warnings
CARGO_PROFILE_RELEASE_LTO=false cargo +1.97.0 nextest run \
--profile ci --locked --release --features default-release-binaries \
--run-ignored=all
cargo +1.91.0 hack check --workspace
The ci nextest profile excludes stateful, end-to-end, and no-Git-dependency
release tests. For per-crate feature combinations, minimum Rust versions,
dependencies, documentation, and integration tests, follow the applicable
workflows and CI guide.
Resolve a do-not-merge hold before removing the label; see the
label rules.
Architecture
Zebra separates protocol data, networking, consensus verification, chain state,
and RPC into Rust crates. The zebrad application connects their
services and runs synchronization and the mempool. A crate dependency is not
necessarily a service call.
Crates
The root workspace contains these crates:
| Crate | Responsibility | Manifest |
|---|---|---|
zebra-chain | Blocks, transactions, addresses, amounts, network parameters, serialization, and cryptographic data types. | Cargo.toml |
zebra-network | Peer connections, wire messages, address discovery, inventory tracking, and outbound request routing. | Cargo.toml |
zebra-script | Transparent script verification through libzcash_script; the default interpreter is C++, with an optional C++/Rust comparison interpreter. | Cargo.toml |
zebra-consensus | Checkpoint verification, semantic block verification, block and mempool transaction verification, and cryptographic verification services. | Cargo.toml |
zebra-state | Contextual verification, finalized and non-finalized chain storage, chain queries, and tip notifications. | Cargo.toml |
zebra-node-services | Shared mempool request/response types, service traits, and an optional RPC client. It does not implement the mempool. | Cargo.toml |
zebra-rpc | JSON-RPC methods and server, RPC client, indexer gRPC server, and lightwalletd-compatible gRPC server. | Cargo.toml |
zebrad | CLI and configuration, service assembly, synchronization, inbound requests, mempool, monitoring, notifications, and optional zcashd supervision. | Cargo.toml |
tower-batch-control | Tower middleware that collects weighted requests and flushes batches by size or latency. | Cargo.toml |
tower-fallback | Tower middleware that retries a request on a second service when its fallback policy selects it. Consensus uses it for individual verification after a batch fails. | Cargo.toml |
zebra-utils | Developer tools, including zebra-checkpoints, search-issue-refs, and block-template-to-proposal. | Cargo.toml |
zebra-test | Shared test harnesses, test vectors, service mocks, and test initialization. | Cargo.toml |
The zebra-fuzz/ directory contains the separate, unpublished
zebra-fuzz-targets workspace. Its executable targets exercise
transaction and block decoding, consensus checks, scripts, P2P messages, and RPC.
It is not a root-workspace member or a node runtime component.
The zebra-scan and zebra-grpc crates were removed in
Zebra 2.4.0; their names do not refer to the current gRPC servers.
Crate dependencies
An arrow points from a crate to an in-repository crate it depends on, as
declared in the manifests above. A crate also depends on every crate reachable
from it, so the graph omits each edge that a longer path already implies: for
example, zebrad and zebra-rpc both declare zebra-chain directly. The
dotted arrow is an optional dependency. The graph omits external crates and
zebra-test, which every crate in the graph except zebra-node-services and
zebra-utils uses as a development dependency, and which zebra-chain and
zebra-state also pull in optionally for their proptest-impl features (and
zebra-chain for its bench feature).
flowchart TD
daemon["zebrad"] --> rpc["zebra-rpc"]
daemon -.-> utils["zebra-utils"]
utils --> rpc
rpc --> consensus["zebra-consensus"]
rpc --> network["zebra-network"]
consensus --> state["zebra-state"]
consensus --> script["zebra-script"]
consensus --> batch["tower-batch-control"]
consensus --> fallback["tower-fallback"]
state --> interfaces["zebra-node-services"]
interfaces --> chain["zebra-chain"]
network --> chain
script --> chain
The fuzz targets, in their own workspace, depend on zebra-chain,
zebra-consensus, zebra-network, zebra-node-services, zebra-rpc,
zebra-script, and zebra-state.
The direct zebrad dependency on zebra-script forwards the optional
comparison-interpreter feature.
The daemon does not call the library directly.
Service dependencies
A Tower service accepts typed requests and returns response futures. The
start command wraps shared services in bounded buffers and connects them
with cloned handles. The consensus router initializer constructs two
transaction verifier services: BlockTxVerifier for transactions in blocks and
MempoolTxVerifier for unmined transactions. The latter can query the mempool for
outputs created by unconfirmed parent transactions.
Arrows below point from a caller to a service it uses, omitting middleware and download queues. Peer connections call the inbound service; the peer set routes outbound requests to those connections. The state exposes read/write and read-only services over the same database.
flowchart TD
network["NetworkService / PeerSet"] --> connection["peer::Client / Connection"]
connection --> inbound["Inbound"]
inbound --> network
inbound --> mempool["Mempool"]
inbound --> router["BlockVerifierRouter"]
inbound --> state["StateService"]
sync["ChainSync"] --> network
sync --> router
sync --> state
sync --> read["ReadStateService"]
router --> checkpoint["CheckpointVerifier"]
router --> semantic["SemanticBlockVerifier"]
checkpoint --> state
semantic --> state
semantic --> blocktx["BlockTxVerifier"]
blocktx --> state
mempool --> network
mempool --> state
mempool --> mempooltx["MempoolTxVerifier"]
mempooltx --> state
mempooltx --> mempool
server["RpcServer"] --> rpc["RpcImpl"]
rpc --> mempool
rpc --> state
rpc --> read
rpc --> router
indexer["indexer::server::init"] --> read
lightwallet["lightwalletd::server::init"] --> rpc
lightwallet --> read
lightwallet --> mempool
The startup function also connects background tasks and optional endpoints. Arrows carry notifications or requests. The mempool’s background tasks appear in the mempool diagram.
flowchart LR
state["StateService"] --> tip["LatestChainTip / ChainTipChange"]
sync["ChainSync"] --> status["SyncStatus"]
tip --> mempool["Mempool"]
tip --> progress["show_block_chain_progress"]
status --> progress
progress --> metrics_state["ChainTipMetrics"]
metrics_state --> health["health::init"]
status --> health
addresses["AddressBook"] --> health
tip --> block_gossip["gossip_best_tip_block_hashes"]
status --> block_gossip
submitted["SubmitBlockChannel"] --> block_gossip
block_gossip --> advertiser["BlockAdvertiser"]
tip --> notify["notify::run_block_notify"]
status --> notify
tip --> support["end_of_support::start"]
tip --> indexer["indexer::server::init"]
mempool -->|MempoolChange| indexer
tip --> lightwallet["lightwalletd::server::init"]
mempool -->|MempoolChange| lightwallet
addresses -->|P2P listener address| supervisor["zcashd_compat::run_supervisor"]
application["zebrad::application"] --> metrics["MetricsEndpoint"]
state -.->|metrics recorder| metrics
mempool -.->|metrics recorder| metrics
The application registers MetricsEndpoint outside
StartCmd::start. For activation conditions, see the
configuration reference and
end-of-support policy.
Verification stages
Zebra separates checks to overlap independent work without accepting a block before its ancestors. The stages follow RFC 0002:
- Structural verification parses protocol bytes into valid data structures.
The
zebra-chaindeserializers enforce representation constraints, including transaction encodings and bounded fields. - Semantic verification checks a block or transaction’s internal rules, signatures, proofs, and scripts, overlapping with parent-block verification. Script checks still need spent outputs. The transaction verifiers obtain them and assemble concurrent checks; the semantic block verifier combines their results.
- Contextual verification checks the parent, height, difficulty, spent outputs, nullifiers, commitments, and value-pool changes against a particular chain. The state write path applies these checks, including the protocol’s block-header rules, before committing to non-finalized state.
For transaction admission without a block commit, see mempool contextual checks.
Checkpoint and full verification
A checkpoint is a configured block hash at a known height. The
CheckpointVerifier checks linked ranges against the checkpoint
list instead of individually verifying their transaction signatures and proofs.
It still checks block heights, proof of work where enabled, Merkle roots, and
transaction-hash uniqueness. The finalized-state commit path checks
the chain-history and authorizing-data commitments before storing these blocks;
a matching checkpoint hash alone does not authorize arbitrary transaction data.
The router selects checkpoint verification at or below its maximum checkpoint height and full semantic verification above it. The maximum is either the last checkpoint or the first at or above the mandatory height, as set by consensus configuration. Mandatory checkpoints cannot be disabled: Zebra cannot fully verify pre-Canopy history.
At startup, the consensus initializer also compares cached best-chain hashes with the checkpoint list, stopping at the first height not yet stored. This runs regardless of checkpoint-sync configuration. A mismatch panics.
Finalized and non-finalized state
The state implementation maintains two parts:
- Finalized state stores one chain in RocksDB. Checkpoint-verified blocks go directly here, in chain order.
- Non-finalized state holds recently verified competing chains in memory, ordered by accumulated work. A full-verification commit first enters this state. The write worker finalizes the oldest block of the best chain whenever that chain exceeds the rollback window.
The rollback window is 1,000 blocks, defined by
MAX_BLOCK_REORG_HEIGHT. It is Zebra’s local policy, not a Zcash
consensus rule. Reorganizations cannot replace blocks already finalized by this
node.
With backups enabled, persistent state restores recent blocks from a non-finalized backup; see the state settings.
The StateService interface accepts mutations and queries.
The ReadStateService interface processes ReadRequest queries without sharing
the write service’s request buffer. Both read the same finalized database and
published non-finalized state. The LatestChainTip and ChainTipChange handles
publish tip information without a state-service round trip.
Block flow
- The
ChainSynctask discovers block hashes through peers. TheInboundservice also accepts block advertisements. Their respective sync downloader and inbound downloader request blocks through the network service. - The peer connection decodes wire messages into
zebra-chaindata. The downloaders submitzebra_consensus::Request::Committo the block verifier router. - The checkpoint path validates a hash-linked range and submits
CommitCheckpointVerifiedBlock. The full path runsSemanticBlockVerifierandBlockTxVerifier, then submitsCommitSemanticallyVerifiedBlock. The checkpoint and semantic verifiers return success only after their state commit succeeds. - The state service queues out-of-order blocks until their parents are available. Its write worker commits checkpoint blocks to finalized state, or contextually verifies full-path blocks into non-finalized state and later finalizes them at the rollback limit.
- The state publishes the changed best tip. The block gossip task
sends its hash through
BlockAdvertiser; RPC and other tip subscribers can observe the committed chain. A successful full-path commit does not mean the new block has already reached finalized storage.
Transaction flow
The mempool accepts candidates from inbound peers, its crawler, and RPC. The admission pipeline downloads and verifies them before applying storage policy; gossip advertises accepted IDs.
Concurrency
The daemon creates a multithreaded Tokio runtime. Tower readiness and bounded buffers control admission to shared services; futures let independent network and verification operations overlap. Startup uses one-shot channels to complete the cycles between inbound networking, mempool, and transaction verification after the handles exist.
Cryptographic services use tower-batch-control to aggregate requests
without requiring each caller to assemble a batch. The
tower-fallback wrapper falls back to individual verification when
a batch fails; for example, the Ed25519 service composes both.
CPU-heavy work runs outside ordinary asynchronous task polling. The
Ed25519 verifier submits work to Rayon, while the
Sapling verifier uses tokio::task::spawn_blocking for batch validation.
The application initializes Rayon; see
CPU-thread configuration. State reads use blocking
tasks; state writes run in a dedicated write-worker thread.
Channels publish updated state to readers; block commits remain ordered even
when verification completes out of order.
Diagrams
- Crate dependencies
- Service dependencies
- Zebra network architecture
- Mempool architecture
- State database address index
Zebra network architecture
The zebra-network crate translates the bidirectional Zcash wire
protocol into Tower services accepting
Request values and returning Response values. Each peer has a Connection
task and Client handle. The application supplies an inbound service for remote
requests; a NetworkService handle routes the application’s outbound requests.
Requests and connections
Solid arrows carry requests or wire messages. Dotted arrows carry metadata or peer-set updates. Response futures return to the requesting caller.
flowchart TD
service["NetworkService"] --> buffer["Buffer"]
buffer --> set["PeerSet"]
set --> load["LoadTrackedClient / PeakEwma"]
load --> client["Client"]
client --> connection["Connection"]
connection <--> framed["Framed / Codec"]
framed <--> tcp["TcpStream"]
connection --> inbound["Inbound through LoadShed / Buffer / Timeout"]
inventory["InventoryRegistry"] -.-> set
client -.->|missing inventory| inventory
framed -.->|incoming messages| tracking["register_inventory_status"]
tracking -.->|inventory advertisements| inventory
set -.->|connected-peer channels| advert["BlockAdvertiser"]
advert -->|BlockAdvert| connection
bans["BanList"] -.-> set
version["MinimumPeerVersion"] -.-> set
stalls["FindResponseStallTracker"] -.-> set
The handshake constructs the framed transport, client channels, and
connection task. Its incoming-message pipeline records inventory advertisements
and transport errors before delivering messages to the connection. A separate,
rate-limited heartbeat task updates address liveness after successful ping
responses. The client also records missing-inventory responses. The
load wrapper uses Tower’s PeakEwma estimator, which tracks recent
response latency, for load-sensitive selection.
The peer set routes requests by class:
| Request | Routing |
|---|---|
FindBlocks, FindHeaders | Fair rotation across ready connections, preferring serving peers for up to 1.5 seconds. Fast empty replies do not gain a latency-based routing advantage. |
BlockDownload, single-item BlocksByHash, single-item TransactionsById | Inventory-aware selection prefers a peer that advertised the item and excludes peers recorded as missing it. Block requests can wait for a suitable busy peer; transaction requests do not use that wait. |
AdvertiseTransactionIds | Broadcast to a randomly selected subset of ready peers, rather than selecting by latency. |
PushTransaction | Broadcast with additional handling for configured zcashd-compat sidecar peers. |
| Other ordinary requests | Power-of-two-choices selection: sample two ready peers and choose the less loaded one. |
AdvertiseBlock, AdvertiseBlockToAll | Rejected on the ordinary network service; callers must use BlockAdvertiser. |
The dedicated block advertiser has its own concurrency bound and per-connection channels. A busy ordinary request does not prevent a connection from receiving a block advertisement. Successful advertisement completion means the selected connections completed their local transport writes, not that remote peers received or accepted the block.
The inbound stack bounds remote requests with load shedding, a buffer, and a response timeout. The connection task interprets incoming messages as responses to outstanding requests or inbound requests.
Discovery and address state
The initializer adds peer clients from accepted inbound connections, initial seed/cache connections, and the crawler.
flowchart TD
initial["add_initial_peers"] --> connector["peer::Connector"]
crawler["crawl_and_dial"] --> connector
listener["accept_inbound_connections"] --> handshake["peer::Handshake"]
connector --> handshake
initial -->|DiscoveredPeer| discovery["peerset_tx / peerset_rx"]
crawler -->|DiscoveredPeer| discovery
listener -->|DiscoveredPeer| discovery
discovery --> set["PeerSet"]
set -->|MorePeers| crawler
crawler --> next["NextPeerService"]
crawler --> crawl["CrawlService"]
crawl -->|Peers request| set
next --> bookservice["AddressBookService"]
crawl --> bookservice
bookservice --> updater["AddressBookUpdater"]
handshake --> updater
initial --> updater
updater --> book["AddressBook"]
updater --> bans["BanList"]
cache["peer_cache_updater"] --> bookservice
The initial-peer task resolves configured seed addresses and reads the peer cache. The candidate services select outbound candidates from the address book and store addresses learned from peer crawls. Candidate selection and crawling have separate rate limits. The crawl-and-dial loop crawls on its timer or when the peer set requests more peers and no candidates are available. Dialing respects the configured connection limits.
The address-book updater owns address mutations and publishes ban updates. The peer-set maintenance path removes failed, banned, or protocol-ineligible connections and handles queued requests when suitable peers become available. The minimum protocol version follows the local chain tip; network-upgrade behavior is specified by ZIP 201.
Ownership and shutdown
A NetworkService handle owns the network’s worker and background
tasks. Its clones and unfinished response futures keep those tasks alive.
Releasing the last owner stops the tasks and releases the listener and peer
connections.
Services retained by the network, including the inbound handler’s downloaders
and mempool, use NetworkService::background_service(). That handle shares the
existing buffer without owning network lifetime, avoiding a reference cycle.
A BlockAdvertiser handle also requires an external owning network
handle to remain alive.
Mempool architecture diagram
The Mempool service owns storage and download/verification tasks.
The downloader returns candidates; only the service admits them to storage.
Arrows carry requests or events; gossip_mempool_transaction_id is a task,
not a Tower service.
flowchart TD
inbound["Inbound"] -->|QueueFromPeer| mempool["Mempool"]
rpc["RpcImpl"] -->|Queue or queries| mempool
crawler["Crawler"] -->|MempoolTransactionIds| network["NetworkService / PeerSet"]
crawler -->|Queue| mempool
checker["QueueChecker"] -->|CheckForVerifiedTransactions| mempool
mempool -->|admit candidate| downloads["Downloads"]
downloads -->|TransactionsById when needed| network
downloads -->|Transaction and Tip queries| state["StateService"]
downloads -->|MempoolRequest| verifier["MempoolTxVerifier"]
verifier -->|unspent outputs, anchors, nullifiers, time| state
verifier -->|AwaitOutput| mempool
downloads -->|completed results in poll_ready| mempool
mempool -->|insert or remove| storage["Storage"]
storage --> verified["VerifiedSet"]
storage --> pending["PendingOutputs"]
storage --> rejected["EvictionList"]
tip["ChainTipChange / LatestChainTip"] --> mempool
status["SyncStatus"] -->|initial activation| mempool
status -->|crawl gating| crawler
mempool -->|MempoolChange added| gossip["gossip_mempool_transaction_id"]
gossip -->|AdvertiseTransactionIds| network
Components
| Component | Responsibility |
|---|---|
Mempool | Dispatch requests, collect completed verification results in poll_ready, update storage for tip changes, and publish MempoolChange events. |
Downloads | Deduplicate queued candidates, enforce global and per-peer admission bounds, check whether a transaction is already mined, download missing bytes, and call the verifier. |
MempoolTxVerifier | Verify for the next block height using chain state and, for transparent dependencies, outputs awaited from the mempool. |
Storage | Apply standardness and cost-limit policy; own the verified set, pending-output requests, and rejection caches. |
VerifiedSet | Index transactions, created and spent outputs, nullifiers, and dependencies; perform weighted eviction and remove dependent transactions. |
PendingOutputs | Hold callers waiting for transparent outputs and respond when a parent transaction is inserted. |
EvictionList | Retain bounded, timed lists of transaction IDs rejected for chain effects, including random eviction. |
Crawler | Query peer mempools and queue returned IDs. |
QueueChecker | Trigger completion processing without external requests. |
gossip_mempool_transaction_id | Consume added-transaction events, combine available IDs, and advertise them through the network service. |
The mempool specification defines activation, task startup and timing, admission, eviction, rejection lifetimes, tip resets, and the service API.
Mempool specification
The Mempool service stores verified, unmined transactions in memory
for relay, RPC queries, and block templates. Admission does not confirm a
transaction, reserve a place in a block, or persist it across restarts. Shared
request and response types live in zebra-node-services.
The architecture diagram shows component boundaries; verification stages distinguish consensus checks from relay policy.
Activation
The service starts disabled. Activation requires a chain-tip update and either:
- a
SyncStatusindicating that Zebra is close to the network tip; or - a best-chain height at or above
[mempool] debug_enable_at_height.
After activation, a later sync-status change alone does not disable the service. A reset recreates an already-active mempool at the new tip regardless of sync status.
The crawler, queue checker, and gossip task are spawned by node startup, not by the activation transition. The crawler waits for close-to-tip status or the debug-height condition before each crawl. This is separate from the service’s one-time activation gate.
While disabled, the service returns empty ID, transaction-list, rejection-list,
and statistics responses. It rejects full-transaction, dependency, and output
queries. A Queue response contains a Disabled error per candidate.
The QueueFromPeer request silently drops candidates, avoiding disconnections.
Configuration
For limits, defaults, and the Regtest activation override, see mempool configuration.
Candidate admission and verification
The service accepts candidates through three paths:
- The inbound handler uses
QueueFromPeerfor both advertised IDs and complete transactions received from a known peer. - The crawler requests
MempoolTransactionIdsfrom the network service and submits the returned IDs withQueue. - RPC submits complete transactions through
Queue. The RPC layer also has a separate, bounded retry queue; it is not the mempool download queue.
The download pipeline performs these steps:
- Check storage for an existing transaction or cached rejection. Deduplicate outstanding requests by exact unmined ID. Variants with the same mined ID share a pending slot only when supplied by the same advertising IP address; one host’s authorization variant cannot reserve other hosts’ or local variants. Release reservations when tasks complete, fail, time out, or finish cancellation.
- Enforce a global limit of 500 concurrent download/verification tasks and a limit of 5 per advertising IP address. Multiple connections from the same IP share that allowance. Crawler and local submissions have no source IP and are subject only to the global limit.
- Query chain state for an already-mined transaction and the current tip height.
If the candidate contains only an ID, request its bytes with
TransactionsById. A complete transaction skips that network download. - Call
MempoolTxVerifierwith the candidate and the next block height. Bound the operation with download, verification, and whole-task timeouts. - Return a
VerifiedUnminedTx, its mempool-output dependencies, and the tip height used for verification to the mempool service.
The transaction verifier rejects coinbase transactions and applies
height-dependent consensus checks. It obtains unspent transparent
outputs from the best chain or waits for a mempool parent through AwaitOutput.
It checks maturity and lock time, applies input standardness and fee policy
before expensive cryptography, verifies scripts, signatures, and proofs, and
queries state for valid anchors and unrevealed nullifiers.
The verifier waits up to 60 seconds for a missing mempool output.
The PendingOutputs tracker wakes waiters when storage inserts the
parent’s output. An AwaitOutput response is informational: it does not guarantee
that the output remains unspent. Storage checks conflicts and dependency
availability again at insertion.
The mempool’s poll_ready method collects completed download tasks. If
the chain grew during verification, it requeues the returned candidate for the
new context rather than inserting the stale result. A matching-context result
passes through Storage::insert; only successful storage admission completes a
queued caller’s result with success. A successful initial Queue response alone
means the task was admitted, not that the transaction entered the mempool.
Storage and standardness
The Storage type owns the mined-ID-indexed VerifiedSet,
created and spent transparent outputs, nullifiers, transparent dependencies,
pending-output requests, and tip- or chain-scoped rejection caches.
Storage contains at most one transaction for each mined ID. Before insertion, it rejects cached failures, duplicates, non-standard transactions, conflicts with existing mempool spends or nullifiers, and transactions whose unconfirmed parent outputs disappeared during verification. Removing a transaction also removes transactions that depend on its outputs, directly or indirectly.
The standardness checks require standard transparent scripts,
push-only input scripts within the size limit, standard input spending patterns,
and no dust outputs. They reject bare multisig outputs, more than one
OP_RETURN output, an OP_RETURN script above the configured data-carrier limit,
and more than 4,000 transparent signature operations per transaction.
These are mempool policy, not extra block-consensus rules.
Fee policy and ZIP 317
Zebra 7.0.0 uses ZIP 317’s logical-action accounting with the
proposed fee reduction to a
MARGINAL_FEE of 1,000 zatoshis. Published ZIP 317 still specifies 5,000.
The mempool already accepts the reduced fee; the
getstandardfee recommendation follows its deployment schedule.
The conventional fee is 1,000 times the greater of two and the transaction’s
logical-action count. The zero BLOCK_UNPAID_ACTION_LIMIT rejects unpaid
actions, requiring at least this fee.
The fee checker also retains a serialized-size fee floor of
clamp(100 * size / 1000, 100, 1000) zatoshis, using integer division and size in
bytes. With the unpaid-action limit set to zero, this floor cannot reject a
transaction that passed the conventional-fee check.
Cost limit and eviction
The ZIP 401 weighted eviction policy uses serialized transaction size as its memory-size approximation. Zebra calculates:
cost = max(serialized_size_in_bytes, 10000)
low_fee_penalty = 40000 if fee < implementation_conventional_fee, otherwise 0
eviction_weight = cost + low_fee_penalty
The penalty affects selection probability, not total cost. Admitted transactions already pay the conventional fee, so their penalty is zero.
After inserting a candidate, storage repeatedly selects a transaction
with probability proportional to its eviction weight until total cost is at or
below [mempool] tx_cost_limit. The new candidate participates in that selection
and can be evicted immediately. The verified set removes the selected
transaction and its descendants. Storage records the selected transaction’s
mined ID in the random-eviction rejection list; dependent removals do not each
receive their own random-eviction entry.
The default cost limit permits at most 8,000 transactions, fewer when costs exceed 10,000. This is not an 80-megabyte process-memory limit: deserialized objects, indexes, rejection caches, and in-flight candidates also consume memory.
Rejection caches
The storage error types distinguish authorizing data from transaction effects. For witnessed transactions, an unmined ID includes the authorization digest as well as the mined ID; the mined ID identifies the effects. See transaction identifiers and Zebra’s identifier types.
| Cache | Key | Examples | Lifetime |
|---|---|---|---|
tip_rejected_exact | UnminedTxId | Failed consensus verification or standardness. | Cleared on best-tip growth or reset; cleared if it exceeds 40,000 entries. |
tip_rejected_same_effects | Mined transaction ID | A mempool spend conflict or a missing unconfirmed parent output. | Cleared on best-tip growth or reset; cleared if it exceeds 40,000 entries. |
chain_rejected_same_effects | Mined transaction ID, grouped by reason | Mined, expired, duplicate spend in the chain, or randomly evicted. | Each reason has a timed EvictionList, bounded to 40,000 entries; all are discarded on reset. |
The EvictionList drops its oldest entry at capacity, treats entries
older than [mempool] eviction_memory_time as absent, and prunes them on
insertion and mutable length queries.
Network download failures, state-query errors, cancellations, and whole-task timeouts are not cached. Failed tasks leave the queue without automatic retry; a later crawl or advertisement can resubmit them. Errors returned by the verifier, including its timeout, are cached for the exact tip.
Chain-tip changes
The mempool service processes tip changes before accepting another request:
- Growth by one block: cancel pending work for mined transactions, remove mined transactions and conflicts with the block’s spends or nullifiers, update dependency tracking, and clear tip-scoped rejection caches. Transactions whose parents were mined can remain because those outputs now belong to chain state.
- Reset: a changed best chain, skipped tip updates, or an activated network upgrade clears storage and rejection caches and cancels old downloads. The service then attempts to requeue its previously stored and pending candidates against the new tip. Requeueing remains subject to download-queue capacity.
- Expiry: remove transactions whose expiry height is at or below the current tip, together with their dependent transactions. They cannot be included in the next block. See ZIP 203 and the expiry-removal implementation.
Ordinary growth does not reverify every stored transaction; it performs the targeted removals above.
Crawling, queue checks, and gossip
The crawler issues three MempoolTransactionIds requests per crawl,
not necessarily to distinct peers. Each response has a 6-second timeout,
the crawl has a 73-second timeout, and the crawler sleeps 73 seconds after
completion before its next enabled crawl.
The queue checker sleeps 5 seconds between
CheckForVerifiedTransactions calls. The request causes the buffered service to
run poll_ready, where completion processing occurs. The transaction verifier
also schedules a best-effort check after a 50-millisecond delay to reduce
latency for dependencies and RPC submission. Neither interval is a guaranteed
transaction-admission deadline.
The gossip task consumes MempoolChange events and advertises only
added transactions with AdvertiseTransactionIds. It combines available events
before sending and rate-limits successive broadcasts. Mined and invalidated
events are available to subscribers but are not transaction advertisements.
If a subscriber falls behind the bounded broadcast channel, it can miss events;
the event stream is not a durable transaction log.
Service interface
The Request and Response enums define the internal Tower API.
| Request | Result or action |
|---|---|
TransactionIds | Return the stored unmined IDs. |
TransactionsById | Return transactions whose full unmined IDs match, including the authorization digest. |
TransactionsByMinedId | Look up transactions using mined IDs. |
TransactionWithDepsByMinedId | Return a verified transaction and its recorded dependency IDs. |
FullTransactions | Return verified transactions, dependency information, and the last processed tip hash. |
RejectedTransactionIds | Return supplied IDs matching a cached rejection. |
AwaitOutput | Wait for a transparent output created by a mempool transaction; callers must impose a timeout. |
UnspentOutput | Report whether an output is created or spent in the mempool, or unknown there. |
Queue | Admit candidates and return per-candidate initial errors or one-shot receivers for the eventual result. |
QueueFromPeer | Admit peer-tagged candidates under the per-IP cap; return an empty Queued vector rather than per-candidate receivers. |
CheckForVerifiedTransactions | Trigger readiness processing and acknowledge the check. |
QueueStats | Return transaction count, serialized bytes, the current usage estimate, and notification status. |
The storage lookup resolves TransactionsById through the mined-ID
index, then checks the full unmined ID. A different authorization digest does
not match. The statistics handler sets usage equal to serialized
bytes and leaves fully_notified unset. Neither field is a measurement of all
mempool allocations or proof that every notification was delivered.
Metrics
The metrics endpoint converts these recorder names to Prometheus form:
| Metric | Meaning | Source |
|---|---|---|
zcash.mempool.size.transactions | Stored transaction count. | Verified set |
zcash.mempool.size.bytes | Sum of serialized sizes. | Verified set |
zcash.mempool.cost.bytes | Sum of ZIP 401 transaction costs. | Verified set |
mempool.currently.queued.transactions | Current download/verification queue length. | Downloads |
mempool.rejected.transaction.ids | Entries in rejection caches; an ID in multiple lists is counted multiple times. | Storage |
mempool.full_queue.per_peer.total | Admission attempts rejected by the per-IP queue cap. | Downloads |
mempool.failed.verify.tasks.total | Failed task count, labeled by reason. | Mempool |
mempool.gossiped.transactions.total | Transaction IDs submitted to network broadcasts. | Gossip |
State database format and upgrades
Zebra stores finalized chain data in RocksDB. A column family is a named key-value map within that database. The database format includes the column family names, the byte encodings of their keys and values, and the relationships between records.
For deployment, direct-reader compatibility, and rollback, follow Upgrading Zebra.
Format versions and paths
The database format version is independent of Zebra and Rust crate versions. Zebra 7.0.0 implements format 29.0.0:
| Constant | Current value | Change requiring a bump |
|---|---|---|
DATABASE_FORMAT_VERSION | 29 | An incompatible encoding change or column family removal |
DATABASE_FORMAT_MINOR_VERSION | 0 | A significant compatible change, including a new column family |
DATABASE_FORMAT_PATCH_VERSION | 0 | A significant format compatibility fix |
A change that would otherwise be incompatible can use a minor bump only when
all supported Zebra versions already contain the required compatibility code.
The function state_database_format_version_in_code() returns the complete
semantic version. Builds with the indexer feature return
29.0.0+indexer; other builds return 29.0.0.
For persistent databases, the method Config::db_path() constructs this path:
<state.cache_dir>/state/v<major>/<network>/
The network directory is mainnet, testnet, or regtest for the standard
networks. A configured non-Regtest Testnet uses its lowercase name followed
by - and its four-byte wire magic encoded as eight lowercase hexadecimal
digits. The method Network::directory_name() defines
this namespace. Databases for the same network with different minor or patch
versions still use the same major-version directory.
The file version inside the database directory records the full semantic
version, for example 29.0.0 or 29.0.0+indexer, without a trailing newline.
The version reader also accepts the legacy minor.patch form and
supplies the major version from the requested database path. If the file is
absent but the directory exists, it reports <major>.0.0. If both are absent,
it reports no database. Malformed or unreadable version files are errors,
not requests to resynchronize.
The version writer replaces the file atomically. During a reusable major-version upgrade, the directory can temporarily contain a version file with the preceding major version while the format upgrade is pending.
The state configuration sets the cache root or selects a temporary directory named with the database kind, major version, and network.
Upgrading the state database
A major version selects a different directory. It does not necessarily require resynchronization: a registered reusable major upgrade permits the writer to move a compatible database from the immediately preceding major version into the new directory, then apply the required format changes.
The registry restorable_db_versions() contains destination majors
26, 27, 28, and 29. The reuse implementation considers only
current_major - 1: format 29 reuses format 28, not format 27 or earlier.
Reuse requires all of the following:
- The current major version has a registered reusable upgrade.
- RocksDB recognizes the previous network directory as a database.
- RocksDB does not recognize an existing database at the new path.
- The canonical previous database path lies inside the canonical
[state] cache_dir. - The directory move succeeds.
The writer renames the previous network directory into the new major-version
path and records its old semantic version in the relocated version file.
It removes the old major-version directory if that directory is now empty.
It does not copy the database. A symlink to a database outside the cache root
is not reused. If reuse does not occur, Zebra opens the current path or creates
a new database there; an empty database requires synchronization from genesis.
The v28-to-v29 upgrade
Format 29 adds the Network Sustainability Mechanism (NSM) balance to stored
value pools. The registered upgrade uses NoMigration: existing records do
not need rewriting because the new readers accept their older encodings.
| Record | Format 28 writes | Format 29 writes |
|---|---|---|
ValueBalance<NonNegative> | 48 bytes | 56 bytes |
BlockInfo | 52 bytes | 60 bytes |
Legacy records read with a zero NSM balance; new writes use the wider encoding even when that balance is zero. See the byte layouts.
Old-database cleanup
When enabled, the cleanup task scans major-version directories under
the database-kind root, such as <state.cache_dir>/state, and removes older
versions except predecessors of registered reusable upgrades. It skips current
and newer versions, paths outside the canonical cache root, and ephemeral
databases.
The current registry protects v25 through v28. Cleanup removes an eligible
major-version directory as a whole, including its network subdirectories.
Upgrade execution and validity checks
The wrapper ZebraDb::new() compares the on-disk version with the
running format version and chooses one of four operations:
| Database state | Operation |
|---|---|
| No database | Create the database and mark it with the running version |
| Older version | Apply registered upgrades in version order |
| Same version | Check the existing format |
| Newer version | Mark it with the older running version, then check it |
The last case applies to a database the binary actually opens. It is not a mechanism for opening another major-version directory. Within a compatible major version, recording the lower version makes a later newer writer repeat the required upgrades over data that the older writer could have changed. Compatibility remains the responsibility of the new format’s implementation.
The upgrade framework runs on a separate thread. It captures the finalized tip at startup so migrations can process existing records while normal block commits write the current format above that tip. For each registered upgrade newer than the initial disk version, it:
- Calls
prepare(),run(), andvalidate()ifneeds_migration()is true. - Atomically records the completed upgrade’s version.
An empty finalized database skips record migrations and is marked with the
running version. The trait DiskFormatUpgrade defaults needs_migration() to
true. Its default is_reusable_major_upgrade() returns true for a version
whose minor and patch components are both zero. The existing NoMigration
implementation sets needs_migration() to false for reader-compatible
changes that require no record rewrite.
Detailed format checks run after startup format handling. They combine quick
checks with every registered upgrade’s validate() method. Quick checks cover
the on-disk height limit and selected tree/subtree invariants; individual
validators check their own formats. These checks do not reverify every block
or prove every database record correct. An invalid result causes a panic with
a delete-and-resynchronize diagnostic.
Consensus separately checks cached chain hashes against the checkpoint list at startup. See Checkpoint and full verification.
Writer shutdown cancels the upgrade thread and checks the format if the recorded version has reached the running version. An interrupted upgrade retains the last completed version and resumes the remaining work on restart. Periodic detailed checks are optional. Read-only secondary instances skip this upgrade/check thread.
Design constraints
Format upgrades run concurrently with block commits and state queries. A migration must preserve readable, correct data at every committed batch boundary, including cancellation and restart boundaries.
- Make readers accept the old, partial, and new states that the migration can expose. Return an explicit unavailable result when appropriate; do not silently substitute incorrect wallet or validation data.
- Reuse normal block-write calculations and encodings in migration code. An upgraded database and a database synchronized from genesis must represent the same chain identically.
- Make repeat execution safe. Cancellation can occur after a batch is committed but before the version file advances. Compatible downgrades can also require an upgrade to run again.
- Write related changes in one
DiskWriteBatch. Separate successful writes do not make a multi-column change atomic. - Check the cancellation receiver during long operations and before committing
batches. Return
CancelFormatChangewhen cancellation is requested. - Bound work and memory use. Use bounded key ranges rather than retaining a long-lived iterator over unrelated records.
The current block writer demonstrates a concurrency-sensitive case: it uses
merge operations for transparent address balances while format upgrades are
pending, then switches to direct inserts after the format-upgrade flag is set.
The migration and normal writes must not overwrite each other’s balance
changes. See ZebraDb::write_block() and the
address-received upgrade.
Non-finalized state backup
When enabled for a persistent database, non-finalized backups use:
<state.cache_dir>/non_finalized_state/<network>/<block-hash>
The network namespace matches the finalized database, but the backup path has
no database-format version component. The backup implementation
stores an eight-byte little-endian signed deferred-pool balance change followed
by the block’s ZcashSerialize encoding. These are not RocksDB records and are
not relocated by a major-version database move.
The backup task writes blocks from all retained non-finalized chains, removes files for blocks no longer in that state, and waits at least five seconds between updates, so the backup can lag the in-memory tip.
At startup, the state service restores backup blocks only when the finalized tip exists and is at or above the maximum checkpoint height supplied to the service. Otherwise it downloads and verifies blocks above the finalized tip again; subsequent backup updates remove files not retained in the new non-finalized state. Restoration deserializes blocks, skips already-finalized blocks, and reapplies contextual validation in height order. Invalid or unreadable backup blocks are logged and skipped.
Read-only and secondary instances
An in-process ReadStateService reads the writer’s shared database. The
separate API zebra_state::init_read_only() instead opens an existing database
as a RocksDB secondary: a reader that catches up with another instance’s
on-disk state. The function spawn_init_read_only() performs that initialization
on a blocking task.
The secondary open path uses a separate temporary working directory.
It does not create the primary cache, reuse an older major-version directory,
run format upgrades, rewrite the primary version file, or flush or delete the
primary database at shutdown. It requires an existing readable cache and
database. Combining read-only mode with [state] ephemeral = true returns
StateInitError::ReadOnlyEphemeralConflict.
The returned ZebraDb exposes try_catch_up_with_primary() and
spawn_try_catch_up_with_primary() for explicit refreshes. Initialization
creates an empty non-finalized-state watch channel and returns its sender to
the caller; it does not restore or automatically receive the writer’s
in-memory chains. Applications using this API must supply their own
non-finalized updates if they need them.
Current state database format
RocksDB orders keys lexicographically by bytes. Zebra encodes numeric index components in big-endian order so byte ordering matches chain ordering. Composite keys concatenate fixed-width components, allowing one-to-many indexes without storing a growing list in a single value.
The array STATE_COLUMN_FAMILIES_IN_CODE registers all 29 families
below, regardless of the indexer feature. The RocksDB default family is
unused and must remain empty. Opening a database also preserves existing
families not listed by the running code.
For byte encodings, see Data formats. For nullifier values,
SpendLocation means () without indexer, or TransactionLocation with it;
this is a table abbreviation, not a Rust type.
| Column family | Key | Value |
|---|---|---|
hash_by_height | block::Height | block::Hash |
height_by_hash | block::Hash | block::Height |
block_header_by_height | block::Height | block::Header |
tx_by_loc | TransactionLocation | transaction::Transaction |
hash_by_tx_loc | TransactionLocation | transaction::Hash |
tx_loc_by_hash | transaction::Hash | TransactionLocation |
balance_by_transparent_addr | transparent::Address | AddressBalanceLocation |
tx_loc_by_transparent_addr_loc | AddressTransaction | () |
utxo_by_out_loc | OutputLocation | transparent::Output |
utxo_loc_by_transparent_addr_loc | AddressUnspentOutput | () |
tx_loc_by_spent_out_loc | OutputLocation | TransactionLocation |
sprout_nullifiers | sprout::Nullifier | SpendLocation |
sprout_anchors | sprout::tree::Root | sprout::tree::NoteCommitmentTree |
sprout_note_commitment_tree | () | sprout::tree::NoteCommitmentTree |
sapling_nullifiers | sapling::Nullifier | SpendLocation |
sapling_anchors | sapling::tree::Root | () |
sapling_note_commitment_tree | block::Height | sapling::tree::NoteCommitmentTree |
sapling_note_commitment_subtree | NoteCommitmentSubtreeIndex | NoteCommitmentSubtreeData<sapling_crypto::Node> |
orchard_nullifiers | orchard::Nullifier | SpendLocation |
orchard_anchors | orchard::tree::Root | () |
orchard_note_commitment_tree | block::Height | orchard::tree::NoteCommitmentTree |
orchard_note_commitment_subtree | NoteCommitmentSubtreeIndex | NoteCommitmentSubtreeData<orchard::tree::Node> |
ironwood_nullifiers | ironwood::Nullifier | SpendLocation |
ironwood_anchors | orchard::tree::Root | () |
ironwood_note_commitment_tree | block::Height | orchard::tree::NoteCommitmentTree |
ironwood_note_commitment_subtree | NoteCommitmentSubtreeIndex | NoteCommitmentSubtreeData<orchard::tree::Node> |
history_tree | () | HistoryTreeParts |
tip_chain_value_pool | () | ValueBalance<NonNegative> |
block_info | block::Height | BlockInfo |
Indexer-dependent values
The feature indexer populates tx_loc_by_spent_out_loc and records spending
transaction locations in all four nullifier families. Without it, new
nullifier values are empty and new transparent spending-location entries are
not written.
At startup, an indexer-enabled writer checks and backfills these indexes over
existing blocks. A non-indexer writer opening a database marked +indexer
removes the transparent spending-location entries, rewrites nullifier values
as empty, and removes the build metadata after completion. During conversion,
nullifier values can contain either encoding. The reader for
Option<TransactionLocation> returns a location only for a five-byte value.
This feature conversion runs separately from numbered migrations.
Data formats
The traits IntoDisk and FromDisk define serialization and deserialization
in disk_format. In the following tables, ‖ denotes byte
concatenation; lengths are uncompressed byte counts, not RocksDB disk usage.
| Type | Encoding | Bytes |
|---|---|---|
block::Height | Unsigned 24-bit big-endian integer | 3 |
TransactionIndex | Unsigned 16-bit big-endian integer | 2 |
TransactionLocation | Height ‖ TransactionIndex | 5 |
OutputIndex | Unsigned 24-bit big-endian integer | 3 |
OutputLocation | TransactionLocation ‖ OutputIndex | 8 |
AddressLocation | The first output location associated with the address | 8 |
AddressTransaction | AddressLocation ‖ TransactionLocation | 13 |
AddressUnspentOutput | AddressLocation ‖ OutputLocation | 16 |
transparent::Address | One-byte variant tag followed by its 20-byte hash | 21 |
Amount | Signed 64-bit little-endian integer | 8 |
AddressBalanceLocation | Amount ‖ AddressLocation ‖ received, with received an unsigned 64-bit little-endian integer | 24 |
block::Hash, transaction::Hash | Their raw internal bytes, not the displayed hexadecimal string | 32 |
| Shielded nullifiers and anchor roots | Their pool-specific byte representation | 32 |
NoteCommitmentSubtreeIndex | Unsigned 16-bit big-endian integer | 2 |
NoteCommitmentSubtreeData<Node> | end_height ‖ root, using a three-byte height and 32-byte node | 35 |
() | Empty byte string | 0 |
The transparent address encoder assigns tags 0 and 1
to Mainnet public-key-hash and script-hash addresses, 2 and 3 to their
Testnet/Regtest equivalents, and 4 and 5 to Mainnet and Testnet/Regtest TEX
addresses. A legacy 16-byte AddressBalanceLocation lacks received; its
reader supplies zero until the relevant upgrade computes the total received.
Block headers, transactions, and transparent outputs use ZcashSerialize
encodings, with ZcashDeserialize on reads. Database location keys are
Zebra-specific, not transaction outpoints or protocol height encodings.
Value pools and block information
The value-pool encoding writes seven eight-byte amounts: transparent, Sprout, Sapling, Orchard, deferred, Ironwood, NSM. Readers also accept exactly 32, 40, or 48 bytes and supply zero for missing trailing pools. Other lengths are rejected.
The block-information encoding preserves the v28 prefix:
Byte offsets Field
0..48 transparent, Sprout, Sapling, Orchard, deferred, Ironwood
48..52 block size, unsigned 32-bit little-endian integer
52..60 NSM balance, signed 64-bit little-endian amount
The reader also accepts the 44-byte pre-Ironwood layout, with its size at
40..44, and the 52-byte pre-NSM layout. It ignores trailing bytes after the
layout selected by the record length. This behavior does not make v28 readers
compatible with v29 state as a whole; the value-pool decoder also has to
support the new width.
Tree records
Note commitment trees use bincode::DefaultOptions with their serde
implementations. Ironwood uses the Orchard tree, root, and subtree-node types
but stores them separately.
The history-tree value is HistoryTreeParts: network kind, tree size, a map of
peak positions to history entries, and current height. Its bincode encoding
reconstructs NonEmptyHistoryTree using the configured network. The reader
also accepts the legacy 253-byte history-entry width and zero-pads those entries
to the current width.
These derived encodings depend on struct layout and field order. Do not use
serde-derived bincode formats for new database records. Define an explicit
encoding and retain compatibility readers when required.
Record relationships and updates
The block-height and block-hash families index each other. Transaction locations likewise index transaction hashes in both directions. Zebra stores headers and transactions separately: a block lookup resolves its height, reads the header, and collects that height’s transactions in index order.
The type AddressLocation aliases OutputLocation and identifies an address’s
first output even after it is spent. The address-balance record holds the
current balance, that stable location, and the cumulative received amount.
Address-transaction keys index transactions that send or receive funds;
spending an output does not remove that history.
For example, an address with first output at height 100, transaction 2, output 3 has this eight-byte location:
00 00 64 | 00 02 | 00 00 03
height tx output
A transaction at height 101, index 4 that affects the same address has this
13-byte AddressTransaction key:
00 00 64 00 02 00 00 03 | 00 00 65 00 04
address location transaction location
Prefix-range reads over that address location retrieve its transaction or
unspent-output locations in chain order. The values in those index families
are empty. Spending an output removes both its utxo_by_out_loc entry and its
utxo_loc_by_transparent_addr_loc entry. The stored output and its location
supply the fields needed to reconstruct a transparent::Utxo.
Address index diagram
Solid arrows name column-family lookups; thick arrows show composite fields; dotted arrows show address-location prefix lookups.
flowchart TD
address["transparent::Address (21 bytes)"] -->|balance_by_transparent_addr| balance["AddressBalanceLocation (24 bytes)"]
balance ==>|balance| amount["Amount (8 bytes)"]
balance ==>|address_location| location["AddressLocation (8 bytes)"]
balance ==>|received| received["u64 (8 bytes)"]
location -.->|"tx_loc_by_transparent_addr_loc: AddressTransaction (13 bytes)"| transaction["TransactionLocation (5 bytes)"]
location -.->|"utxo_loc_by_transparent_addr_loc: AddressUnspentOutput (16 bytes)"| output_location["OutputLocation (8 bytes)"]
output_location -->|utxo_by_out_loc| output["transparent::Output"]
output_location ==> transaction
output_location ==> output_index["OutputIndex (3 bytes)"]
transaction ==> height["block::Height (3 bytes)"]
transaction ==> transaction_index["TransactionIndex (2 bytes)"]
Shielded and chain records
The Sprout tree family stores the current tree under the empty key; Sprout anchors map roots to trees. The tree writer stores Sapling, Orchard, and Ironwood trees at genesis and when they change. Reads find the latest tree at or below the requested height. Subtree families use subtree indexes as keys and store each completion height and root.
The history tree and tip value pools each use a single empty key. Per-block
value pools and block sizes are stored separately in block_info.
A normal finalized block commit writes its related records in one RocksDB write batch. Separate reads across families are not a shared snapshot merely because each block write is atomic. The address-balance query detects intervening finalized commits and retries a bounded number of times. It then removes overlapping finalized blocks from its cached non-finalized chain before combining balances, avoiding double-counting. Preserve these consistency rules when extending state queries.
Adding a column family
Use a named constant, a TypedColumnFamily<Key, Value> alias, and a database
accessor instead of repeating a string and choosing types at every read or
write. Add the name to STATE_COLUMN_FAMILIES_IN_CODE; declaring an accessor
does not create the family.
The block_info accessor in zebra_db follows this pattern:
#![allow(unused)]
fn main() {
pub const BLOCK_INFO: &str = "block_info";
pub type BlockInfoCf<'cf> = TypedColumnFamily<'cf, Height, BlockInfo>;
impl ZebraDb {
pub(crate) fn block_info_cf(&self) -> BlockInfoCf<'_> {
BlockInfoCf::new(&self.db, BLOCK_INFO)
.expect("column family was created when database was created")
}
}
}
The typed handle enforces the key and value types at its call sites. Both types
must implement IntoDisk and FromDisk; the key also implements Debug.
The method zs_get(&height) returns Option<BlockInfo> for this family.
Use with_batch_for_writing() to add a typed write to an existing block or
migration batch. The caller writes that batch only after all related changes
are prepared. The method new_batch_for_writing() creates a separate owned
batch; finish it with write_batch(). The method take_batch_for_writing()
wraps an existing owned batch. See column_family.rs.
When a migration needs a legacy encoding, define a separate legacy alias and
accessor for the same family. The existing HistoryTreePartsCf and
LegacyHistoryTreePartsCf distinguish empty keys from legacy height keys.
Do not spread raw-byte casts or independently chosen read/write types through
normal access paths.
Implementing a format change
- Classify compatibility using the version rules above. Check every supported
reader, including direct database consumers and both
indexerbuilds. Increment the appropriate constants and record the format change inconstants.rs. Reset lower components for a major or minor bump. - Implement the new encoding and normal finalized-block write path. Keep required legacy readers. Register new column families and update every affected query.
- Add a
DiskFormatUpgradeimplementation underdisk_format/upgrade/and append it toformat_upgrades()in increasing version order. UseNoMigrationonly when existing records already satisfy the new reader’s requirements. Do not register a reusable major upgrade unless the preceding major’s database is safe to open and migrate. - Implement cancellation-safe migration and validity checks. Account for the startup tip, concurrent new blocks, partial migration, and repeat execution. Let the framework advance the version only after completion.
- Add serialization, compatibility, migration, and normal-write tests. Update the raw column-family snapshots when names or bytes intentionally change.
- Update this format reference and operator upgrade instructions. Add Changie fragments for affected users and library consumers, including rollback and coordinated-reader requirements. Follow the changelog guidelines; do not hand-edit generated changelogs.
The format-28 add_ironwood_tree migration adds the empty
Ironwood tree and anchor at genesis to existing databases through the normal
create_ironwood_tree() batch method. It skips an already-populated genesis
tree and validates the genesis tree and tip anchor.
Required tests
Exercise both migration and normal writes; rerunning an upgrade over an already-correct database can hide skipped records.
- Check current byte encodings and all supported legacy layouts. Include nonzero values so offset or field-order mistakes are observable.
- Upgrade an actual old-format database, not only a new database with a lowered version marker. Compare its records and query results with normal writes.
- Interrupt and repeat migration, then restart, commit more blocks, and restart again. Check the recorded version and format validity at each completed step.
- Test empty state, genesis, relevant activation boundaries, and partially populated new indexes.
- For reusable major changes, test directory movement, cleanup protection, preservation of legacy records, and subsequent current-format writes.
- Test both
indexerconfigurations when spend indexes or feature metadata are affected. Test read-only startup and refresh when reader behavior changes.
The existing
v28_cache_is_reused_with_legacy_value_pools test covers cleanup
protection, directory reuse even with cleanup disabled, unchanged legacy bytes,
and wider writes for subsequent blocks. The
block_info_reads_every_record_layout test checks old layouts
and the NSM offset. The raw column-family snapshot tests cover
family names and bytes for empty databases and the first three blocks; they do
not replace tests for shielded data or later upgrades.
Manually triggering a format upgrade
Use only a disposable copy of a test database. Stop all processes accessing it
before changing its version file. Replace the file with the full semantic
version whose registered upgrades the test needs to exercise, preserving the
actual data layout and supported major-version path.
Deleting the file does not select an arbitrary older format: it selects
<directory-major>.0.0. Deleting state/v29/<network>/version therefore reports
29.0.0 and does not trigger the 29.0.0 upgrade. Changing the marker does not
convert record encodings, create an old-format fixture, or prove that a
major-version move works. Use the persistent-cache upgrade test pattern for
those cases; never use version-file edits as an operator recovery procedure.
Randomized property testing
Zebra uses proptest to check invariants over generated inputs. A strategy
defines the inputs to generate; an Arbitrary implementation supplies a
strategy for a type. When a case fails, proptest attempts to shrink it to a
smaller failing input and saves a regression seed for subsequent runs.
Use fixed vectors for protocol boundaries and regressions, and strategies for input combinations. Generated transactions and blocks are not necessarily consensus-valid; select guarantees appropriate to the layer under test.
Sharing strategies across crates
Zebra exposes test generators through the proptest-impl Cargo feature. The
feature exists in zebra-chain, zebra-consensus, zebra-network,
zebra-state, zebra-rpc, and zebrad. Dependent test features forward to the
crates whose generators they need. The zebra-script tests enable
zebra-chain/proptest-impl without defining their own feature.
Gate a generator with any(test, feature = "proptest-impl"). Derive
Arbitrary when independent arbitrary fields satisfy the test’s requirements:
#![allow(unused)]
fn main() {
#[cfg_attr(
any(test, feature = "proptest-impl"),
derive(proptest_derive::Arbitrary)
)]
struct Example(u32);
}
Use a manual strategy for dependent fields or constrained ranges. Reuse the existing block and transaction strategies before adding a generator.
The test configuration flag applies to the crate being tested, not to its
ordinary dependencies. Enable a dependency’s generators explicitly in the
consumer’s development dependencies, following the existing manifests:
[dev-dependencies]
zebra-chain = { path = "../zebra-chain", features = ["proptest-impl"] }
For a new generator dependency, follow the chain manifest:
- Add it as an optional dependency in
[dependencies]. - Enable it from the crate’s
proptest-implfeature. - Add it as a required
[dev-dependencies]entry for that crate’s own tests. - Gate its imports, modules, and implementations with the same configuration condition as the generator.
Do not enable test features in production dependency declarations.
Running and reproducing tests
Run the block serialization round-trip test:
PROPTEST_CASES=64 cargo test --locked -p zebra-chain block_roundtrip
| Variable | Effect |
|---|---|
PROPTEST_CASES | Set the requested number of successful generated cases for tests that use the environment-derived configuration. |
PROPTEST_MAX_SHRINK_ITERS | Limit attempts to shrink a failure. Setting it to 0 disables shrinking. |
PROPTEST_RNG_SEED | Set a decimal u64 random seed instead of selecting one randomly. |
PROPTEST_VERBOSE | Increase proptest diagnostics; pass -- --nocapture to Cargo to display output from successful tests. |
For a repeatable generated sequence, keep the dependency versions and strategy unchanged and set the seed:
PROPTEST_CASES=64 PROPTEST_RNG_SEED=12345 \
cargo test --locked -p zebra-chain block_roundtrip -- --nocapture
Proptest defaults to 256 cases; Zebra’s block round-trip test defaults to 16
and reads PROPTEST_CASES. Several state tests default to one case. A fixed
Config::with_cases overrides PROPTEST_CASES; check the selected test’s
configuration.
Do not use the reduced coverage settings as the only evidence for a generator or consensus change.
Preserve relevant files under each crate’s proptest-regressions/ directory
when fixing a generated failure. Add a fixed regression test when the failure
represents a protocol boundary or when later strategy changes would make the
saved seed insufficient. Keep the failing input and the test command in the
pull request’s test evidence.
Profiling and benchmarking Zebra
Record the commit, Rust toolchain, Cargo features, configuration, network, height range, and machine. Keep them fixed except for the change measured.
Profiling
The samply sampling profiler records native call stacks. Zebra defines a
profiling Cargo profile in .cargo/config.toml; it inherits
release optimizations and enables debug information for workspace crates.
Release-profile package overrides disable debug information for dependencies.
Install the profiler and build from the repository root:
cargo install --locked samply
cargo build --locked -p zebrad --profile profiling --bin zebrad
samply record ./target/profiling/zebrad --config zebrad.toml start
Stop the workload with Ctrl+C after collecting the intended interval.
Samply opens the profile in the Firefox Profiler interface and serves local
symbol information. Profiles remain local unless uploaded through the
profiler. On Linux, samply needs permission to use performance events; follow
its platform setup instructions rather than running the node as
root.
Use the Tokio console for async polling and wake activity.
Tracing flamegraphs
Zebra’s flamegraph feature records time in tracing spans, which are named
instrumented operations. It does not sample every native function. Build
without compile-time log-level restrictions when collecting detailed spans:
cargo build --locked -p zebrad --release --no-default-features \
--features flamegraph --bin zebrad
Add the output path and the required target filters to the node’s profiling configuration:
[tracing]
flamegraph = "zebra-profile"
filter = "info,zebrad=trace,zebra_consensus=trace,zebra_state=trace"
Run that binary with the configuration file:
./target/release/zebrad --config zebrad-profile.toml start
The flamegraph layer records folded tracing stacks in
zebra-profile.folded. During normal component shutdown, Zebra flushes the
folded data and generates zebra-profile.svg. Any extension supplied in the
configuration is replaced with .folded and .svg. A forced termination can
prevent the SVG from being generated. The output directory must already exist
and be writable.
Trace-level logging and span collection add work to the process. Use the same instrumentation when comparing profiles, and use uninstrumented benchmark runs for the performance claim.
Benchmarking
Zebra has Criterion benchmarks in zebra-chain/benches/ and
zebra-consensus/benches/. Run an individual target from the workspace root:
| Crate | Target | Operation | Command |
|---|---|---|---|
zebra-chain | block | Block serialization and deserialization | cargo bench --locked -p zebra-chain --bench block --features bench |
zebra-chain | transaction | Transaction serialization and deserialization | cargo bench --locked -p zebra-chain --bench transaction |
zebra-chain | redpallas | Batched and individual RedPallas signature verification | cargo bench --locked -p zebra-chain --bench redpallas |
zebra-consensus | groth16 | Sprout Groth16 proof verification | cargo bench --locked -p zebra-consensus --bench groth16 |
zebra-consensus | halo2 | Orchard Halo2 proof verification | cargo bench --locked -p zebra-consensus --bench halo2 |
zebra-consensus | sapling | Sapling proof and signature verification | cargo bench --locked -p zebra-consensus --bench sapling |
The transaction target uses version 1 through 5 vectors; the Halo2 target uses pre-NU6.2 Orchard vectors and its verifying key. Add representative inputs for paths these benchmarks omit.
To compare two revisions locally, save a Criterion baseline before changing the source:
cargo bench --locked -p zebra-chain --bench transaction -- --save-baseline before
After switching to the proposed revision, rerun with the same build settings and target directory:
cargo bench --locked -p zebra-chain --bench transaction -- --save-baseline after
cargo install --locked critcmp
critcmp before after
Keep other CPU- and I/O-intensive jobs off the benchmark host during the measurement. Repeat comparisons that are close to run-to-run variation; report the workload and measured uncertainty rather than only a percentage.
CI comparisons and dashboard
Dispatch the benchmark workflow to update the benchmark dashboard, or
apply the run-benchmarks pull request label for a base/head comparison.
See benchmark CI for triggers, target
registration, and result handling. Confirm small differences on a controlled
host.
Local synchronization workload
Measure a second node downloading from a synchronized source node to reduce public-peer variability. Initial peer lists are seeds, not allowlists; enforce isolation outside Zebra to restrict connections to the source. Disabling the peer cache and delaying crawls do not prevent peer requests when the peer set is busy.
Give the source its own persistent state directory. Replace the example path with the synchronized Mainnet cache:
[network]
network = "Mainnet"
listen_addr = "127.0.0.1:8233"
[state]
cache_dir = "/path/to/synced-mainnet-cache"
Build the binary and start the source outside the timed interval:
cargo build --locked --release -p zebrad --bin zebrad
./target/release/zebrad --config zebrad-source.toml start
Use a separate configuration for the measured node:
[network]
network = "Mainnet"
listen_addr = "127.0.0.1:8234"
initial_mainnet_peers = ["127.0.0.1:8233"]
cache_dir = false
peerset_initial_target_size = 1
crawl_new_peer_interval = "10d"
[state]
ephemeral = true
debug_stop_at_height = 10_000
The measured node uses a temporary database, flushing and removing it when finalized state reaches height 10,000. A crash can leave temporary files. The height-triggered exit bypasses tracing shutdown and SVG generation. Time the process directly:
time ./target/release/zebrad --config zebrad-measured.toml start
Use distinct ports and state directories. Keep the source running with the required blocks and wait for the measured process to exit between runs. Operating-system and source-node caches survive database recreation; record whether measurements use warm or cold caches.
The 10,000-block workload measures checkpoint synchronization. To measure full
verification, set [consensus] checkpoint_sync = false and stop above the
mandatory checkpoint coverage.
Tokio console
The Tokio console displays asynchronous tasks, resources, and operations.
Zebra’s subscriber requires the
tokio-console feature and experimental
tokio_unstable configuration.
Setup
Build from the repository root with both the feature and the compiler configuration enabled:
RUSTFLAGS="--cfg tokio_unstable" cargo build --locked -p zebrad \
--no-default-features --features tokio-console --bin zebrad
cargo install --locked tokio-console
The --no-default-features option removes release_max_level_info. Do not
enable any max_level_* or release_max_level_* features: they discard trace
events that runtime filters cannot restore. Zebra already enables Tokio’s
tracing feature.
Start the instrumented binary with the configuration for the node being investigated:
./target/debug/zebrad --config zebrad.toml start
In another terminal, connect the console client:
tokio-console
The subscriber and client use the same default endpoint.
Keep this diagnostics endpoint on loopback. Override the server address with
TOKIO_CONSOLE_BIND and configure the client to match; see the
subscriber configuration for other environment variables.
Interpreting the output
Inspect polling and wake activity for tasks consuming time or remaining pending. The resource view links tasks to synchronization primitives. For native call stacks and cryptographic computation, use sampling profiles.
If the console connects but shows no task data, check the build flags and
compile-time tracing filters before changing the node’s log filter. Zebra uses
console_subscriber::spawn(), which installs the console’s required target
filter separately from ordinary log output.
Private Testnet testing
A coordinated private Testnet tests a consensus transition across nodes, miners, and wallets using an agreed chain prefix. Use the fork procedure for state preparation and the custom parameter reference for the configuration schema. Use Regtest for tests that do not need public Testnet history.
Agree on the test network
Record and distribute these inputs before starting nodes:
- Exact source revisions or binary versions for every implementation.
- The public-prefix tip height and hash.
- The custom network name and unique four-byte magic.
- Every activation height and the first height at which private rules differ from public Testnet.
- The checkpoint file and its checksum.
- Funding streams, lockbox disbursements, proof-of-work policy, and any Network Sustainability Mechanism (NSM) seed or reissuance override.
- Participant P2P addresses, miner ownership, and the cases to test before, at, and after activation.
The first incompatible height is not necessarily the new private activation height. Delaying an upgrade that activates earlier on public Testnet makes the chains incompatible from the public activation onward. Keep all copied blocks below that boundary.
For NU7, choose an activation height divisible by three under ZIP 259, overriding the public schedule.
Check protocol compatibility
Zebra advertises protocol version 170190; configured Testnets require at
least that version from Overwinter onward, not public Testnet’s
upgrade-specific minimum
(implementation).
Check each implementation’s handshake rules: changing activation heights
does not change its advertised version.
Use the consensus branch identifier assigned to the implemented upgrade.
For NU7 it is 0x77190ad9. Do not invent a branch identifier to simulate an
implemented upgrade. A genuinely new protocol requires coordinated
implementation changes, including transaction construction and signature
verification, as described by
ZIP 200.
Check wallet and library parameters
Zebra passes its configured upgrade heights to librustzcash through
zcash_protocol::consensus::Parameters.
Other nodes, wallet libraries, and transaction producers must use the same
heights and branch identifiers. An unmodified wallet using public Testnet
activation heights is not sufficient evidence of private-upgrade compatibility.
For light-wallet tests, configure the
lightwalletd-compatible service and the wallet for
the private network. Wallet infrastructure is not required to submit an
already constructed transaction: Zebra accepts it through sendrawtransaction.
Use addresses whose keys the test participants control.
Prepare an isolated snapshot
Follow Prepare public state
for bounded snapshot or copy-state preparation and private storage paths.
Stop writers and direct database readers before copying; preserve an untouched
backup and use an empty destination. Follow the
upgrade and rollback procedure; do not rename or
symlink version directories to simulate an upgrade.
Copy only blocks valid under the private parameters. Do not copy a public peer cache or reuse a database from a previous incompatible test. A format upgrade preserves storage compatibility, not consensus compatibility.
Configure and connect participants
Use the isolated-network configuration. Change its activation height, prefix cutoff, and checkpoints together.
List participant endpoints in [network] initial_testnet_peers. Bind
[network] listen_addr to the intended private interface, and allow traffic
only between participants. A VPN or firewall supplies access control; network
magic does not authenticate peers. Initial peers are discovery seeds, not a
connection allowlist. Test reachability in each required direction.
Keep JSON-RPC on loopback unless another participant needs access. Retain cookie authentication and use a protected transport for remote RPC access. Give each local process a distinct listener, state root, and cookie directory.
Getting peers
Use the fork procedure’s
authenticated RPC examples.
Compare getpeerinfo on every node with the intended participants, then
compare getbestblockhash and getblockchaininfo. Begin activation testing
only when all nodes agree on the starting tip.
If peer caching is enabled, the fork example’s peer file is
network/nu7private-00010007.peers below the configured network cache root.
It contains cached addresses, not current connections; inspect connectivity
through RPC, not by patching address-redaction code.
Produce blocks and exercise the transition
Designate miners before starting the test. Proof-of-work-enabled private
Testnets require synchronization before getblocktemplate returns work, but
an old tip alone does not block mining as it does on Mainnet. The
[rpc] debug_force_finished_sync setting only changes the synchronization
status reported by getblockchaininfo; it does not bypass the mining check.
For proof-of-work tests, use the mining interface or the
experimental internal miner. The latter computes Equihash solutions; do not
assume it produces blocks immediately. For consensus tests that deliberately
exclude proof of work, agree on disable_pow = true and use generate or
generatetoaddress. Record that limitation with the results.
Exercise the agreed cases on both sides of activation:
- Confirm that nodes accept the same pre-activation blocks and transactions.
- Mine through activation and compare height, best-block hash, branch identifier, and upgrade status on every implementation.
- Submit valid post-activation transactions and deliberately invalid cases for the changed rules. Check both mempool admission and block validation.
- Test a restart and, when required by the test plan, a reorganization across the boundary. Verify state and transaction behavior after recovery.
For NU7, include the
ZIP 218 spacing and shielded-action limits,
ZIP 235 aggregate fee contribution, and
ZIP 237 reserve and reissuance behavior. NU7
activation and the reissuance height are distinct. Use an explicit
nsm_reissuance_height if the accelerated schedule needs reissuance before
its calculated crossover; every implementation must agree on the override.
Preserve configuration, checkpoint checksums, tip hashes, transaction and block identifiers, and the relevant logs with the test results. Never publish private RPC cookies or copy the fork back into public Testnet state.
Repository automation
The Regtest suite and NU7 cases cover templates, mempool transitions, funding, reserve accounting, reorganizations, and restarts locally, not multi-host private forks.
The zcashd-compat Regtest workflow
runs make compat-test-regtest with a pinned zcashd binary. Follow its
preparation steps: stock zcashd rejects Zebra’s null-proof-of-work Regtest
blocks. This sidecar suite does not rehearse a public-Testnet activation.
Refactoring consensus-critical code
Consensus-critical code determines which blocks and transactions Zebra accepts. Refactoring must preserve acceptance of valid inputs and rejection of invalid ones through production callers, not just helpers.
Use the protocol specification and applicable ZIPs to identify the rules.
Use the dependency versions in Cargo.lock, not an upstream repository’s
current branch, to determine which checks a library performs.
Author checklist
- Inventory rejection behavior. List the errors produced by the deserializers, constructors, conversions, and verifiers being replaced. Map each consensus rejection to its new location: an upstream check, a retained Zebra check, or another verification stage. Include implicit checks enforced by a type’s representable range.
- Trace production callers. Follow each affected path from wire bytes to verification, including block and mempool paths.
- Preserve checks when moving layers. Establish that every applicable production path still rejects the same invalid input. Do not move a check past an allocation or expensive operation that it protects from untrusted input. Test the new rejection point through the production entry point.
- Audit fallible conversions. Examine
.ok(),and_then,unwrap_or, default values, narrowing conversions, and optional accessors between the wire value and its check. Distinguish an absent field from an invalid value that could not be represented. Check similar accessors individually. - Test representation boundaries. Cover the largest valid value, first invalid value, first unrepresentable value, and maximum wire value. Include network-upgrade boundaries, malformed inputs, and valid vectors.
- Keep the change reviewable. Separate mechanical renames from behavior changes. Follow the rename procedure for code, tests, documentation, and log assertions affected by renamed symbols.
Regression examples
The transaction parsing tests exercise coinbase script-length boundaries through transaction deserialization, not just the transparent-input helper.
The same test module checks that Transaction::expiry_height() preserves
out-of-range wire values. A field that cannot be converted to an ordinary
height must not become None and bypass the expiry check. The test covers
499_999_999, 500_000_000, 2_147_483_648, and u32::MAX, as well as the
zero value that disables expiry. The transaction verifier tests exercise
rejection of out-of-range expiry heights for transaction versions 4, 5, and
6. The protocol’s transaction consensus rules distinguish non-coinbase
expiry bounds from the NU5-onward coinbase requirement; preserve that
distinction when changing the representation.
A regression test must fail when its production check is removed or bypassed.
Reviewer checklist
- The author checklist is satisfied, with a rejection inventory and source references for upstream checks.
- A full sync passes before release. Record the network, height range, and checkpoint configuration.
A sync tests acceptance of its chain, not rejection of malformed transactions.
For full-verification changes, set [consensus] checkpoint_sync = false;
mandatory checkpoints
still apply. Retain targeted rejection tests even when the sync succeeds.
Mass renames
Include affected documentation links, configuration examples, workflows, and log assertions in a Rust rename. Tests match or wait for log messages; compilation does not detect stale strings.
Separate mechanical replacements from semantic changes. Record commands, path scope, exclusions, and manual exceptions so reviewers can reproduce the change.
Interactive renames with fastmod
The fastmod tool previews each replacement for acceptance or rejection. Install it and restrict the search to affected directories:
cargo install --locked fastmod
fastmod --hidden --fixed-strings 'OldName' 'NewName' \
zebra-chain zebra-consensus zebrad book/src .github
The --hidden option includes hidden files and directories, including
.github. The tool still respects ignore files; --hidden does not mean
--no-ignore. Avoid searching generated build output.
The --fixed-strings option treats the search text literally. Without it,
fastmod uses Rust regular expressions. Replacement captures use ${1}, not
sed’s \1; quote replacement expressions with single quotes so that the
shell does not expand dollar signs. See the fastmod documentation for
replacement syntax.
Review each match: identical text can name unrelated types or fixed external interfaces. Record skipped matches; commands alone do not reproduce an interactive run.
Noninteractive renames with GNU sed
Use tracked text files and NUL-delimited paths for uniform replacements.
This example requires GNU sed and xargs:
git grep -Ilz -F -e 'OldName' -e 'OtherOldName' -- \
'*.rs' '*.md' '*.toml' '*.yml' '*.yaml' \
':(exclude)CHANGELOG.md' ':(exclude)**/CHANGELOG.md' |
xargs -0 -r sed -i \
-e 's/OldName/NewName/g' \
-e 's/OtherOldName/OtherNewName/g'
The search excludes binary files (-I) and outputs matching paths (-l)
terminated by NUL (-z). The xargs -0 option preserves spaces and newlines;
-r skips sed when no files match. Add exclusions for files that must not
change. Update changelog source fragments, not generated release history.
Sed replaces substrings, not Rust identifiers. Escape regex and replacement metacharacters when adapting these examples. The in-place syntax is not portable to BSD sed on macOS; use fastmod for portability or per-match review.
Preparing and reviewing the change
-
Start from a clean worktree and record the exact base commit. Apply the replacements only within the intended scope.
-
Format the Rust code after completing all replacements:
cargo fmt --all -
Search the tracked tree for each old name. Classify remaining matches as intentional or missed replacements. Run affected tests, including integration tests when log text changes, and build the documentation.
-
Include the replacement commands, formatter/toolchain versions, and manual exceptions in the review evidence.
To reproduce a noninteractive rename, create a separate detached worktree at
the recorded base commit. Replace BASE_COMMIT below with that commit:
git worktree add --detach ../zebra-rename-replay BASE_COMMIT
Run the recorded replacements and formatter in that worktree. Export its
patch relative to the base, and export the proposed patch from the original
repository. Replace PR_COMMIT with the proposed commit:
git -C ../zebra-rename-replay diff --binary BASE_COMMIT > replay.patch
git diff --binary BASE_COMMIT PR_COMMIT > proposed.patch
git diff --no-index -- replay.patch proposed.patch
The final command exits with 0 for matching patches or 1 for differences. Compare differences with the recorded exceptions. Do not compare entire worktrees: Git metadata and build output differ. Remove temporary patch files.
Reproducibility does not establish correctness. Review meaning, public API impact, log consumers, and tests; review each interactive replacement.
Using rustdoc links to detect name changes
Use intra-doc links for Rust items in documentation comments. Rustdoc resolves these links and reports unresolved targets; Zebra’s documentation workflow treats warnings as errors.
#![allow(unused)]
fn main() {
//! Use [`u32`] for the integer type and [`drop()`] for the function.
//! Call [`Clone::clone()`] to clone a value.
//! Match [`Option::None`] or [`Option::Some`].
//! Store ordered keys in [`BTreeSet<String>`](std::collections::BTreeSet).
}
Use a fully qualified link target when an item is not in scope. Do not add an otherwise unused import solely to make a documentation link resolve. Rustdoc cannot validate ordinary strings or prose that names an item without a link, so retain the text search and log-consumer review.
Updating Zcash protocol dependencies
Zebra shares transaction, proof, key, and script libraries with other Zcash implementations. Review consensus validation and shared types alongside upstream changes and the resolved dependency graph; a successful build does not establish consensus compatibility.
Dependency inventory
The root Cargo.toml holds most direct requirements;
member manifests select features and package aliases. The table lists version
requirements, not the resolved versions in Cargo.lock.
| Group | Direct dependencies and requirements |
|---|---|
| Transaction and protocol libraries | zcash_primitives = "0.31.0", zcash_proofs = "0.31.0", zcash_protocol = "0.11.0", zcash_transparent = "0.11.0" |
| Addresses and keys | zcash_address = "0.14.0", zcash_keys = "0.17.0" |
| Shielded protocols | orchard = "0.16.0", sapling-crypto = "0.9", zcash_note_encryption = "0.5.1" |
| Encoding and trees | zcash_encoding = "0.5", zcash_history = "0.6.0", incrementalmerkletree = "0.9" |
| Transparent scripts | libzcash_script = "0.2", zcash_script = "0.6" |
The cryptographic dependency graph also includes bellman, bls12_381,
equihash, ff, group, groth16, jubjub, reddsa, redjubjub, and
sinsemilla. Several member manifests depend on halo2_proofs = "0.4" under
the local name halo2. Check those declarations as well as the workspace
requirements. The halo2_gadgets package is a transitive dependency.
The script crate uses libzcash_script for the default C++ interpreter from
zcashd and zcash_script for Rust types and interpreter support. Test changes
with the comparison-interpreter feature.
Match versions where crates exchange types or require compatible
implementations, not to a zcashd release’s entire dependency list:
libzcash_script depends on zcash_script, not the Orchard or Sapling graph.
Select and apply the update
-
Record the target releases and read their changelogs. Identify changes to serialization, consensus checks, feature defaults, proof verification, and public types. Apply the consensus refactor checklist when validation moves between Zebra and an upstream library.
-
Check the dependency requirements of the selected upstream releases. If script changes require a new
libzcash_scriptorzcash_scriptrelease, coordinate that release in the script repository before selecting it in Zebra. Do not require a script release for an unrelated library update. -
Update the relevant
[workspace.dependencies]entries and any member-level aliases or feature selections. Do not replace inherited workspace requirements with independent per-crate versions. -
Update
Cargo.lockfor the selected packages:cargo update -p orchard -p sapling-crypto \ -p zcash_primitives -p zcash_proofs cargo tree --workspace --duplicates cargo tree --workspace --invert zcash_primitivesAdjust the package list to the update. If more than one version of a package is present, use a version-qualified package specification such as
[email protected]for an inverse tree. Review transitive changes in the lockfile; a targeted update can still update required dependencies. -
Adapt callers without weakening validation, following the consensus refactor checklist.
Dependency policy and audits
The deny policy rejects unapproved duplicate versions, registries, Git
sources, and licenses. It also checks security advisories. Remove obsolete
exceptions when the update eliminates their dependency paths. Do not add a
broad skip-tree exception to hide an unexplained duplicate.
Install cargo-deny and run the default-feature checks from the workspace root:
cargo deny --workspace check bans sources licenses advisories
For CI-only failures, reproduce the lint feature matrix and allowances rather than weakening the shared policy.
Zebra also stores cargo-vet review records in supply-chain/:
config.toml: imported audit sources, package policy, and exemptions.audits.toml: local full-version and version-to-version audit records.imports.lock: the imported audit records used by the repository.
Run the audit check after updating the dependency graph:
cargo vet
For missing coverage, inspect the package with cargo vet inspect, or review
the version delta with cargo vet diff. After completing that review, record
the applicable criteria with cargo vet certify. These commands take the
package name and the version or version pair reported by the tool; see the
cargo-vet command reference. Commit the resulting supply-chain changes with
the dependency update. An exemption records missing audit coverage, not a
successful audit; document and review any required exemption explicitly.
Validation and review
Run affected tests and feature combinations, including comparison-interpreter
for script changes, and the CI checks. For
consensus-sensitive updates, follow the refactor validation requirements.
The pull request must identify the upstream releases, explain necessary version coupling and policy exceptions, and record test evidence. Keep unrelated dependency changes out of the update so that reviewers can trace behavior changes to their source.
Zebra checkpoints
A checkpoint is a trusted block height and hash compiled into Zebra. The checkpoint verifier queues blocks until their hashes link two checkpoints, then commits the chain as finalized state, avoiding most contextual checks during initial synchronization. Review checkpoint updates as consensus-critical data: they change which chain Zebra trusts.
Checkpoint coverage and spacing
Mandatory checkpoints are a Zebra-specific mechanism, not a Zcash consensus rule. Zebra cannot fully validate pre-Canopy blocks. Its mandatory checkpoint height is the block immediately before Canopy: 1,046,399 on Mainnet and 1,028,499 on the default Testnet.
The checkpoint_sync setting selects the full list or its first checkpoint at or above the mandatory height. Blocks above the selected checkpoint use full verification.
The generator uses these spacing constants:
| Limit | Value | Effect |
|---|---|---|
MAX_CHECKPOINT_HEIGHT_GAP | 400 blocks | Emit a checkpoint when the height gap reaches 400. |
MAX_CHECKPOINT_BYTE_COUNT | 32 MiB (33,554,432 bytes) | Emit a checkpoint when cumulative serialized block size reaches this threshold. |
MAX_BLOCK_REORG_HEIGHT | 1,000 blocks | Exclude the recent chain segment when generating checkpoints. |
The byte threshold is evaluated after adding a block, so an interval can exceed 32 MiB. These limits bound queued work and memory, not protocol-valid checkpoint placement.
The generator reads the source node’s height once and considers only heights
strictly below that height minus 1,000. It always includes genesis when
starting a new list. Otherwise, it starts immediately after --last-checkpoint
and emits only new checkpoint lines. Each line has the form height hash.
Automated updates
The checkpoint update workflow consumes artifacts from
Integration Tests on GCP. The generator jobs use synced Mainnet and Testnet
states. A successful integration run on main triggers the update workflow;
a manual dispatch selects the latest successful run with both artifacts:
generate-checkpoints-mainnet-checkpointsgenerate-checkpoints-testnet-checkpoints
The workflow appends entries above each file’s last height, validates the
result, and opens or updates the chore/update-checkpoints pull request for:
zebra-chain/src/parameters/checkpoint/main-checkpoints.txtzebra-chain/src/parameters/checkpoint/test-checkpoints.txt
The workflow skips runs missing either artifact and unchanged files. The pull request links the source integration run.
Manual generation
Use a synchronized node with JSON-RPC enabled. Verify its network before collecting hashes. From the repository root, build the utility from the same source as the checkpoint lists:
cargo build --release --locked -p zebra-utils \
--features zebra-checkpoints --bin zebra-checkpoints
./target/release/zebra-checkpoints --help
The binary requires zebra-utils/zebra-checkpoints; the
same-named zebrad feature enables tests.
The utility accepts these options:
| Option | Default | Meaning |
|---|---|---|
-b, --backend | zebrad | Use zebrad for raw-block RPC responses, or zcashd for verbose responses containing height, hash, and size. |
-t, --transport | cli | Use zcash-cli as a subprocess, or direct for HTTP JSON-RPC. |
-c, --cli | zcash-cli | RPC client executable for the cli transport. |
-a, --addr | Transport-dependent | RPC IP address and port. Direct mode defaults to 127.0.0.1:8232; CLI mode otherwise uses the client’s configuration. |
-l, --last-checkpoint | None | Last existing checkpoint height; omit only when generating from genesis. |
-- followed by arguments | None | Pass arguments to zcash-cli, such as -testnet or authentication settings. |
Direct transport sends no authentication credentials. For the example below,
use a dedicated local node with [rpc] listen_addr = "127.0.0.1:8232" and
[rpc] enable_cookie_auth = false. Never disable authentication on a publicly
reachable endpoint. To retain RPC authentication,
use CLI transport with the node’s credentials configured in zcash-cli.
Generate a Mainnet extension into a temporary output file rather than appending directly to the tracked list:
checkpoints=zebra-chain/src/parameters/checkpoint/main-checkpoints.txt
last_checkpoint=$(tail -n 1 "$checkpoints" | cut -d ' ' -f 1)
./target/release/zebra-checkpoints \
--backend zebrad --transport direct --addr 127.0.0.1:8232 \
--last-checkpoint "$last_checkpoint" > new-main-checkpoints.txt
For Testnet, use test-checkpoints.txt, a Testnet node’s RPC address, and a
separate output file. The utility has no network-selection flag: direct mode
uses the network of the node at --addr; CLI mode can pass -- -testnet.
The zebrad backend also works with zcashd raw-block responses.
Check that generation exited successfully before appending the output. Generation can leave a partial output file after an RPC failure. An empty successful output means that no new checkpoint met the spacing thresholds.
Review and validation
-
Compare every added height and hash with the source run’s artifact or the source node’s block at that height. Check the network, source height, and settlement margin. Preserve existing entries during a routine extension.
-
Check that the first new checkpoint continues the existing list without duplicates or a gap above 400 blocks. For independent confirmation, compare the added hashes with another fully synchronized node on the same network.
-
Append the reviewed output to the corresponding tracked file. Remove the temporary output files after retaining the source evidence in the pull request.
-
Run the structural validator and checkpoint-list tests:
.github/scripts/validate-checkpoints.sh zebra-chain/src/parameters/checkpoint/main-checkpoints.txt .github/scripts/validate-checkpoints.sh zebra-chain/src/parameters/checkpoint/test-checkpoints.txt cargo nextest run --locked -p zebra-chain checkpoint_list
The script checks line format, genesis height, increasing heights, duplicate heights and hashes, and the 400-block maximum gap. Rust tests also check mandatory coverage and the minimum gap derived from the byte threshold and maximum block size. Neither establishes that a hash belongs to the intended chain; review the source data.
Zebra continuous integration
Zebra uses GitHub Actions for pull-request checks, container builds, and stateful tests on Google Cloud Platform (GCP). See the workflow inventory and dependency diagrams; workflow YAML defines triggers, permissions, and job conditions.
Required checks
The main branch ruleset requires these status contexts:
| Required context | Producer | Purpose |
|---|---|---|
lint | lint.yml, job lint | Rust, dependency, Dockerfile, and shell checks |
unit-tests | tests-unit.yml, job unit-tests | Linux tests, Windows zebra-test tests, and release-only dependency checks |
test-crates | test-crates.yml, job test-crates | Per-crate builds with different features and minimum supported Rust versions |
pr-gate-result | pr-gate.yml, job pr-gate-result | Pull-request declarations, change fragments, and release readiness |
merge-policy | merge-policy.yml, job merge-policy | Rejects the do-not-merge label |
mergefreeze | Merge Freeze GitHub App | Release-window hold |
The first four contexts aggregate worker results. An aggregator runs with
if: always() and uses re-actors/alls-green to interpret failures,
cancellations, and explicitly permitted skips. A green aggregator does not
mean every worker ran. The pr-gate-result aggregator also permits failures
from api-diff and changelog-gate; review their findings separately.
Path filters
The shared path filters
gate lint, unit-test, and crate-build workers through dorny/paths-filter
on PRs and merge groups, while retaining required status reporting.
Pushes to main bypass filtering and set worker-selection outputs to true.
The filter keys include lint, hadolint, shellcheck, api_diff,
unit_tests, test_crates, test_docker, book, book_links, coverage,
and docs_check. Edit the shared file rather than duplicating these globs in a
new job. Each filter includes the shared file itself. GCP integration and
node-deployment workflows instead have their own event and path filters;
changing the shared file does not automatically run those workflows.
The unit_tests filter also matches three book pages that unit tests compare
with the code: the configuration reference and the network upgrade table
(zebrad/tests/unit/book.rs), and the JSON-RPC method table
(zebra-rpc/src/methods/tests/book.rs). A change that adds a configuration key
or an RPC method, or changes an activation height, fails these tests until the
book is updated.
On PRs targeting main, labels other than release do not rerun the main
Linux unit-test job. PRs targeting next run it on edits and label events
when the path filter matches; next has no merge queue to rerun it.
Add gated workers to the aggregator’s needs and legitimate skips to
allowed-skips; retain merge_group. Require a new context only after a
workflow produces it. The crate matrix discovers workspace crates with
cargo tree; new crates need no matrix entry.
Documentation
The Docs Check workflow
runs on PRs and merge groups that change Markdown or the book: spell-check
(codespell) and markdown-lint (markdownlint) under the docs_check filter,
and book-links under the book_links filter, which builds the book and
checks its internal links and anchors offline. It is not a required
context. External links are checked weekly by the
Advisory Checks workflow.
The Book workflow
publishes https://zebra.zfnd.org/ on every push to main and on manual runs.
The site root is the book of the latest release, so operators read the
documentation of the version they can install; the book of main is under
/main/, and the internal rustdoc of main under /internal/. A new release
reaches the root on the next run. To publish another ref at the root, for
example main when a documentation fix should not wait for a release, set the
BOOK_ROOT_REF repository variable; delete it to return to the latest
release.
Runners and build caches
Linux unit tests use Rust 1.97.0, ubuntu-26.04, cargo nextest with the
ci profile, release mode without link-time optimization, and
default-release-binaries. Windows tests cover only zebra-test on
windows-latest.
Reuse the build setup action
for Linux Rust jobs needing Clang, Protocol Buffers, or system RocksDB;
it installs them and sets ROCKSDB_LIB_DIR=/usr/lib/.
Follow the Rust cache policy and inspection commands when adding jobs, changing keys, or deleting caches.
The reusable container builder defaults to ubuntu-26.04 for AMD64 and
ubuntu-24.04-arm for ARM64. Repository variables
DOCKER_BUILD_RUNNER_AMD64 and DOCKER_BUILD_RUNNER_ARM64 override them.
Runtime images build for both architectures; test images build for AMD64.
The DOCKER_BUILDER variable selects Docker Build Cloud when set to
cloud; otherwise the default is the docker-container driver. The
no_cache dispatch input disables Docker build-cache reads, not Rust
caches or cached blockchain state.
Integration tests
GCP stateful tests
The Integration Tests on GCP workflow does not run on every PR update:
| Event | Selection |
|---|---|
Pull request targeting main | labeled, synchronize, or reopened; the head must be in the repository and the PR must carry run-stateful-tests |
Push to main | Rust, dependency, test-data, Docker, or explicitly listed integration-workflow paths |
| Schedule | Friday at 12:00 UTC, only in ZcashFoundation/zebra |
| Manual dispatch | Selected ref and network, with optional cache-regeneration or full-sync inputs |
The workflow has no merge_group trigger and does not gate merges.
For fork coverage, review the code before copying it to a repository-owned
branch and dispatching the secret-bearing workflow or labeling its PR.
Run the ordinary suite on a repository-owned branch:
gh workflow run zfnd-ci-integration-tests-gcp.yml \
--repo ZcashFoundation/zebra --ref my-branch -f network=Mainnet
Select regenerate-disks=true to rebuild checkpoint state,
run-full-sync=true to rebuild tip state for the selected network, or
run-lwd-sync=true for a Mainnet lightwalletd full sync. These inputs retain
cache prerequisites.
Tests cover full validation beyond the mandatory checkpoint, tip updates,
checkpoint generation, and RPCs. Full Zebra syncs run weekly, on request,
or when matching tip state is missing. Mainnet lightwalletd update,
transaction, and gRPC wallet jobs run on schedules, manual dispatches, and
eligible labeled PRs, not main pushes. Both Testnet jobs pass
allow_failure: true to the reusable workflow.
The test deployment workflow
runs tests in a GCP VM container; test-result uses self-hosted
zfnd-runners to supervise and collect logs. Timeouts are 180 minutes,
or 7,200 minutes for is_long_test. The entrypoint uses NEXTEST_PROFILE
and NEXTEST_FILTER; ci-stateful and ci-e2e require an explicit filter.
GCP jobs request id-token: write and authenticate through Workload
Identity Federation using GCP_WIF and the appropriate service-account
variable. Builds use GCP_ARTIFACTS_SA; VM and disk operations use
GCP_DEPLOYMENTS_SA. Container builds receive DOCKERHUB_USERNAME and
DOCKERHUB_TOKEN; test supervision receives GCP_SSH_PRIVATE_KEY. Approving
a fork workflow does not provide these repository secrets.
Cached blockchain state
The cached-disk workflow
searches Compute Engine images using the database major version from
zebra-state/src/constants.rs and the lowercased network. It reports Zebra
checkpoint, Zebra tip, and combined lightwalletd/Zebra tip availability.
Tests create private disks from the selected image.
When state saving is selected, create-state-image runs after successful
test-result. It saves updated images when the height exceeds the input
image by more than 576 blocks; full-sync images and force_save_to_disk=true
bypass that threshold. See image selection.
Cross-project interoperability tests
The separate Trigger Integration Tests workflow
sends a zebra-interop-request repository-dispatch event to
zcash/integration-tests. It runs
on Mondays at 06:00 UTC and on manual dispatch, not automatically on PRs.
Its GitHub App token comes from Z3_APP_ID and Z3_APP_PRIVATE_KEY.
To test a PR’s head commit, including a fork PR, dispatch from main with
its PR number:
gh workflow run trigger-integration-tests.yml \
--repo ZcashFoundation/zebra --ref main -f pr_number=12345
Replace 12345 with the PR number; omit pr_number to send the selected
ref’s commit. Inspect results in the integration-tests repository: a
successful dispatch does not mean its tests passed.
Benchmarks
The benchmark workflow
runs six targets listed in BENCHES. Register new targets there and in the
crate’s Cargo.toml with [[bench]] and harness = false. Manual dispatch
runs cargo-criterion and publishes results to gh-pages/dev/bench. The
book workflow
includes that dashboard in the documentation deployment. The alert
threshold is 150% of the previous result; alerts do not fail the job.
PRs labeled run-benchmarks run base and head targets on label events and
subsequent pushes, save Criterion baselines, and post the critcmp comparison
to the run summary and a PR comment. Failed or missing base targets appear
in the summary without a comparison. See
benchmark invocation and results.
Coverage
The coverage workflow
sets PROPTEST_CASES=1 and PROPTEST_MAX_SHRINK_ITERS=0. These reduced runs
do not replace property testing.
Automated merges
A maintainer selects Merge when ready after the PR meets the review and
check requirements. GitHub’s merge queue tests a cumulative merge result
against main; a PR need not be up to date before enrollment. Inspect the
merge_group run when a queue entry fails, not only the source PR’s runs.
PRs targeting next merge directly after their required
checks; they do not use this queue.
Queue configuration
The PR Requirements ruleset requires one approving review and resolved
review conversations. See the
queue settings
and repository rules
before changing policy.
The merge-policy check rejects do-not-merge on both a PR and the PRs
identified in a merge-group ref. The external mergefreeze status is a
separate hold; the release label does not bypass it. Manage freezes and
release exceptions through Merge Freeze.
For an immediate freeze, remove active non-release queue entries rather
than relying on a status already reported for a merge group being revoked.
Repository administrators change required contexts in the ruleset. Keep each context pinned to its producing app. Document any emergency bypass on the PR and restore normal enforcement after repairing the branch.
Troubleshooting
Start with the earliest failed build or test, not a downstream aggregation or missing-output error. For stateful tests, inspect the selected image, VM creation, container logs, test exit status, image creation, and cleanup in order. Rerun the complete integration workflow when its VM has been removed; rerunning only a later job does not recreate its prerequisites.
For a sync timeout, inspect peer connectivity, cached-state height, and recent performance changes before extending the timeout. The checkpoint procedure covers updating checkpoints. Do not treat a repeated test failure as a transient infrastructure error.
Fixing duplicate dependencies in Check deny.toml bans
The lint workflow
checks bans, sources, advisories, and licenses with default features,
default-release-binaries, and all features. Its all-features jobs use
--allow banned because that set activates optional OpenSSL; the other
feature sets use --allow unmatched-skip-root. Advisory failures are
non-blocking.
The bans jobs are named Check deny bans with a feature-selection suffix.
Use cargo tree -d to locate duplicate versions and
cargo tree -i crate@version to find their source. Prefer compatible
dependency updates or removing unused features over an exception.
When an exception is necessary, document its cause in
deny.toml.
Remove obsolete skip-tree entries when the all-features bans check
reports an unmatched skip root:
cargo deny --all-features check bans
Resource failures
For full disks, distinguish GitHub runner storage, Docker build storage,
and GCP state disks. Integration VMs request 400 GB pd-balanced disks.
Changing the creation size does not resize an existing persistent disk.
Inspect the failing branch’s Rust cache before deleting anything. Avoid clearing all repository caches to address one failure. GCP resource retention and node recovery procedures are in the deployment runbook.
Zebra continuous delivery
The deployment wrapper
builds a container image and passes its immutable Google Artifact Registry
(GAR) identity, repository@sha256:digest, to the
reusable deployment workflow.
The source commit is a separate, full 40-character SHA input.
See the workflow inventory, deployment runbook, and naming rationale.
Triggers and environments
| Trigger | GitHub environment | Deployment |
|---|---|---|
Matching push to main | stage | Mainnet and Testnet in all three zones |
Matching push to next | next | Mainnet and Testnet in all three zones |
release: released with a tag starting with v | prod | Mainnet and Testnet in all three zones |
| Manual dispatch | Selected dev, next, or prod | One selected network and zone |
| Matching repository-owned PR | dev | Image build only; no node deployment |
Push and PR path filters include Rust source, text files, Cargo manifests
and lockfiles, .cargo/config.toml, Clippy configuration, docker/**,
.dockerignore, and the three build/deployment workflow files listed in
the wrapper. These filters are independent of .github/path-filters.yml.
A documentation-only push does not deploy nodes. Fork PRs skip the
wrapper’s image build.
The released activity covers stable publication and promotion of a
prerelease to a stable release. Prerelease publication does not trigger this
workflow. A release whose tag marks a pre-release from next,
any pre-release version other than a release candidate, never deploys, even
after promotion. Manual dispatch still uses the selected environment. Docker Hub
publication belongs to release automation, not this deployment wrapper.
See the release process before publishing a release
or dispatching a production deployment.
Deployment jobs use environment-scoped machine type, subnet, region, logging, and Sentry variables. Authentication uses the GCP workflow credentials.
Image publication
The image builder
publishes to us-docker.pkg.dev/<GCP_PROJECT>/zebra/<image_name>.
The GCP_PROJECT variable comes from the selected GitHub environment.
| Caller | Image name | Dockerfile target |
|---|---|---|
release-binaries.yml | zebra | runtime |
zfnd-deploy-nodes-gcp.yml | zebrad | runtime |
zfnd-ci-integration-tests-gcp.yml | Value of CI_IMAGE_NAME | tests |
Docker Hub publication additionally requires publish_to_dockerhub=true.
Only the release-binaries caller enables it, for every published release and
prerelease. Nonrelease builds publish only to GAR, not Docker Hub or GitHub
Container Registry (GHCR); the builder has no GHCR destination.
The tag recipes produce these development tags; matching conditions combine:
| Ref or event | Tag |
|---|---|
| Pull-request ref | pr-<number> |
| Branch ref, including manual dispatch from a branch | Sanitized branch name |
Prerelease tag from next (any prerelease except -rc) | edge |
| Scheduled run | nightly, the default type=schedule pattern |
| Nonrelease event with a commit SHA | sha-<first-seven-characters> |
The pinned metadata action
defines these defaults. The workflow disables SHA tags for release events in
both registries; PR, branch-push, scheduled, and manual runs keep them for CI.
For version tags and latest, see
Docker images.
Zonal instance groups
A managed instance group (MIG) maintains the requested VM configuration.
Zebra uses one zonal MIG per environment, branch identity, network, and
zone. Each new MIG has one instance and one persistent chain-state disk.
The supported zones are us-east1-b, us-east1-c, and us-east1-d.
Resource names use lowercase mainnet or testnet and the final zone
letter.
| Environment | Project | MIG | State disk |
|---|---|---|---|
prod | zfnd-prod-zebra | zebrad-${network}-${zone-letter} | zebrad-cache-${network}-${zone-letter} |
stage | zfnd-dev-zebra | zebrad-main-${network}-${zone-letter} | zebrad-cache-main-${network}-${zone-letter} |
next | zfnd-dev-zebra | zebrad-next-${network}-${zone-letter} | zebrad-cache-next-${network}-${zone-letter} |
dev | zfnd-dev-zebra | zebrad-${ref-slug}-${network}-${zone-letter} | zebrad-cache-${ref-slug}-${network}-${zone-letter} |
Manual dispatch selects the source ref independently of the destination. Stable destinations replace their MIGs even from a feature branch; check the dispatch targets.
Update mechanics
The reusable workflow validates its image, source SHA, environment, network list, and zone list. It then ensures the network health checks exist and resolves cached state separately for each network when requested.
Each deploy-nodes matrix job:
- Rejects a disk held by an instance belonging to another MIG.
- Reuses the named state disk, or creates a 400 GB
pd-balanceddisk from a matching cache image. A missing disk without a cache image is an error; the workflow does not create an empty disk. - Creates or reuses an instance template whose name includes a digest of
its configuration. The template runs the immutable container image on
a Spot VM using
cos-cloud/cos-stableand mounts the state disk at/home/zebra/.cache/zebra. - Creates a size-one MIG if absent. The disk’s stateful policy preserves it across VM replacement but deletes it on permanent instance deletion.
- Updates an existing MIG with
--replacement-method=recreate,--max-surge=0, and--max-unavailable=1. - Waits for the requested template, then optionally for group stability; see deployment verification.
A per-MIG concurrency group covers both waits and does not cancel an active
deployment. Matrix fail-fast is disabled. The six network/zone jobs run
independently: the workflow does not serialize zones. The one-instance
unavailability limit applies within each zonal MIG, not across a network’s
three MIGs. Use single-zone dispatches when the operation requires other
zones to remain untouched.
Manual dispatches omit the stability wait. Check application health before serving clients.
State and addresses
Push deployments request cached images for new disks. Release deployments
disable cache lookup and require pre-existing disks. Manual dispatch
defaults to need_cached_disk=true and cached_disk_type=tip;
need_cached_disk=false means reuse a disk already present, not initialize
an empty one. Existing disks do not require a matching image. The runbook
covers cache-image lookup and
manual bootstrapping.
Stable deployments use reserved external IPs.
Other feature-branch dev deployments use --no-address and do not set
ZEBRA_NETWORK__EXTERNAL_ADDR; outbound connectivity uses the configured
subnet.
GCP deployment operations
Use the continuous-delivery topology
to select the project, zone, managed instance group (MIG), and state disk.
Authenticate gh and gcloud with authorized repository and project
accounts. Set the target explicitly; these values select one stage
Mainnet node:
P=zfnd-dev-zebra
ZONE=us-east1-b
MIG=zebrad-main-mainnet-b
DISK=zebrad-cache-main-mainnet-b
For next, use the next- prefix instead of main-. For production, set
P=zfnd-prod-zebra and omit the prefix:
MIG=zebrad-mainnet-b, DISK=zebrad-cache-mainnet-b.
Use --zone for these MIGs, not --region.
Instance templates and new disks receive app, environment, network,
zone, created_by, github_ref, and github_sha labels; instances
inherit template labels. Existing disk labels are not refreshed. Find MIGs
by name: they do not receive these labels.
Deploy success has two stages
Each deployment job reports:
- Template rollout:
wait-until --version-target-reached --timeout=600. - Group stability:
wait-until --stable --timeout=5400, only withverify=true.
The per-MIG lock covers both stages.
Pushes and releases enable the second stage; manual dispatches do not.
Check instanceHealth and container logs even after the group is stable.
The workflow creates an HTTP health check on port 8080 at /healthy, with
a 60-second interval, a 10-second timeout, three failed probes to become
unhealthy, and two successful probes to become healthy. New MIGs have a
3,600-second autohealing initialization delay. The node requires at least
one recently live peer through ZEBRA_HEALTH__MIN_CONNECTED_PEERS=1.
Check sync progress separately before serving clients:
/healthy checks liveness, not sync readiness.
Rollout or stability failures print instance state and recent container
logs; failure-issue opens or updates an issue labeled ci-fail/release.
Investigate the failing stage.
Automatic deployments do not serialize zones. Use one-zone manual deployments for a controlled migration.
PR deploys
Run one
Dispatch from a repository-owned feature branch to create or update one branch-specific MIG:
gh workflow run zfnd-deploy-nodes-gcp.yml \
--repo ZcashFoundation/zebra --ref my-branch \
-f network=Mainnet -f zone=us-east1-b -f environment=dev \
-f need_cached_disk=true -f cached_disk_type=tip
The network input accepts Mainnet or Testnet. The environment input
accepts dev, next, or prod; there is no stage choice. To update stage
manually, select dev from the main ref. That targets stage resource
names but uses dev environment variables and Sentry labeling.
Selecting next or prod targets the stable environment regardless of
the selected source ref. Selecting dev from next also targets next’s
resource names. Do not use these combinations for an isolated PR test.
Find yours
Resource names use the URL-safe ref slug produced by
rlespinasse/github-slug-action. For the example branch:
gcloud compute instance-groups managed list --project "$P" \
--filter='name=zebrad-my-branch-mainnet-b'
gcloud compute instances list --project "$P" \
--filter='labels.created_by=workflow_dispatch AND labels.github_ref=my-branch'
See branch-deployment networking for external IP and outbound connectivity behavior.
Reap one
Confirm that the MIG is a disposable branch deployment, not stage, next,
or production. The workflow’s disk policy is
auto-delete=on-permanent-instance-deletion: deleting the MIG or reducing
its size can delete its state disk. Snapshot state that must survive first.
MIG=zebrad-my-branch-mainnet-b
ZONE=us-east1-b
gcloud compute instance-groups managed delete "$MIG" \
--zone "$ZONE" --project "$P"
Keep the confirmation prompt. Remove a template only after checking that no
MIG references it. Daily cleanup does not reap branch MIGs;
keep_until and delete_protection labels do not protect resources.
Diagnose a stuck MIG
Inspect template convergence, instance health, and creation errors:
gcloud compute instance-groups managed describe "$MIG" \
--zone "$ZONE" --project "$P" \
--format='yaml(targetSize,currentActions,status,versions,statefulPolicy)'
gcloud compute instance-groups managed list-instances "$MIG" \
--zone "$ZONE" --project "$P" \
--format='table(instance.basename(),instanceStatus,currentAction,instanceHealth[0].detailedHealthState,lastAttempt.errors.errors[0].message)'
gcloud compute instance-groups managed list-errors "$MIG" \
--zone "$ZONE" --project "$P" --limit=10
For a disk attachment conflict, inspect the disk’s users:
gcloud compute disks describe "$DISK" --zone "$ZONE" --project "$P" \
--format='yaml(name,sizeGb,users)'
Inspect each reported instance’s created-by metadata to identify its MIG.
Do not delete an instance or another MIG merely because it holds the disk.
Confirm which deployment owns the state, stop competing deployments, and
preserve any required disk before removing an obsolete owner. A read-write
state disk must not be shared between independent MIGs.
Read recent container logs for the selected instance:
INSTANCE=$(gcloud compute instance-groups managed list-instances "$MIG" \
--zone "$ZONE" --project "$P" --format='value(instance.basename())')
INSTANCE_ID=$(gcloud compute instances describe "$INSTANCE" \
--zone "$ZONE" --project "$P" --format='value(id)')
gcloud logging read \
"logName=\"projects/${P}/logs/cos_containers\" AND resource.labels.instance_id=\"${INSTANCE_ID}\"" \
--project "$P" --freshness=2h --limit=100 \
--format='value(jsonPayload.message)'
These commands require exactly one instance. If there are none, inspect creation errors first. An ordinary deployment updates a size-zero MIG’s template without resizing it.
Recover a corrupted cache disk
Recover one zone at a time. Stop concurrent deployments to the same MIG before changing its disk. Select a known-good snapshot or compatible cache image before deleting anything, and retain a snapshot of the failed disk for diagnosis.
Detach and preserve the existing disk
Change the group’s disk policy to preserve the disk on permanent deletion:
gcloud compute instance-groups managed update "$MIG" \
--zone "$ZONE" --project "$P" \
--stateful-disk="device-name=${DISK},auto-delete=never"
gcloud compute instance-groups managed describe "$MIG" \
--zone "$ZONE" --project "$P" --format='yaml(statefulPolicy)'
gcloud compute instance-groups managed instance-configs list "$MIG" \
--zone "$ZONE" --project "$P" --format=json
Confirm the effective disk policy
is NEVER; change any conflicting per-instance auto-delete rule to never
before proceeding. Preserve reserved external addresses; their workflow
policy already uses auto-delete=never.
Delete only the selected MIG, then confirm that the disk has no users:
gcloud compute instance-groups managed delete "$MIG" \
--zone "$ZONE" --project "$P"
gcloud compute disks describe "$DISK" --zone "$ZONE" --project "$P" \
--format='yaml(name,users)'
Do not continue while the disk is attached. Snapshot the detached disk:
SNAPSHOT="${DISK}-recovery-$(date -u +%Y%m%d-%H%M%S)"
gcloud compute snapshots create "$SNAPSHOT" --project "$P" \
--source-disk "$DISK" --source-disk-zone "$ZONE"
gcloud compute snapshots describe "$SNAPSHOT" --project "$P" \
--format='yaml(status,sourceDisk,creationTimestamp)'
Require READY before discarding the original disk. This snapshot preserves
the failed state; it is not the known-good restore source.
Restore and redeploy
Set RESTORE_SNAPSHOT to the verified, known-good snapshot in the selected
project. Check its source disk, network, and database compatibility:
RESTORE_SNAPSHOT=known-good-snapshot
gcloud compute snapshots describe "$RESTORE_SNAPSHOT" --project "$P" \
--format='yaml(status,sourceDisk,diskSizeGb,creationTimestamp)'
After confirming it is READY, replace the disk under the same name:
gcloud compute disks delete "$DISK" --zone "$ZONE" --project "$P"
gcloud compute disks create "$DISK" --zone "$ZONE" --project "$P" \
--source-snapshot "$RESTORE_SNAPSHOT" --type=pd-balanced
Alternatively, create the disk from a verified cache image.
Dispatch the intended source ref, network, zone, and environment with
need_cached_disk=false. For the stage defaults above:
gh workflow run zfnd-deploy-nodes-gcp.yml \
--repo ZcashFoundation/zebra --ref main \
-f network=Mainnet -f zone="$ZONE" -f environment=dev \
-f need_cached_disk=false
For next, use --ref next -f environment=next. For production, use the
approved release tag with -f environment=prod. The absent MIG is recreated
at size one, its stateful disk policy is restored, and the workflow attempts
to bind its reserved address. Verify health and the address before
recovering another zone.
Bootstrap a disk without a cache image
Deployment cache policy permits missing images, not missing disks. Production disks must exist before a release-triggered deployment.
To seed a missing disk manually, select a verified image for the same
network and a compatible database version. Set IMAGE_PROJECT to the
project containing it and IMAGE to its exact name:
IMAGE_PROJECT=zfnd-dev-zebra
IMAGE=verified-cache-image
gcloud compute images describe "$IMAGE" --project "$IMAGE_PROJECT" \
--format='yaml(name,status,diskSizeGb,labels,description)'
Require READY and sufficient disk capacity, then create the disk:
gcloud compute disks create "$DISK" --zone "$ZONE" --project "$P" \
--image="$IMAGE" --image-project="$IMAGE_PROJECT" \
--size=400GB --type=pd-balanced
Increase the size if the source image requires more than 400 GB. A snapshot
can be used instead with --source-snapshot. Dispatch with
need_cached_disk=false after creating the disk.
Check state compatibility and rollback requirements before selecting an older image; it does not necessarily avoid a full resync.
DB-format-version-break release
A database major-version change does not by itself require a fresh disk. Use the upgrade and rollback requirements, not the version number, to plan recovery.
Before a format-changing rollout:
- Check source and destination state compatibility.
- Take independent, consistent snapshots before the new binary writes to each disk. Stop the writer and any direct database readers while taking the rollback copy. Use the preservation procedure above when removing a MIG to detach its disk.
- Deploy one network/zone manually, then verify its application health and logs before proceeding. Publishing a release starts the automatic six-cell deployment; it does not provide this one-zone ordering.
- Retain the snapshots until rollback to the previous binary is no longer required.
For rollback, use disk recovery with a pre-upgrade snapshot and dispatch the previous approved tag to the same production destination. Do not start the old binary against a disk whose format it cannot read. Preserve the failed upgraded disk for diagnosis before replacing it. Verify each restored node before moving to another zone.
Regional-to-zonal migration
A regional MIG cannot be renamed into three zonal MIGs. For an existing regional deployment, schedule the cutover explicitly; the workflow does not migrate it automatically.
- Inventory the regional MIG, its three instances, zonal disks, and reserved addresses. Record the current template and exact source ref.
- Stop writers and preserve every state disk. Set each effective stateful
disk policy to
neverbefore deleting the regional MIG; use--regionfor the old group, not the zonal commands above. - Confirm all disks are detached. Snapshot each zone’s disk and require
READYfor all snapshots before replacing or deleting old resources. - Create a disk in each destination zone from the corresponding snapshot, using the current naming scheme.
- Dispatch the selected source ref once per network and zone with
need_cached_disk=false. Verify each new MIG’s health and reserved IP. - Remove obsolete disks and templates only after the new deployment is operational and the rollback copies are retained.
Repeat for both networks. Do not run an unattended loop that deletes all zones before verifying the first replacement.
Reference
Static IPs
| Zone | Stage and production address name | Next address name |
|---|---|---|
us-east1-b | zebra-${network} | zebra-next-${network} |
us-east1-c | zebra-${network}-secondary | zebra-next-${network}-secondary |
us-east1-d | zebra-${network}-tertiary | zebra-next-${network}-tertiary |
Reserve addresses before deployment. Stage and production reuse names in different projects; next has separate names in the development project.
The workflow resolves the address for stable destinations and advertises
it through ZEBRA_NETWORK__EXTERNAL_ADDR, but assigns it only during fresh
MIG creation. A missing reservation produces a warning and skips binding.
An existing MIG does not get its binding repaired by rerunning deployment.
Inspect the instance’s actual networkInterfaces and its per-instance
configuration before repairing it. For a missing per-instance
configuration, use the same operation as the workflow:
IP_NAME=zebra-mainnet
IP_ADDRESS=$(gcloud compute addresses describe "$IP_NAME" \
--region=us-east1 --project "$P" --format='value(address)')
gcloud compute instance-groups managed instance-configs create "$MIG" \
--instance="$INSTANCE" --zone="$ZONE" --project "$P" \
--stateful-external-ip="address=${IP_ADDRESS},interface-name=nic0,auto-delete=never"
Require a nonempty address first. Select the correct name for the environment and zone. If the configuration already exists, update it rather than creating another. The command applies the configuration to the instance by default. Verify that its actual external address matches the advertised address after the update; successful submission alone does not establish that the binding is live.
Cache images
The integration workflow’s create-state-image job produces names of the
form:
{prefix}-{ref-slug}-{sha}-v{major}-{network}-{tip|checkpoint}[-u]-{HHMMSS}
The prefix is zebrad-cache or lwd-cache. The ref slug is truncated to 12
characters, the SHA to seven, and the time is UTC. The database major comes
from the running test’s logs, not merely from the source constant.
The lookup script
requires a READY image matching purpose, database major, network, and
state kind. It tries the current branch’s truncated ref slug followed by a
commit SHA, then main, then any branch, selecting the newest match in each
query. Branch slugs can collide after truncation; selection does not require
the current commit. Inspect the chosen image in the job output.
The allow_missing input permits an empty selected name, not a failed
image-list query. List recent v29 Mainnet tip images with:
gcloud compute images list --project zfnd-dev-zebra \
--filter='status=READY AND name~^zebrad-cache-.*-v29-mainnet-tip' \
--sort-by='~creationTimestamp' --limit=5
Sentry releases
Successful stage, next, and prod deployments run sentry-release-name.
It reads the node’s release name from the exact image with zebrad --version,
using an environment, id-token: write, and GCP_ARTIFACTS_SA to pull from GAR.
The separate sentry-release job has no GitHub environment and only
deployments: read permission, keeping GCP credentials out of the Sentry
action’s container. Using optional repository secret SENTRY_AUTH_TOKEN, it
records the release and deployment environment in organization zfnd,
project zebrad.
The commit range ends at the deployed source SHA. Among the latest 50 GitHub
deployments in the same environment, the job finds a different SHA whose
latest status is success and passes it as previous_commit. Without a
match, it supplies no previous commit; it never uses another environment’s
newest Sentry release.
Without SENTRY_AUTH_TOKEN, the recording job skips its steps; the image
version reader still runs. Deployments selecting dev do not enter this
recording path, even when their ref targets stage or next resource names.
Zebra reads CI and Git metadata
from the container’s runtime environment. Pass these values at deployment;
do not bake environment-specific metadata into shared images. Sentry events
carry tags and a ci context; Logs carry the same tags as attributes and
prefix the context fields with ci..
| Runtime variable | Sentry metadata |
|---|---|
GITHUB_ACTIONS | Sets ci.provider=github-actions when equal to true, ignoring case. |
GITHUB_EVENT_NAME | deploy.trigger tag. |
CI_PR_NUMBER, CI_TEST_ID | pr.number and test.id tags. |
GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT | run_id and run_attempt in the CI context. |
GITHUB_WORKFLOW, GITHUB_JOB | workflow and job in the CI context. |
GITHUB_SHA | git.sha tag; otherwise uses the build-time SHORT_SHA, then VERGEN_GIT_SHA, when available. |
GITHUB_REF_POINT_SLUG_URL, GITHUB_HEAD_REF_SLUG_URL, GITHUB_REF_NAME_SLUG_URL | First nonempty value becomes the git.ref tag, in this order. Raw GITHUB_REF_NAME is not used. |
Runtime values are trimmed; empty values are ignored. The
node deployment
passes its source SHA as GITHUB_SHA and its selected environment as
SENTRY_ENVIRONMENT. The
integration-test deployment
uses SENTRY_ENVIRONMENT=ci for every trigger, keeping test runs out of the
fleet’s release-health environment.
Daily cleanup
The cleanup workflow
runs daily at 07:00 UTC, or manually, with the dev environment. Its
scripts use age and name filters, not keep_until or delete_protection:
| Resource | Selection and retention |
|---|---|
| Instances | Names ending in -[0-9a-f]{7,}; three-day age cutoff, or six days with long-test=true |
| Instance templates | The same name suffix; two-day age cutoff |
| Disks | Unattached, older than the two-day cutoff, and either the suffix above or a zebrad- prefix excluding zebrad-cache |
| Cache images | Preserve all newer than the two-day cutoff and the newest two older images per network/state-kind category |
Cutoffs are dates, not exact elapsed-hour deadlines. Decimal GitHub run IDs
match the hexadecimal-looking suffix filter. Current stable cache disk
names end in a zone letter and begin with zebrad-cache-, so neither disk
selector matches them, even when detached. Current CD template names also
end in a zone letter and are outside the template selector. Do not assume
the cleanup workflow reaps branch MIGs or their templates.
Image retention is not partitioned by branch or database major. Preserve required rollback snapshots independently of CI cache-image retention.
The next branch
The next branch deploys changes before merging them into main, under
the deployment triggers and environments.
Workflow
- Open a pull request against
next. Use the same conventional commit title and change fragments required formain, because merge commits preserve the branch’s commits. - Merge after the required checks pass. The
next branch ruleset
requires
lint,unit-tests,test-crates,pr-gate-result, andmerge-policy. It requires a PR and merge commits, but no approving review or merge queue. It does not requiremergefreeze. - Keep
nextcurrent through the automaticmain-to-nextsync PR, which merges itself with a merge commit once its checks pass. To hold it, add thedo-not-mergelabel. Resolve conflicts on its branch as below. - To promote the branch, open a PR from
nexttomain. It followsmain’s review and merge-queue requirements. Review the entire branch difference, not only the latest change.
Dispatch GCP stateful tests
manually for next; they do not run on its PRs or gate its merges.
Synchronization from main
The sync workflow
runs on every main push and on manual dispatch from main. Its sync job
uses ubuntu-26.04 with a five-minute timeout and:
- Exits successfully if
nextalready contains every commit frommain. - Moves the
sync/main-to-nextbranch up tomain: it creates the branch, fast-forwards it, or, once a maintainer’s conflict resolution blocks the fast-forward, mergesmaininto it. It never force-pushes over that resolution. - Finds an open PR from this repository’s
sync/main-to-nextintonext, rather than accepting a fork’s branch with the same name, and createschore: sync main to nextif none exists. - Enables auto-merge with a merge commit when the PR is mergeable. A conflicting PR can’t merge itself; the run summary reports the conflict.
The job opens PRs with a GitHub App token from RELEASE_APP_ID and
RELEASE_APP_PRIVATE_KEY, requesting contents-write and pull-requests-write
permissions. Using GITHUB_TOKEN would not trigger required checks; the
job’s ordinary GitHub token has no permissions.
Resolve sync conflicts
Resolve conflicts on the sync PR’s own branch. The example assumes origin
points to the Zebra repository:
git fetch origin
git switch -C sync/main-to-next origin/sync/main-to-next
git merge origin/next
Resolve each conflict, stage the resolved files, complete the merge commit,
and push it with git push. For simple text conflicts, the PR’s
Resolve conflicts button makes the same merge in the browser. Merge the
PR with a merge commit after its checks pass. Do not force-push or reset
next to discard the changes under evaluation.
Later pushes to main are merged into the branch, never force-pushed over
your resolution. If that merge conflicts, merge origin/main into the
branch the same way.
Publish a pre-release
A pre-release from next, such as an alpha or a beta,
publishes what next runs as a GitHub pre-release with signed binaries and
a Docker Hub image tagged with its version and edge, without publishing
crates or deploying production. Release candidates are cut from main
instead.
-
Merge a
chore(release): set zebrad version to <version>PR intonextthat sets thezebradversion to a pre-release version, such asX.Y.Z-alpha.NorX.Y.Z-beta.N, and updatesCargo.lock. Do not use thercidentifier: it marks a release candidate frommain. Later sync PRs can conflict on that version: keep the higher one. -
Generate the release notes from the pending change fragments, without committing them:
changie batch v<version> --project zebrad --dry-run > release-notes.md -
Publish the pre-release on the merge commit from step 1, with your own GitHub credentials. A release created with a workflow’s
GITHUB_TOKENdoes not start the release workflows.gh release create v<version> --target <commit> --prerelease \ --title "Zebra <version>" --notes-file release-notes.md
Never promote it to a full release: that would make it the Latest release and trigger a production deployment.
Compare next with main
Compare nodes on the same network and zone, such as
zebrad-next-mainnet-b and zebrad-main-mainnet-b. Check actual deployed
commits and health, not branch tips.
Normal next deployments report Sentry environment next; main pushes
report stage. Both record Sentry deployments.
For equivalent comparisons, avoid manual dispatches
selecting dev, even when they target the same resource names.
Use the deployment runbook for per-zone inspection and deployment. Follow its disk compatibility procedure before testing a state-format change.
Read the next book
The book from next is published at zebra.zfnd.org/next,
beside the latest release’s book at zebra.zfnd.org
and main’s at zebra.zfnd.org/main. Compare
the same page in each, for example /main/dev/next-branch.html and
/next/dev/next-branch.html.
/next updates after each push to next that changes the book. If the
next book doesn’t change for 90 days, /next is removed until its next
change.
Zebra versioning and releases
Zebra releases publish Rust crates, a zebrad GitHub Release, and downstream
artifacts. Separate planning, changelog, and publication steps let failed
publications resume without replacing immutable artifacts.
Zebra versioning
Zebra uses Semantic Versioning, with
independent versions for zebrad and each library crate. Judge compatibility
against each package’s audience.
| Release level | zebrad | Library crates |
|---|---|---|
| Major | Mainnet network-upgrade support or breaking operator interfaces, including configuration, CLI, RPC, and downgrade-incompatible state changes | Incompatible public APIs, semantics, features, or public dependency types |
| Minor | Backward-compatible features and behavior changes | Backward-compatible API additions |
| Patch | Compatible fixes without a new feature | Compatible fixes |
Follow Cargo’s SemVer rules, including pre-1.0 versions, and the
changelog guidelines. A library major bump does not
itself require a zebrad major bump.
The release-plz configuration selects releases from conventional
commits and dependency changes. Commit types feat, fix, perf, refactor,
and build trigger release consideration; dependency cascades can release other
packages. Release-plz disables changelog generation and SemVer checking;
maintainers must audit proposed versions and source-authored changelogs.
Supported releases
Plan releases within the Mainnet support window.
A release must support its network’s active upgrades regardless of its support cutoff.
Supported update paths
Follow the upgrade and rollback requirements.
Preview releases
Zebra publishes three kinds of pre-release:
| Pre-release type | Source | Details |
|---|---|---|
Alpha (X.Y.Z-alpha.N) | next branch | An early preview of work on next. |
Beta (X.Y.Z-beta.N) | next branch | A release under active development and testing. |
Release candidate (X.Y.Z-rc.N) | main, through the Release PR | A release selected for final testing. |
Any other pre-release identifier also marks a release from next. Every
pre-release has signed Linux binaries attached to its GitHub Release and a
Docker Hub image tagged with its exact version; pre-releases from next also
move the edge image tag. Pre-releases from next do not publish crates.
For a release candidate, package versions, tags, and versioned changelogs must agree. Use the normal Release PR and readiness process:
- Agree on the candidate’s scope and prerelease version. Check that the final
Release PR manifests, dependency requirements, and versioned changelogs use
that version; there is no separate
rcdispatch operation inrelease.yml. - Review the readiness report’s exact source commit and package plan, then approve publication through the normal release workflow.
- Test the published candidate and record the results. Publish fixes with new versions; never replace the candidate’s tag or crate archive.
- Publish the stable version through a new reviewed version plan after testing. Do not turn an RC-version GitHub prerelease into a stable release by changing its channel. The controller derives the channel from the package’s SemVer suffix and rejects a conflicting existing channel on recovery.
Publishing a GitHub prerelease does not deploy it to production. The
binary workflow handles release: published, so every
v-prefixed release and prerelease gets binaries and a Docker Hub image. The
GCP deployment workflow handles release: released for
v-prefixed tags: stable publication or promotion of a prerelease.
Prereleases, including release candidates, skip production deployment.
Manual dispatch can still target production independently of the release
channel.
Distribution tags
Git tags use v<version> for zebrad and <crate>-v<version> for libraries.
For container tags, see Docker images.
Feature flags
Keep experimental features behind Cargo features so main
remains releasable. Agree on the treatment of breaking changes before merging;
a feature gate does not replace versioning or migration documentation. Security
fixes can require an immediate incompatible change.
Release frequency
Publication is maintainer-controlled, not scheduled by a timer. Non-release
pushes to main refresh the Release PR; review and merge authorize publication.
Plan upgrades and release preparation before the current Mainnet support window
expires. Network-upgrade deadlines and urgent security fixes determine release
timing independently of routine feature and fix releases.
Deprecation practices
Announce deprecated operator interfaces in the changelog with replacements and removal timelines. Retain them until the next major release, accepting only critical and security fixes. If a critical security or reliability problem requires earlier removal, explain why at the top of the release notes and provide migration instructions.
Library APIs follow their crate’s SemVer compatibility policy.
Release candidate and release process
The release workflow maintains a release-plz- branch labeled
release. After review and merge, ZcashFoundation/cargo-release publishes it.
This action is not the local cargo release CLI configured by release.toml.
Prepare and review the Release PR
Track the release with the release issue template. Complete
preparation on main before the final Release PR refresh:
- Confirm a successful full sync since the last state change and the required integration evidence.
- Review and merge the checkpoint update.
- Set
ESTIMATED_RELEASE_HEIGHTfor the intended release. Check the support interval against upcoming upgrades: a release lacking an upgrade must expire before that upgrade; restore the intended interval in its supporting release. - For network upgrades, check activation heights, consensus branch IDs, minimum peer protocol version, and supporting dependencies.
- Review outstanding security advisories and coordinate any private fixes under SECURITY.md.
Review each package against its own previous published version. The
PR Gate / release-api-diff comment starts at the latest reachable zebrad
release tag and identifies crates with older release baselines. Review those
older ranges separately. Generated version bumps are not proof of correct SemVer.
Follow the changelog batching and correction procedure.
Approve and merge only the latest reviewed commit after required checks pass.
A new Release PR commit invalidates review of the previous version plan. Approve
the release environment deployment when its protection rules require it.
What release readiness reports
The PR Gate / Release readiness job requires current main, validates each
released package’s versioned changelog, and runs Cargo 1.91’s multi-package
publication dry run. Changelog and Cargo validation run independently and both
appear in the summary. The controller observes crates.io, Git tags, and the
GitHub Release without publishing.
Before publication, a successful report with reason: "incomplete" is expected:
the desired plan is valid, but the external artifacts do not all exist yet.
Review the plan and observed state, not only the job’s green status.
Before first publication, have a crate owner confirm Trusted Publishing bindings.
What happens after merge
The controller fixes the source to the Release PR’s merge commit and the base to
its first parent. It publishes missing crates in dependency order and verifies
source provenance. When zebrad is in the plan, the workflow installs that exact
published version with cargo install --locked --version, runs it with
--version, and only then creates missing tags and the GitHub Release.
Release notes come from the generated changelog. For downstream jobs, see continuous delivery; for artifacts and signatures, see binary verification.
Confirm publication, binary distribution, and production health before
announcing success. Then run the
Book workflow
manually, so the site root documents the new release without waiting for the
next push to main; if BOOK_ROOT_REF is set, delete it first.
If release readiness fails
Read the failed job summary before retrying.
| Failure | Action |
|---|---|
Release PR is behind main | Let release-plz refresh it, then review the new head. |
| Missing or empty versioned changelog | Add missing source fragments on main for direct changes. A dependency-only omission requires a batching fix, not a generated-file patch. |
| Cargo dry run fails | Fix the source or dependency problem on main. |
| Conflicting crate provenance, tag target, or release channel | Stop publication and investigate; do not overwrite immutable external state. |
If readiness does not start, rerun the PR checks. For manual fallback, set
PR_NUMBER to the open Release PR:
gh workflow run release.yml --repo ZcashFoundation/zebra --ref main \
-f operation=check -f release_pr_number="${PR_NUMBER}"
If publication stops after merge
Read the failure summary. Set PR_NUMBER to the same merged Release PR:
gh workflow run release.yml --repo ZcashFoundation/zebra --ref main \
-f operation=resume -f release_pr_number="${PR_NUMBER}"
The resume operation keeps the immutable source, skips matching crates and
tags, and fills missing external state. It repairs configured mutable metadata
(name, notes, draft state), but rejects conflicting crate sources, tag targets,
or prerelease channels.
If automation remains unavailable, obtain maintainer authorization to use the legacy checklist. Never run manual publication concurrently with the controller.
Withdraw a bad release
Obtain maintainer approval and select versions for withdrawal only for consensus failures, data corruption, security vulnerabilities, or sync regressions that stall nodes. Fix other defects through a regular release.
Set BAD to the withdrawn version and GOOD to the proposed rollback version,
both without v. The rollback target must lack the defect, remain supported on
Mainnet, and support upgrades active or activating before the fix ships on each
affected network. Compare its zebra-chain/src/parameters/constants.rs with the
network’s activation heights. If no safe rollback exists, ship the fix without
redirecting Latest, Docker latest, or production to an obsolete release.
Steps 1–4 can run in the order that limits harm fastest. Start step 5 when the fix is ready; step 6 waits for fixed crate versions to be published.
-
Warn on the GitHub Release and remove installable assets. Prepend a warning naming affected versions, the defect, and any replacement; retain the original notes:
gh release view "v${BAD}" --repo ZcashFoundation/zebra \ --json body --jq .body > notes.md # Edit notes.md to prepend the withdrawal warning. gh release edit "v${BAD}" --repo ZcashFoundation/zebra \ --notes-file notes.mdIf the withdrawn release is Latest and a safe rollback exists, select it:
gh release edit "v${GOOD}" --repo ZcashFoundation/zebra --latestDownload assets needed for analysis before deleting them:
gh release view "v${BAD}" --repo ZcashFoundation/zebra \ --json assets --jq '.assets[].name' | while IFS= read -r asset; do gh release delete-asset "v${BAD}" "${asset}" \ --repo ZcashFoundation/zebra --yes doneKeep the release and Git tag as the record of what shipped. Do not change its prerelease/stable channel or convert it to a draft: channel changes conflict with recovery, and promotion triggers a production deployment. Do not run
resumefor the withdrawn PR; it can restore original notes and release metadata. -
Remove the Docker installation path. Compare the current
latestand withdrawn image digests before changinglatest:docker buildx imagetools inspect --format '{{.Manifest.Digest}}' \ zfnd/zebra:latest docker buildx imagetools inspect --format '{{.Manifest.Digest}}' \ "zfnd/zebra:${BAD}"Only when those digests match and a safe rollback exists, copy the good release’s image index:
docker buildx imagetools create -t zfnd/zebra:latest "zfnd/zebra:${GOOD}" docker buildx imagetools inspect --format '{{.Manifest.Digest}}' \ zfnd/zebra:latest docker buildx imagetools inspect --format '{{.Manifest.Digest}}' \ "zfnd/zebra:${GOOD}"Require matching digests. Copying the image index preserves its digest-bound signatures and attestations. Delete the withdrawn Docker Hub version tag and any aliases with the withdrawn digest, including SHA tags from older releases. Pulls by digest can still succeed, and downloaded copies remain usable.
-
Roll production back only where safe. Check disk compatibility and inspect the good tag’s deployment workflow: the tag supplies both code and workflow. For compatible state, select one supported network and zone:
NETWORK=Mainnet ZONE=us-east1-b gh workflow run zfnd-deploy-nodes-gcp.yml \ --repo ZcashFoundation/zebra --ref "v${GOOD}" \ -f network="${NETWORK}" -f zone="${ZONE}" \ -f environment=prod -f need_cached_disk=falseManual dispatch waits for rollout, not node-health verification. Check the selected managed instance group (MIG) using deployment diagnostics, with
P=zfnd-prod-zebraand its production name, such asMIG=zebrad-mainnet-b. Verify health before repeating for another zone or network; dispatching all six jobs together does not serialize rollouts.If the old binary cannot use the disk format, follow the database-format rollback procedure. Confirm a pre-upgrade snapshot before destructive disk operations; snapshots named
-pre-major-<timestamp>exist only if preparation created them. Choose the procedure by migration and rollback compatibility, not major version. -
Notify operators. Keep the GitHub Release warning as the primary notice. For a vulnerability in a tagged release, publish a GitHub Security Advisory naming affected and fixed versions under the disclosure policy. Coordinate disclosure timing rather than exposing exploit details prematurely.
-
Publish the fix. Use the normal release process and a
changie newfragment naming the withdrawn versions and the fix. Never reuse a published version number. -
Yank defective crate versions after publishing the fix. Always yank the affected
zebradversion; yank libraries only if their code carries the defect. Use the publication plan’s exact crate versions and a token restricted to yanking; follow the credential procedure:cargo yank zebrad --version "${BAD}"A yank prevents new dependency resolution from selecting that version; existing
Cargo.lockfiles remain usable. It does not uninstall affected binaries or revoke existing source archives.
Zebra audits
Complete release preparation, then fix the audit baseline and scope:
- Select a release candidate; record its immutable tag and full commit hash. Publishing a GitHub prerelease does not deploy it to production.
- Declare a feature and nonessential-fix freeze for the audited scope. Defer unrelated changes until the audit ends. Gate new experimental features with Cargo features and agree which features the audit includes.
- Give auditors the exact dependency lockfile, target platforms, enabled features, and scope lists. Classify dependencies and code as in scope, partially in scope, or out of scope, with a reason for each exclusion.
- Prioritize Foundation-maintained production code, consensus-critical and security-critical paths, and code without prior audit coverage. Record prior audits’ baselines and coverage; an audited dependency version does not cover later changes.
- Agree on the channel for findings and the process for reviewing fixes. Handle vulnerabilities through the security disclosure policy, not public issues containing exploit details.
Use specialist review for cryptography or consensus-critical components beyond the primary auditors’ expertise.
Use an audit branch if auditors need a stable branch for fix review or cannot
track changes on main. Map commits to findings and obtain auditor review before
closing them.
Publish fixes through the release process, with versioned changelog entries and required advisories. The security policy treats tagged candidates as shipped releases, not unreleased code.
Changelog guidelines for Zebra
Document effects on operators and library consumers, not implementation details.
| Project | Audience | Generated file |
|---|---|---|
zebrad | Operators using configuration, CLI, RPC, metrics, or Docker | Root CHANGELOG.md |
| Library crate | Rust developers using its public API | <crate>/CHANGELOG.md |
How changelogs are written
Changie generates every active changelog from .changes/.
Never hand-edit root or crate CHANGELOG.md files: the next changie merge
replaces them. The configuration defines project keys,
output paths, and section order.
Source pull requests
Use Changie v1.25.2, pinned by the release workflow:
changie new -j zebrad -k Changed
changie new -j zebra-chain -j zebra-state -k breaking
The commands prompt for a body; repeated -j flags reuse it for each project.
Use separate commands when wording differs by audience, or -b 'entry text'
for noninteractive input. Single shell quotes preserve Markdown backticks;
use the prompt for bodies containing apostrophes.
The body contains the entry text without a leading bullet or heading. Include a
link to the actual PR in the form ([#number](PR-URL)); do not invent a PR number.
Commit the generated .changes/unreleased/*.yaml files with the change. Ordinary
source PRs do not batch releases or regenerate changelogs.
Use these configured project keys:
zebradzebra-chain,zebra-consensus,zebra-network,zebra-node-serviceszebra-rpc,zebra-script,zebra-state,zebra-test,zebra-utilstower-batch-control,tower-fallback
Use exact kind keys from the section-order table, not display headings.
The PR gate dry-runs every project’s pending fragments on PRs, merge
groups, and main. Malformed fragments fail even when introduced by another PR.
Prefer changie new over handwritten YAML. In YAML single-quoted bodies, double
apostrophes; quote text containing colon-space or use a block scalar.
Maintainers can comment /changelog on a PR for a
draft suggestion. It requires the repository AI integration,
does not write to the branch, and reads at most 20,000 diff bytes. Review the full
PR for accuracy and coverage before accepting it.
Release batching and corrections
The release workflow batches each project into the release
planner’s exact version, then merges version files into generated changelogs.
Never substitute changie batch auto for the approved plan. For an approved
manual batch, set PROJECT to the project key and VERSION without a v prefix:
changie batch "v${VERSION}" --project "${PROJECT}"
changie merge
The result is .changes/<project>/v<version>.md. Correct an already-batched
entry, including a historical typo, in that version file and run changie merge.
Do not recreate consumed fragments as entries for the next release. Commit the
source and generated output together.
The batching script restores .changes/ from the Release PR’s
base revision, overwriting branch-local curation. Land missing fragments on
main before refreshing; preserve and reapply version-file corrections after
the final refresh. Never rerun over unpreserved corrections.
The script adds a generic dependency entry only for released projects without
fragments. Audit API and dependency changes, including dependency-driven version
bumps, against each package’s previous published release, not the last zebrad
release or the generated entries.
The command changie latest -j zebra-state prints a project-prefixed version,
such as zebra-state-v13.0.0. For zebrad, remove the zebrad- prefix to obtain
its Git tag; its tags have the form v<version>.
CHANGELOG.md vs GitHub Release
A GitHub Release adds upgrade guidance to the changelog. The controller uses the
matching zebrad section of the generated root changelog as
release notes. Put summaries, upgrade instructions, and
compatibility information in .changes/zebrad/v<version>.md, then regenerate
before publication. Recovery can overwrite GitHub-only edits.
Part 1: Common guidelines
One entry per distinct change
Write one bullet per distinct change per affected project, keeping migration instructions with it even across multiple lines. Never combine different PRs.
Describe the net change from the previous release. If an unreleased addition is renamed, update its existing fragment. If it is removed before release, remove the fragment rather than documenting an addition and removal nobody received. Exclude defects introduced and fixed entirely between releases.
Documentation, verification-only changes, and internal refactors without a consumer-visible effect do not need entries. An entry whose only point is that existing deployments need no changes does not belong in the operator changelog.
Section order and priority
The configured display order differs from the priority used to classify a change:
| Display order | CLI kind | Use for |
|---|---|---|
| Breaking Changes | breaking | Incompatible interfaces or behavior, with migration guidance |
| Added | Added | New capabilities |
| Changed | Changed | Observable behavior, output, logs, or error changes |
| Deprecated | Deprecated | Features scheduled for removal, with replacement and timeline |
| Removed | Removed | Removed capabilities |
| Fixed | Fixed | Fixes invisible to users: the defect is gone but output is unchanged |
| Security | Security | Security fixes, coordinated with the disclosure process |
Choose the most impactful applicable category: breaking, Security, Removed,
Changed, Deprecated, Added, then Fixed. Do not repeat one change in several
sections. A breaking addition gets one breaking fragment that explains the
impact and names the new capability. Examples include a variant added to an
exhaustive public enum or a field added to a public struct constructed with
literals. Distinct changes in the same PR still need separate entries.
Only sections with entries are rendered. Do not add empty headings. Use the version heading and date generated by Changie.
Tone and language
Write factual, concise entries naming affected configuration keys, RPC methods, types, or functions and what consumers observe or must change. Avoid implementation narration, hyperbole, marketing, intensifiers, hedging, and em dashes. Match the project’s style.
Lead release summaries with the most consequential change, especially a break. Name operational requirements for operators and API migration work for libraries.
Gate coverage is not editorial policy
The source changelog gate checks PR types feat, fix, perf, refactor, and
build, plus titles declaring a break. It expects a fragment for every directly
changed publishable package, excluding documentation and verification files.
A root manifest change also triggers the zebrad requirement. A lockfile-only
change triggers it when the node’s production dependency graph changes,
including supported non-default features and target-specific dependencies, but
excluding development dependencies and test-only features.
Do not invent operator-facing changes to satisfy this broader file-based check;
explain justified omissions to reviewers. Follow the
breaking-change declaration rules.
A breaking fragment requires ! in the PR title; a declared package break
requires a breaking fragment. Unlike advisory API and changelog coverage
checks, malformed-fragment batching failures block the aggregate.
Part 2: zebrad changelog (node operators)
What is breaking for node operators?
An operator-facing change is breaking when an existing deployment or client can fail after upgrading. Explain the affected setup and the replacement.
| Interface | Breaking changes |
|---|---|
| Configuration | Removed, renamed, or retyped fields; incompatible defaults |
| Environment | Removed or renamed variables used by scripts or services |
| CLI | Removed or renamed arguments, incompatible parsing |
| RPC | Removed methods; incompatible parameters, responses, or semantics |
| State | A database-format change that prevents downgrade |
| Behavior | New defaults or policy that invalidate existing integrations |
| Build requirements | An increased minimum supported Rust version for source builds |
An optional configuration field with a compatible default, a new RPC method or response field, new metrics, and a behavior-preserving optimization are not inherently breaking. Describe actual compatibility constraints; do not assert that every client tolerates an added response field.
Rust API changes belong in library changelogs unless they also affect operators. Omit CI changes and ordinary dependency-version lists. Describe dependency changes only for security fixes or concrete operational effects.
Network upgrade releases and ZIP references
For an upgrade-support release, record:
- The upgrade name and network.
- Activation height and approximate activation date, clearly labeled as an estimate rather than a time-based activation condition.
- The deadline for upgrading and the consequence of missing it.
- Each implemented ZIP, linked using
https://zips.z.cash/zip-NNNN. - Testnet activation information and the first supporting release when relevant.
Link consensus or protocol changes to the corresponding ZIP or protocol specification. Keep activation heights consistent with the release’s network parameters. Put operational context above the normal category sections; a release-specific network-upgrade section is not a new Changie kind.
Deprecations
Follow the deprecation policy. For staged transitions, name the implemented stage: enabled by default with opt-out, disabled by default with opt-in, or removed. Claim controls only when they exist.
Security releases
State the affected versions, severity, upgrade urgency, and fixed version. Keep public technical detail limited until coordinated disclosure permits it. Link the advisory when published and credit reporters according to SECURITY.md. An unpublished fix does not justify disclosing an exploit in an ordinary PR fragment.
Part 3: Crate changelogs (library consumers)
Describe affected public APIs and semantics, including fixes without signature changes, and the migration for Rust callers.
What is breaking for library consumers?
Apply Cargo’s SemVer guidance. Breaking changes include removed or renamed public items, incompatible signatures or return types, new trait bounds, changed error types, reduced visibility, removed features, newly required feature gates, and increased minimum supported Rust versions. Additions can also break callers: exhaustive enum variants, public struct fields used in literals, and required methods on traits implemented downstream.
Record observable semantics, including validation, ordering, privacy, security, and resource use.
New independent public types or functions and behavior-preserving optimizations
are not inherently breaking. Check trait implementations and method additions
for coherence or name-resolution conflicts. Adding an Option<T> parameter
still changes the signature; Rust has no default arguments.
Dependency updates
A SemVer-incompatible dependency update breaks an API that exposes its types,
even if Zebra’s signatures are textually unchanged. This includes 0.x to
0.(x+1) updates. Use breaking, name the dependency and new version, identify
the exposed API, and explain the callers’ required lockstep upgrade.
A compatible update needs a Changed entry for consumer-visible behavior.
Internal-only and development dependency changes need no fragment without such
an effect. Release-time dependency cascades can require a versioned entry for a
newly published package; audit the generated entry rather than treating it as
API evidence.
Part 4: GitHub Release format
Use the release’s generated changelog plus the operational information below. Do not copy compatibility values or upgrade instructions from a previous release without checking the actual source and artifacts.
Update priority
Identify who must upgrade and why. Use a table when priorities differ by audience.
| Priority | Meaning |
|---|---|
| Critical | Immediate security response or a mandatory network-upgrade deadline |
| High | Breaking changes or consensus updates requiring prompt action |
| Medium | New features or notable fixes |
| Low | Minor improvements without an immediate operational requirement |
Include the network-upgrade deadline or security-release details specified above.
How to upgrade
Give release-specific binary, source-build, and Docker instructions; link the upgrade procedure. Identify configuration migrations, required backups, database upgrades, and downgrade restrictions before restart. Do not invent service names or shutdown log messages.
Compatibility
Record supported binary platforms and system-library requirements, active or
scheduled network upgrades, relevant RPC differences, and minimum Rust versions
for source builds. Read Rust requirements from that release’s package manifests;
libraries and zebrad need not have the same requirement. Do not claim blanket
compatibility with every RPC in a zcashd version.
Release structure
Use this order, omitting sections that do not apply:
- Two-to-four-sentence summary and upgrade urgency.
- Update-priority table when different audiences need different actions.
- Network-upgrade details or security context, if applicable.
- Changelog categories in the configured order.
- Upgrade instructions and compatibility requirements.
- Contributor and reporter credit where appropriate.
Do not include raw commit logs, contributor-only implementation details, or repeated documentation. Link detailed procedures instead.
Labels and issue types
Issue types classify work; labels route review, hold merging, or request automation. Labels below are current as of October 24, 2026. Inspect the live set before changing workflow references:
gh label list --repo ZcashFoundation/zebra --limit 300
Issue types replace type labels
Choose an issue type when filing an issue:
| Type | Use for |
|---|---|
| Bug | Incorrect behavior in Zebra, CI, documentation, or tooling |
| Task | Work without a new user-visible feature: maintenance, infrastructure, research |
| Feature | New user-visible functionality |
| Epic | A high-level goal broken into smaller issues |
Issue types apply to issues, not PRs. A PR declares its kind in its conventional-commit title.
Labels people apply
| Label | Purpose and rule |
|---|---|
security | Security-relevant work of any severity. Report vulnerabilities privately through SECURITY.md, not through a public issue. |
consensus | Validation, cryptography, or script changes requiring consensus-aware review. |
devops | Build, CI, test infrastructure, or release work. The devops issue template applies it automatically. |
urgent | Needs attention today. It does not bypass checks or trigger special automation. |
blocked | Waiting on an external dependency. Add a comment naming the blocker and a date to reassess it. |
do-not-merge | Holds a PR even when checks pass. Resolve the hold with the person who applied it. |
needs-issue | A PR lacks the issue discussion required by the contribution policy. Request or establish that discussion before proceeding. |
help wanted | Maintainers invite outside help; prior discussion still applies. |
good first issue | Suitable for a first contribution; keep this GitHub-native spelling. |
nu-7 | NU7 network-upgrade work. Remove the temporary label after the upgrade ships. |
ai-generated | Existing AI-assisted backlog cohort. Do not apply it to new issues; remove it when the cohort is resolved. It is not a substitute for PR AI disclosure. |
The template named “Private Security Issue” creates a public placeholder for maintainers tracking work discussed privately. It is not a private vulnerability reporting channel. Never put exploit details in it.
Labels wired into automation
| Label | Consumer or writer | Effect |
|---|---|---|
do-not-merge | merge-policy.yml | Fails the source PR’s merge-policy check and checks every PR named in a merge group. |
external-contribution | label-external.yml | Labels new issues and PRs by outside authors. Bots, owners, members, collaborators, and authors with admin, maintain, or write permission are excluded. |
release | release-plz.toml, release.yml, pr-gate.yml, tests-unit.yml, checkpoint-update.yml, release templates | Identifies release work. Release readiness and publication also require a branch beginning with release-plz-; the label alone is insufficient. |
run-stateful-tests | zfnd-ci-integration-tests-gcp.yml | Enables stateful integration runs for same-repository PRs. Fork PRs do not gain access to the privileged jobs by adding this label. |
run-benchmarks | benchmarks.yml | Runs the PR benchmark comparison against its base while the label is present, including on subsequent PR updates. |
Labels do not override workflow event filters and permissions.
Labels written by CI
CI appends failures to the open issue with the matching label, or opens one. Do not apply or repurpose these labels manually; close resolved failure issues.
| Label | Workflow |
|---|---|
ci-fail/advisory | advisory.yml: scheduled advisory checks |
ci-fail/binaries | release-binaries.yml: release artifact failures |
ci-fail/main | zfnd-ci-integration-tests-gcp.yml: selected Mainnet and lightwalletd integration failures outside PRs; Testnet-only jobs are excluded |
ci-fail/release | zfnd-deploy-nodes-gcp.yml and zfnd-deploy-prebuilt-nodes-gcp.yml: image-build and node-deployment failures |
Other repository labels
| Label | Repository description |
|---|---|
dependencies | PRs that update a dependency file |
github_actions | PRs that update GitHub Actions code |
rust | PRs that update Rust code |
wayfinder:map | Wayfinder map of decision tickets |
wayfinder:research | Wayfinder ticket resolved by research |
wayfinder:grilling | Wayfinder ticket resolved by a decision conversation |
wayfinder:task | Wayfinder ticket that unblocks a decision |
The Dependabot configuration sets labels: [] for both ecosystems;
it does not apply dependencies, github_actions, or rust. Wayfinder labels
track decisions, not Zebra merge or release gates.
Adding or removing a label
Propose the decision a new label enables in a Team Decisions discussion. Use kebab-case for new multi-word labels, except GitHub-native community labels. Preserve existing names that integrations consume, including slash, underscore, and colon forms.
Before renaming or deleting a label, search .github/ and release configuration
and inspect saved project-view filters. Update all consumers in the same change:
missing labels can silently disable jobs, and saved filters need updating.
Zebra crates
The workspace publishes 12 crates:
zebradzebra-chain,zebra-consensus,zebra-network,zebra-node-serviceszebra-rpc,zebra-script,zebra-state,zebra-test,zebra-utilstower-batch-control,tower-fallback
Each of these crates has the following crates.io owners, verified on October 24, 2026:
| Owner | Kind |
|---|---|
conradoplg | Individual |
oxarbitrage | Individual |
github:zcashfoundation:owners | GitHub team |
The owners team listing includes other Foundation projects. Inspect the registry’s live list before changing ownership:
cargo owner --list zebrad
Logging in to crates.io
Normal releases use Trusted Publishing: the release workflow obtains short-lived crates.io credentials through GitHub Actions OpenID Connect and publishes from the approved source.
A crate owner must configure each crate’s Trusted Publisher with repository
ZcashFoundation/zebra, workflow release.yml, and environment release.
Repository CI cannot inspect these owner-only bindings. Confirm them before the
first automated publication of a crate.
Manual ownership changes, authorized yanks, and exceptional publication require a crates.io token. Create it in account settings, restrict it to the required crates and operation, and set the shortest practical expiry.
Authenticate interactively rather than putting the token in shell history:
cargo login
Revoke the token in account settings after the operation. Do not put registry tokens in committed configuration, command output, or issue reports.
Publishing new crates
Agree on the crate’s purpose and name before publication, which a Git revert cannot undo:
- Add its manifest and workspace membership through review. Confirm its package name, license, repository metadata, dependencies, and publication settings.
- Add its project mapping and initial changelog source to
.changie.yamland.changes/before generation. A project with no version history is rendered as only the shared header; do not add an existing crate without migrating its changelog history. - Review the version and changelog in the release process. Arrange an authorized API-token first publication: crates.io requires the crate to exist before Trusted Publishing can be configured. Use the normal controller for subsequent releases.
- Add the required owners, confirm acceptance and the live list, and configure Trusted Publishing before automated publication.
Never run a second publisher alongside the release controller.
Changing crate ownership
Both individual and team owners can publish and yank versions. Only individual owners can change the owner set; see Cargo’s ownership reference. Keep at least two individual owners and the Foundation team on each crate.
Invite replacements before removing a departing maintainer. Individual invitees
must accept on crates.io before they appear as owners. Set CRATE,
NEW_OWNER, and OLD_OWNER to the approved crate and GitHub logins:
cargo owner --list "${CRATE}"
cargo owner --add "${NEW_OWNER}" "${CRATE}"
cargo owner --add github:zcashfoundation:owners "${CRATE}"
After the replacement accepts, inspect the list and remove only the intended former owner:
cargo owner --list "${CRATE}"
cargo owner --remove "${OLD_OWNER}" "${CRATE}"
cargo owner --list "${CRATE}"
These commands require no checkout, even for crates outside the workspace.
Zebra RFCs
Zebra requests for comments (RFCs) are historical designs, not current API documentation.
New decisions use architectural decision records (ADRs) and the decision log instructions.
Each RFC starts with its Zebra 7.0.0 status, code location, and divergences. Implemented means the core mechanism exists; Partially implemented means proposed work remains absent; Superseded means another design or process replaces it. Status describes the design, not its tracking issue.
Published records
| RFC | Status | Summary |
|---|---|---|
| 0000: Template | Superseded | Historical proposal template; new decisions use the ADR template. |
| 0001: Pipelinable block syncing and lookup | Implemented | Discover prospective chain tips while downloading and verifying blocks concurrently. |
| 0002: Parallel verification | Implemented | Separate structural, semantic, and contextual verification to defer dependencies. |
| 0003: Inventory tracking | Implemented | Track peer inventory to route requests for recently advertised data. |
| 0004: Asynchronous script verification | Implemented | Await transparent outputs before verifying scripts, independently of block commit order. |
| 0005: State updates | Implemented | Keep non-finalized forks in memory and finalized state in RocksDB. |
| 0006: Contextual difficulty validation | Implemented | Validate difficulty adjustments against ancestor headers. |
| 0007: Treestate | Implemented | Maintain shielded note commitment trees, anchors, and nullifiers. |
| 0009: Zebra client | Superseded | Proposed integrated wallet and scanning components; external tools now own wallet functionality. |
| 0010: Version 5 transactions | Superseded | Proposed Zebra-owned transaction representations; production transactions now wrap librustzcash types. |
| 0011: Async Rust in Zebra | Partially implemented | Concurrency constraints remain relevant; the proposed shutdown-atomic replacement is absent. |
| 0012: Value pools | Implemented | Validate transaction value balances and persist nonnegative chain value pools. |
The published sequence has no RFC 0008. Drafts are not published decisions.
Pipelinable block syncing and lookup
Status (Zebra 7.0.0): Implemented.
Code: the zebrad syncer, downloader, and block locators.
- The syncer downloads and verifies genesis before discovering tips.
- The fanout is 3;
BLOCK_DOWNLOAD_RETRY_LIMITis 3. - The sync restart delay is 67 seconds, or 2 seconds on Regtest.
- The downloader uses
Request::BlockDownload, notRequest::Blocks. - The network API also exposes
Request::BlocksByHash. - The locator reaches the 1,000-block rollback boundary, not 99 blocks.
- Feature Name: Pipelinable Block Syncing and Lookup
- Start Date: 2020-07-02
- Design PR: ZcashFoundation/zebra#583
- Zebra Issue: ZcashFoundation/zebra#504
Summary
The Bitcoin network protocol used by Zcash allows nodes to download blocks from other peers. This RFC describes how we find and download this data asynchronously.
Motivation
To sync the chain, we need to find out which blocks to download and then download them. Downloaded blocks can then be fed into the verification system and (assuming they verify correctly) into the state system. In zcashd, blocks are processed one at a time. In Zebra, however, we want to be able to pipeline block download and verification operations, using futures to explicitly specify logical dependencies between sub-tasks, which we execute concurrently and potentially out-of-order on a threadpool. This means that the procedure we use to determine which blocks to download must look somewhat different than zcashd.
Block fetching in Bitcoin
Zcash inherits its network protocol from Bitcoin. Bitcoin block fetching works roughly as follows. A node can request block information from peers using either a getblocks or getheaders message. Both of these messages contain a block locator object consisting of a sequence of block hashes. The block hashes are ordered from highest to lowest, and represent checkpoints along the path from the node’s current tip back to genesis. The remote peer computes the intersection between its chain and the node’s chain by scanning through the block locator for the first hash in its chain. Then, it sends (up to) 500 subsequent block hashes in an inv message (in the case of getblocks) or (up to) 2000 block headers in a headers message (in the case of getheaders). Note: zcashd reduces the getheaders count to 160, because Zcash headers are much larger than Bitcoin headers, as noted below.
The headers message sent after getheaders contains the actual block headers, while the inv message sent after getblocks contains only hashes, which have to be fetched with a getdata message. In Bitcoin, the block headers are small relative to the size of the full block, but this is not always the case for Zcash, where the block headers are much larger due to the use of Equihash and many blocks have only a few transactions. Also, getblocks allows parallelizing block downloads, while getheaders doesn’t. For these reasons and because we know we need full blocks anyways, we should probably use getblocks.
The getblocks Bitcoin message corresponds to our zebra_network::Request::FindBlocksByHash, and the getdata message is generated by zebra_network::Request::Blocks.
Pipelining block verification
As mentioned above, our goal is to be able to pipeline block download and verification. This means that the process for block lookup should ideally attempt to fetch and begin verification of future blocks without blocking on complete verification of all earlier blocks. To do this, we split the chain state into the verified block chain (held by the state component) and the prospective block chain (held only by the syncer), and use the following algorithm to pursue prospective chain tips.
ObtainTips
- Query the current state to construct the sequence of hashes
[tip, tip-1, tip-2, ..., tip-9, tip-20, tip-40, tip-80, tip-160 ]
The precise construction is unimportant, but this should have a Bitcoin-style dense-first, then-sparse hash structure.
The initial state should contain the genesis block for the relevant network. So the sequence of hashes will only contain the genesis block
[genesis ]
The network will respond with a list of hashes, starting at the child of the genesis block.
-
Make a
FindBlocksByHashrequest to the networkFtimes, whereFis a fanout parameter, to getresp1, ..., respF. -
For each response, starting from the beginning of the list, prune any block hashes already included in the state, stopping at the first unknown hash to get
resp1', ..., respF'. (These lists may be empty). -
Combine the last elements of each list into a set; this is the set of prospective tips.
-
Combine all elements of each list into a set, and queue download and verification of those blocks.
-
If there are any prospective tips, call ExtendTips, which returns a new set of prospective tips. Continue calling ExtendTips with this new set, until there are no more prospective tips.
-
Restart after some delay, say 15 seconds.
ExtendTips
-
Remove all prospective tips from the set of prospective tips, then iterate through them. For each removed tip:
-
Create a
FindBlocksByHashrequest consisting of just the prospective tip. Send this request to the networkFtimes. -
For each response, check whether the first hash in the response is a genesis block (for either the main or test network). If so, discard the response. It indicates that the remote peer does not have any blocks following the prospective tip. (Or that the remote peer is on the wrong network.)
-
Combine the last elements of the remaining responses into a set, and add this set to the set of prospective tips.
-
Combine all elements of the remaining responses into a set, and queue download and verification of those blocks.
DoS resistance
Because this strategy aggressively downloads any available blocks, it could be vulnerable to a DoS attack, where a malicious peer feeds us bogus chain tips, causing us to waste network and CPU on blocks that will never be valid. However, because we separate block finding from block downloading, and because of the design of our network stack, this attack is probably not feasible. The primary reason is that zebra_network randomly loadbalances outbound requests over all available peers.
Consider a malicious peer who responds to block discovery with a bogus list of hashes. We will eagerly attempt to download all of those bogus blocks, but our requests to do so will be randomly load-balanced to other peers, who are unlikely to know about the bogus blocks. When we try to extend a bogus tip, the extension request will also be randomly load-balanced, so it will likely be routed to a peer that doesn’t know about it and can’t extend it. And because we perform multiple block discovery queries, which will also be randomly load balanced, we’re unlikely to get stuck on a false chain tip.
Fork-finding
When starting from a verified chain tip, the choice of block locator can find forks at least up to the reorg limit (99 blocks). When extending a prospective tip, forks are ignored, but this is fine, since unless we are prefetching the longest chain, we won’t be able to keep extending the tip prospectively.
Retries and Fanout
We should consider the fanout parameter F and the retry policy for the different requests. I’m not sure whether we need to retry requests to discover new block hashes, since the fanout may already provide redundancy. For the block requests themselves, we should have a retry policy with a limited number of attempts, enough to insulate against network failures but not so many that we would retry a bogus block indefinitely. Maybe fanout 4 and 3 retries?
Parallel Verification
Status (Zebra 7.0.0): Implemented.
Code: the router, block verifier, checkpoint verifier, and contextual checks.
- The router is
BlockVerifierRouter; it usesSemanticBlockVerifier. - The router accepts
Request::CommitorRequest::CheckProposal, not a bare block. - The verifiers commit blocks to state; the router does not.
- The block verifier sends
CommitSemanticallyVerifiedBlockrequests. - The checkpoint verifier sends
CommitCheckpointVerifiedBlockrequests.
- Feature Name: parallel_verification
- Start Date: 2020-07-27
- Design PR: ZcashFoundation/zebra#763
- Zebra Issue: ZcashFoundation/zebra#682
Summary
Zebra verifies blocks in several stages, most of which can be executed in parallel.
We use several different design patterns to enable this parallelism:
- We download blocks and start verifying them in parallel,
- We batch signature and proof verification using verification services, and
- We defer data dependencies until just before the block is committed to the state (see the detailed design RFCs).
Motivation
Zcash (and Bitcoin) are designed to verify each block in sequence, starting from the genesis block. But during the initial sync, and when restarting with an older state, this process can be quite slow.
By deferring data dependencies, we can partially verify multiple blocks in parallel.
By parallelising block and transaction verification, we can use multithreading and batch verification for signatures, proofs, scripts, and hashes.
Definitions
Blockchain:
- chain fork: Zcash is implemented using a tree of blocks. Each block has a single previous block, and zero to many next blocks. A chain fork consists of a tip and all its previous blocks, back to the genesis block.
- genesis: The root of the tree of blocks is called the genesis block. It has no previous block.
- tip: A block which has no next block is called a tip. Each chain fork can be identified using its tip.
Data:
- consensus rule: A protocol rule which all nodes must apply consistently, so they can converge on the same chain fork.
- context-free: Consensus rules which do not have a data dependency on previous blocks.
- data dependency: Information contained in the previous block and its chain fork, which is required to verify the current block.
- state: The set of verified blocks. The state might also cache some dependent data, so that we can efficiently verify subsequent blocks.
Verification Stages:
- structural verification: Parsing raw bytes into the data structures defined by the protocol.
- semantic verification: Verifying the consensus rules on the data structures defined by the protocol.
- contextual verification: Verifying the current block, once its data dependencies have been satisfied by a verified previous block. This verification might also use the cached state corresponding to the previous block.
Guide-level explanation
In Zebra, we want to verify blocks in parallel. Some fields can be verified straight away, because they don’t depend on the output of previous blocks. But other fields have data dependencies, which means that we need previous blocks before we can fully validate them.
If we delay checking some of these data dependencies, then we can do more of the verification in parallel.
Example: BlockHeight
Here’s how Zebra can verify the different Block Height consensus rules in parallel:
Structural Verification:
- Parse the Block into a BlockHeader and a list of transactions.
Semantic Verification: No Data Dependencies:
- Check that the first input of the first transaction in the block is a coinbase input with a valid block height in its data field.
Semantic Verification: Deferring a Data Dependency:
- Verify other consensus rules that depend on Block Height, assuming that the Block Height is correct. For example, many consensus rules depend on the current Network Upgrade, which is determined by the Block Height. We verify these consensus rules, assuming the Block Height and Network Upgrade are correct.
Contextual Verification:
- Submit the block to the state for contextual verification. When it is ready to be committed (it may arrive before the previous block), check all deferred constraints, including the constraint that the block height of this block is one more than the block height of its parent block. If all constraints are satisfied, commit the block to the state. Otherwise, reject the block as invalid.
Zebra Design
Design Patterns
When designing changes to Zebra verification, use these design patterns:
- perform context-free verification as soon as possible, (that is, verification which has no data dependencies on previous blocks),
- defer data dependencies as long as possible, then
- check the data dependencies.
Minimise Deferred Data
Keep the data dependencies and checks as simple as possible.
For example, Zebra could defer checking both the Block Height and Network Upgrade.
But since the Network Upgrade depends on the Block Height, we only need to defer the Block Height check. Then we can use all the fields that depend on the Block Height, as if it is correct. If the final Block Height check fails, we will reject the entire block, including all the verification we performed using the assumed Network Upgrade.
Implementation Strategy
When implementing these designs, perform as much verification as possible, await any dependencies, then perform the necessary checks.
Reference-level explanation
Verification Stages
In Zebra, verification occurs in the following stages:
- Structural Verification: Raw block data is parsed into a block header and transactions. Invalid data is not representable in these structures: deserialization (parsing) can fail, but serialization always succeeds.
- Semantic Verification: Parsed block fields are verified, based on their
data dependencies:
- Context-free fields have no data dependencies, so they can be verified as needed.
- Fields with simple data dependencies defer that dependency as long as possible, so they can perform more verification in parallel. Then they await the required data, which is typically the previous block. (And potentially older blocks in its chain fork.)
- Fields with complex data dependencies require their own parallel verification designs. These designs are out of scope for this RFC.
- Contextual Verification: After a block is verified, it is added to the state. The details of state updates, and their interaction with semantic verification, are out of scope for this RFC.
This RFC focuses on Semantic Verification, and the design patterns that enable blocks to be verified in parallel.
Verification Interfaces
Verification is implemented by the following traits and services:
- Structural Verification:
zebra_chain::ZcashDeserialize: A trait for parsing consensus-critical data structures from a byte buffer.
- Semantic Verification:
ChainVerifier: Provides a verifier service that accepts aBlockrequest, performs verification on the block, and responds with ablock::Hashon success.- Internally, the
ChainVerifierselects between aCheckpointVerifierfor blocks that are within the checkpoint range, and aBlockVerifierfor recent blocks.
- Contextual Verification:
zebra_state::init: Provides the state update service, which accepts requests to add blocks to the state.
Checkpoint Verification
The CheckpointVerifier performs rapid verification of blocks, based on a set
of hard-coded checkpoints. Each checkpoint hash can be used to verify all the
previous blocks, back to the genesis block. So Zebra can skip almost all verification for blocks in the checkpoint range.
The CheckpointVerifier uses an internal queue to store pending blocks.
Checkpoint verification is cheap, so it is implemented using non-async
functions within the CheckpointVerifier service.
Here is how the CheckpointVerifier implements each verification stage:
- Structural Verification:
- As Above: the
CheckpointVerifieraccepts parsedBlockstructs.
- As Above: the
- Semantic Verification:
check_height: makes sure the block height is within the unverified checkpoint range, and adds the block to its internal queue.target_checkpoint_height: Checks for a continuous range of blocks from the previous checkpoint to a subsequent checkpoint. If the chain is incomplete, returns a future, and waits for more blocks. If the chain is complete, assumes that theprevious_block_hashfields of these blocks form an unbroken chain from checkpoint to checkpoint, and starts processing the checkpoint range. (This constraint is an implicit part of theCheckpointVerifierdesign.)process_checkpoint_range: makes sure that the blocks in the checkpoint range have an unbroken chain of previous block hashes.
- Contextual Verification:
- As Above: the
CheckpointVerifierreturns success to theChainVerifier, which sends verifiedBlocks to the state service.
- As Above: the
Block Verification
The BlockVerifier performs detailed verification of recent blocks, in parallel.
Here is how the BlockVerifier implements each verification stage:
- Structural Verification:
- As Above: the
BlockVerifieraccepts parsedBlockstructs.
- As Above: the
- Semantic Verification:
- As Above: verifies each field in the block. Defers any data dependencies as long as possible, awaits those data dependencies, then performs data dependent checks.
- Note: Since futures are executed concurrently, we can use the same function
to:
- perform context-free verification,
- perform verification with deferred data dependencies,
- await data dependencies, and
- check data dependencies. To maximise concurrency, we should write verification functions in this specific order, so the awaits are as late as possible.
- Contextual Verification:
- As Above: the
BlockVerifierreturns success to theChainVerifier, which sends verifiedBlocks to the state service.
- As Above: the
Zcash Protocol Design
When designing a change to the Zcash protocol, minimise the data dependencies between blocks.
Try to create designs that:
- Eliminate data dependencies,
- Make the changes depend on a version field in the block header or transaction,
- Make the changes depend on the current Network Upgrade, or
- Make the changes depend on a field in the current block, with an additional consensus rule to check that field against previous blocks.
When making decisions about these design tradeoffs, consider:
- how the data dependency could be deferred, and
- the CPU cost of the verification - if it is trivial, then it does not matter if the verification is parallelised.
Drawbacks
This design is a bit complicated, but we think it’s necessary to achieve our goals.
Rationale and alternatives
- What makes this design a good design?
- It enables a significant amount of parallelism
- It is simpler than some other alternatives
- It uses existing Rust language facilities, mainly Futures and await/async
- Is this design a good basis for later designs or implementations?
- We have built a UTXO design on this design
- We believe we can build “recent blocks” and “chain summary” designs on this design
- Each specific detailed design will need to consider how the relevant data dependencies are persisted
- What other designs have been considered and what is the rationale for not choosing them?
- Serial verification
- Effectively single-threaded
- Awaiting data dependencies as soon as they are needed
- Less parallelism
- Providing direct access to the state
- Might cause data races, might be prevented by Rust’s ownership rules
- Higher risk of bugs
- Serial verification
- What is the impact of not doing this?
- Verification is slow, we can’t batch or parallelise some parts of the verification
Prior art
TODO: expand this section
- zcashd
- serial block verification
- Zebra implements the same consensus rules, but a different design
- tower
Unresolved questions
- Is this design good enough to use as a framework for future RFCs?
- Does this design require any changes to the current implementation?
- Implement block height consensus rule (check previous block hash and height)
- Check that the
BlockVerifierperforms checks in the following order:- verification, deferring dependencies as needed,
- await dependencies,
- check deferred data dependencies
Out of Scope:
-
What is the most efficient design for parallel verification?
- (Optimisations are out of scope.)
-
How is each specific field verified?
-
How do we verify fields with complex data dependencies?
-
How does verification change with different network upgrades?
-
How do multiple chains work, in detail?
-
How do state updates work, in detail?
-
Moving the verifiers into the state service
Future possibilities
- Separate RFCs for other data dependencies
- Recent blocks
- Overall chain summaries (for example, total work)
- Reorganisation limit: multiple chains to single chain transition
- Optimisations for parallel verification
Inventory tracking
Status (Zebra 7.0.0): Implemented.
Code: the inventory registry and peer set.
- The registry tracks available/missing inventory and failed requests.
- Bounded
IndexMap/IndexSetcollections replace the proposed maps. - Limits are 1,000 hashes per map and 70 peers per hash.
- Changes use
InventoryStatus<(AtLeastOne<InventoryHash>, PeerSocketAddr)>. - The rotation interval is 53 seconds, not 75; missed ticks delay rather than replay.
- Feature Name:
inventory_tracking - Start Date: 2020-08-25
- Design PR: ZcashFoundation/zebra#952
- Zebra Issue: ZcashFoundation/zebra#960
Summary
The Bitcoin network protocol used by Zcash allows nodes to advertise data (inventory items) for download by other peers. This RFC describes how we track and use this information.
Motivation
In order to participate in the network, we need to be able to fetch new data that our peers notify us about. Because our network stack abstracts away individual peer connections, and load-balances over available peers, we need a way to direct requests for new inventory only to peers that advertised to us that they have it.
Definitions
- Inventory item: either a block or transaction.
- Inventory hash: the hash of an inventory item, represented by the
InventoryHashtype. - Inventory advertisement: a notification from another peer that they have some inventory item.
- Inventory request: a request to another peer for an inventory item.
Guide-level explanation
The Bitcoin network protocol used by Zcash provides a mechanism for nodes to
gossip blockchain data to each other. This mechanism is used to distribute
(mined) blocks and (unmined) transactions through the network. Nodes can
advertise data available in their inventory by sending an inv message
containing the hashes and types of those data items. After receiving an inv
message advertising data, a node can determine whether to download it.
This poses a challenge for our network stack, which goes to some effort to abstract away details of individual peers and encapsulate all peer connections behind a single request/response interface representing “the network”. Currently, the peer set tracks readiness of all live peers, reports readiness if at least one peer is ready, and routes requests across ready peers randomly using the “power of two choices” algorithm.
However, while this works well for data that is already distributed across the network (e.g., existing blocks) it will not work well for fetching data during distribution across the network. If a peer informs us of some new data, and we attempt to download it from a random, unrelated peer, we will likely fail. Instead, we track recent inventory advertisements, and make a best-effort attempt to route requests to peers who advertised that inventory.
Reference-level explanation
The inventory tracking system has several components:
- A registration hook that monitors incoming messages for inventory advertisements;
- An inventory registry that tracks inventory presence by peer;
- Routing logic that uses the inventory registry to appropriately route requests.
The first two components have fairly straightforward design decisions, but the third has considerably less obvious choices and tradeoffs.
Inventory Monitoring
Zebra uses Tokio’s codec mechanism to transform a byte-oriented I/O interface
into a Stream and Sink for incoming and outgoing messages. These are
passed to the peer connection state machine, which is written generically over
any Stream and Sink. This construction makes it easy to “tap” the sequence
of incoming messages using .then and .with stream and sink combinators.
We already do this to record Prometheus metrics on message rates as well as to
report message timestamps used for liveness checks and last-seen address book
metadata. The message timestamp mechanism is a good example to copy. The
handshake logic instruments the incoming message stream with a closure that
captures a sender handle for a mpsc channel with a large buffer (currently 100
timestamp entries). The receiver handle is owned by a separate task that shares
an Arc<Mutex<AddressBook>> with other parts of the application. This task
waits for new timestamp entries, acquires a lock on the address book, and
updates the address book. This ensures that timestamp updates are queued
asynchronously, without lock contention.
Unlike the address book, we don’t need to share the inventory data with other
parts of the application, so it can be owned exclusively by the peer set. This
means that no lock is necessary, and the peer set can process advertisements in
its poll_ready implementation. This method may be called infrequently, which
could cause the channel to fill. However, because inventory advertisements are
time-limited, in the sense that they’re only useful before some item is fully
distributed across the network, it’s safe to handle excess entries by dropping
them. This behavior is provided by a broadcast/mpmc channel, which can be
used in place of an mpsc channel.
An inventory advertisement is an (InventoryHash, SocketAddr) pair. The
stream hook should check whether an incoming message is an inv message with
only a small number (e.g., 1) inventory entries. If so, it should extract the
hash for each item and send it through the channel. Otherwise, it should
ignore the message contents. Why? Because inv messages are also sent in
response to queries, such as when we request subsequent block hashes, and in
that case we want to assume that the inventory is generally available rather
than restricting downloads to a single peer. However, items are usually
gossiped individually (or potentially in small chunks; zcashd has an internal
inv buffer subject to race conditions), so choosing a small bound such as 1
is likely to work as a heuristic for when we should assume that advertised
inventory is not yet generally available.
Inventory Registry
The peer set’s poll_ready implementation should extract all available
(InventoryHash, SocketAddr) pairs from the channel, and log a warning event
if the receiver is lagging. The channel should be configured with a generous
buffer size (such as 100) so that this is unlikely to happen in normal
circumstances. These pairs should be fed into an InventoryRegistry structure
along these lines:
#![allow(unused)]
fn main() {
struct InventoryRegistry{
current: HashMap<InventoryHash, HashSet<SocketAddr>>,
prev: HashMap<InventoryHash, HashSet<SocketAddr>>,
}
impl InventoryRegistry {
pub fn register(&mut self, item: InventoryHash, addr: SocketAddr) {
self.0.entry(item).or_insert(HashSet::new).insert(addr);
}
pub fn rotate(&mut self) {
self.prev = std::mem::take(self.current)
}
pub fn peers(&self, item: InventoryHash) -> impl Iterator<Item=&SocketAddr> {
self.prev.get(item).chain(self.current.get(item)).flatten()
}
}
}
This API allows pruning the inventory registry using rotate, which
implements generational pruning of registry entries. The peer set should
maintain a tokio::time::Interval with some interval parameter, and check in
poll_ready whether the interval stream has any items, calling rotate for
each one:
#![allow(unused)]
fn main() {
while let Poll::Ready(Some(_)) = timer.poll_next(cx) {
registry.rotate();
}
}
By rotating for each available item in the interval stream, rather than just
once, we ensure that if the peer set’s poll_ready is not called for a long
time, rotate will be called enough times to correctly flush old entries.
Inventory advertisements live in the registry for twice the length of the timer, so it should be chosen to be half of the desired lifetime for inventory advertisements. Setting the timer to 75 seconds, the block interval, seems like a reasonable choice.
Routing Logic
At this point, the peer set has information on recent inventory advertisements.
However, the Service trait only allows poll_ready to report readiness based
on the service’s data and the type of the request, not the content of the
request. This means that we must report readiness without knowing whether the
request should be routed to a specific peer, and we must handle the case where
call gets a request for an item only available at an unready peer.
This RFC suggests the following routing logic. First, check whether the
request fetches data by hash. If so, and peers() returns Some(ref addrs),
iterate over addrs and route the request to the first ready peer if there is
one. In all other cases, fall back to p2c routing. Alternatives are suggested
and discussed below.
Rationale and alternatives
The rationale is described above. The alternative choices are primarily around the routing logic.
Because the Service trait does not allow applying backpressure based on the
content of a request, only based on the service’s internal data (via the
&mut self parameter of Service::poll_ready) and on the type of the
request (which determines which impl Service is used). This means that it
is impossible for us to apply backpressure until a service that can process a
specific inventory request is ready, because until we get the request, we
can’t determine which peers might be required to process it.
We could attempt to ensure that the peer set would be ready to process a specific inventory request would be to preemptively “reserve” a peer as soon as it advertises an inventory item. But this doesn’t actually work to ensure readiness, because a peer could advertise two inventory items, and only be able to service one request at a time. It also potentially locks the peer set, since if there are only a few peers and they all advertise inventory, the service can’t process any other requests. So this approach does not work.
Another alternative would be to do some kind of buffering of inventory requests that cannot immediately be processed by a peer that advertised that inventory. There are two basic sub-approaches here.
In the first case, we could maintain an unbounded queue of yet-to-be
processed inventory requests in the peer set, and every time poll_ready is
called, we check whether a service that could serve those inventory requests
became ready, and start processing the request if we can. This would provide
the lowest latency, because we can dispatch the request to the first
available peer. For instance, if peer A advertises inventory I, the peer set
gets an inventory request for I, peer A is busy so the request is queued, and
peer B advertises inventory I, we could dispatch the queued request to B
rather than waiting for A.
However, it’s not clear exactly how we’d implement this, because this
mechanism is driven by calls to poll_ready, and those might not happen. So
we’d need some separate task that would drive processing the buffered task to
completion, but this may not be able to do so by poll_ready, since that
method requires owning the service, and the peer set will be owned by a
Buffer worker.
In the second case, we could select an unready peer that advertised the
requested inventory, clone it, and move the cloned peer into a task that
would wait for that peer to become ready and then make the request. This is
conceptually much cleaner than the above mechanism, but it has the downside
that we don’t dispatch the request to the first ready peer. In the example
above, if we cloned peer A and dispatched the request to it, we’d have to
wait for A to become ready, even if the second peer B advertised the same
inventory just after we dispatched the request to A. However, this is not
presently possible anyways, because the peer::Clients that handle requests
are not clonable. They could be made clonable (they send messages to the
connection state machine over a mpsc channel), but we cannot make this change
without altering our liveness mechanism, which uses bounds on the
time-since-last-message to determine whether a peer connection is live and to
prevent immediate reconnections to recently disconnected peers.
A final alternative would be to fail inventory requests that we cannot route to a peer which advertised that inventory. This moves the failure forward in time, but preemptively fails some cases where the request might succeed – for instance, if the peer has inventory but just didn’t tell us, or received the inventory between when we dispatch the request and when it receives our message. It seems preferable to try and fail than to not try at all.
In practice, we’re likely to care about the gossip protocol and inventory fetching once we’ve already synced close to the chain tip. In this setting, we’re likely to already have peer connections, and we’re unlikely to be saturating our peer set with requests (as we do during initial block sync). This suggests that the common case is one where we have many idle peers, and that therefore we are unlikely to have dispatched any recent requests to the peer that advertised inventory. So our common case should be one where all of this analysis is irrelevant.
Asynchronous script verification
Status (Zebra 7.0.0): Implemented.
Code: transaction verification, the script service, and pending UTXOs.
- The lookup API uses
Request::AwaitUtxoandResponse::Utxo. - Lookup neither enforces spendability nor guarantees valid-chain membership.
- Block coinbase spend checks return contextual-validation errors, not lookup timeouts.
- The transaction verifier applies a 6-minute UTXO lookup timeout.
- Pending requests share broadcast senders per outpoint, not locked oneshots.
- Scripts run on Rayon through
CachedFfiTransactionandlibzcash_script, notzcash_consensus.
- Start Date: 2020-08-10
- Design PR: ZcashFoundation/zebra#868
- Zebra Issue: ZcashFoundation/zebra#964
Summary
This RFC describes an architecture for asynchronous script verification and its interaction with the state layer. This architecture imposes constraints on the ordering of operations in the state layer.
Motivation
As in the rest of Zebra, we want to express our work as a collection of work-items with explicit dependencies, then execute these items concurrently and in parallel on a thread pool.
Definitions
- UTXO: unspent transparent transaction output.
Transparent transaction outputs are modeled in
zebra-chainby thetransparent::Outputstructure. - outpoint: a reference to an unspent transparent transaction output, including a transaction hash and output index.
Outpoints are modeled in
zebra-chainby thetransparent::OutPointstructure. - transparent input: a previous transparent output consumed by a later transaction (the one it is an input to).
Modeled in
zebra-chainby thetransparent::Input::PrevOutenum variant. - coinbase transaction: the first transaction in each block, which creates new coins.
- lock script: the script that defines the conditions under which some UTXO can be spent.
Stored in the
transparent::Output::lock_scriptfield. - unlock script: a script satisfying the conditions of the lock script, allowing a UTXO to be spent.
Stored in the
transparent::Input::PrevOut::unlock_scriptfield.
Guide-level explanation
Zcash’s transparent address system is inherited from Bitcoin. Transactions spend unspent transparent transaction outputs (UTXOs) from previous transactions. These UTXOs are encumbered by locking scripts that define the conditions under which they can be spent, e.g., requiring a signature from a certain key. Transactions wishing to spend UTXOs supply an unlocking script that should satisfy the conditions of the locking script for each input they wish to spend.
This means that script verification requires access to data about previous UTXOs, in order to determine the conditions under which those UTXOs can be spent. In Zebra, we aim to run operations asynchronously and out-of-order to the greatest extent possible. For instance, we may begin verification of a block before all of its ancestors have been verified or even downloaded. So we need to design a mechanism that allows script verification to declare its data dependencies and execute as soon as all required data is available.
It’s not necessary for this mechanism to ensure that the transaction outputs remain unspent, only to give enough information to perform script verification. Checking that all transaction inputs are actually unspent is done later, at the point that its containing block is committed to the chain.
At a high level, this adds a new request/response pair to the state service:
Request::AwaitSpendableUtxo { output: OutPoint, ..conditions }requests a spendabletransparent::Output, looked up usingOutPoint.Response::SpendableUtxo(Utxo)supplies the requestedtransparent::Outputas part of a newUtxotype, if the output is spendable based onconditions;
Note that this request is named differently from the other requests,
AwaitSpendableUtxo rather than GetUtxo or similar. This is because the
request has rather different behavior:
- the request does not complete until the state service learns about a UTXO matching the request, which could be never. For instance, if the transaction output was already spent, the service is not required to return a response.
- the request does not complete until the output is spendable, based on the
conditionsin the request.
The state service does not cancel long-running UTXO requests. Instead, the caller is responsible for deciding when a request is unlikely to complete. (For example, using a timeout layer.)
This allows a script verifier to asynchronously obtain information about previous transaction outputs and start verifying scripts as soon as the data is available. For instance, if we begin parallel download and verification of 500 blocks, we should be able to begin script verification of all scripts referencing outputs from existing blocks in parallel, and begin verification of scripts referencing outputs from new blocks as soon as they are committed to the chain.
Because spending outputs from older blocks is more common than spending outputs from recent blocks, this should allow a significant amount of parallelism.
Reference-level explanation
Data structures
We add the following request and response to the state protocol:
#![allow(unused)]
fn main() {
enum Request::AwaitSpendableUtxo {
outpoint: OutPoint,
spend_height: Height,
spend_restriction: SpendRestriction,
}
/// Consensus rule:
/// "A transaction with one or more transparent inputs from coinbase transactions
/// MUST have no transparent outputs (i.e.tx_out_count MUST be 0)."
enum SpendRestriction {
/// The UTXO is spent in a transaction with transparent outputs
SomeTransparentOutputs,
/// The UTXO is spent in a transaction with all shielded outputs
AllShieldedOutputs,
}
}
As described above, the request name is intended to indicate the request’s behavior. The request does not resolve until:
- the state layer learns of a UTXO described by the request, and
- the output is spendable at
heightwithspend_restriction.
The new Utxo type adds a coinbase flag and height to transparent::Outputs
that we look up in the state, or get from newly committed blocks:
#![allow(unused)]
fn main() {
enum Response::SpendableUtxo(Utxo)
pub struct Utxo {
/// The output itself.
pub output: transparent::Output,
/// The height at which the output was created.
pub height: block::Height,
/// Whether the output originated in a coinbase transaction.
pub from_coinbase: bool,
}
}
Transparent coinbase consensus rules
Specifically, if the UTXO is a transparent coinbase output, the service is not required to return a response if:
spend_heightis less thanMIN_TRANSPARENT_COINBASE_MATURITY(100) blocks after theUtxo.height, orspend_restrictionisSomeTransparentOutputs.
This implements the following consensus rules:
A transaction MUST NOT spend a transparent output of a coinbase transaction from a block less than 100 blocks prior to the spend.
Note that transparent outputs of coinbase transactions include Founders’ Reward outputs and transparent funding stream outputs.
A transaction with one or more transparent inputs from coinbase transactions MUST have no transparent outputs (i.e.tx_out_count MUST be 0).
Inputs from coinbase transactions include Founders’ Reward outputs and funding stream outputs.
https://zips.z.cash/protocol/protocol.pdf#txnencodingandconsensus
Parallel coinbase checks
We can perform these coinbase checks asynchronously, in the presence of multiple chain forks, as long as the following conditions both hold:
-
We don’t mistakenly accept or reject spends to the transparent pool.
-
We don’t mistakenly accept or reject mature spends.
Parallel coinbase justification
There are two parts to a spend restriction:
- the
from_coinbaseflag, and - if the
from_coinbaseflag is true, the coinbaseheight.
If a particular transaction hash h always has the same from_coinbase value,
and h exists in multiple chains, then regardless of which Utxo arrives first,
the outputs of h always get the same from_coinbase value during validation.
So spends can not be mistakenly accepted or rejected due to a different coinbase flag.
Similarly, if a particular coinbase transaction hash h always has the same height value,
and h exists in multiple chains, then regardless of which Utxo arrives first,
the outputs of h always get the same height value during validation.
So coinbase spends can not be mistakenly accepted or rejected due to a different height value.
(The heights of non-coinbase outputs are irrelevant, because they are never checked.)
These conditions hold as long as the following multi-chain properties are satisfied:
from_coinbase: across all chains, the set of coinbase transaction hashes is disjoint from the set of non-coinbase transaction hashes, and- coinbase
height: across all chains, duplicate coinbase transaction hashes can only occur at exactly the same height.
Parallel coinbase consensus rules
These multi-chain properties can be derived from the following consensus rules:
Transaction versions 1-4:
[Pre-Sapling ] If effectiveVersion = 1 or nJoinSplit = 0, then both tx_in_count and tx_out_count MUST be nonzero. … [Sapling onward] If effectiveVersion < 5, then at least one of tx_in_count, nSpendsSapling, and nJoinSplit MUST be nonzero.
A coinbase transaction for a block at block height greater than 0 MUST have a script that, as its first item, encodes the block height height as follows.
For height in the range {1 .. 16}, the encoding is a single byte of value 0x50 + height.
Otherwise, let heightBytes be the signed little-endian representation of height, using the minimum nonzero number of bytes such that the most significant byte is < 0x80. The length of heightBytes MUST be in the range {1 .. 8}. Then the encoding is the length of heightBytes encoded as one byte, followed by heightBytes itself.
https://zips.z.cash/protocol/protocol.pdf#txnencodingandconsensus
The transaction ID of a version 4 or earlier transaction is the SHA-256d hash of the transaction encoding in the pre-v5 format described above.
https://zips.z.cash/protocol/protocol.pdf#txnidentifiers
Transaction version 5:
[NU5 onward] If effectiveVersion ≥ 5, then this condition must hold: tx_in_count > 0 or nSpendsSapling > 0 or (nActionsOrchard > 0 and enableSpendsOrchard = 1). … [NU5 onward] The nExpiryHeight field of a coinbase transaction MUST be equal to its block height.
https://zips.z.cash/protocol/protocol.pdf#txnencodingandconsensus
non-malleable transaction identifiers … commit to all transaction data except for attestations to transaction validity … A new transaction digest algorithm is defined that constructs the identifier for a transaction from a tree of hashes … A BLAKE2b-256 hash of the following values: … T.1e: expiry_height (4-byte little-endian block height)
https://zips.z.cash/zip-0244#t-1-header-digest
Since:
- coinbase transaction hashes commit to the block
Height, - non-coinbase transaction hashes commit to their inputs, and
- double-spends are not allowed;
Therefore:
- coinbase transaction hashes are unique for distinct heights in any chain,
- coinbase transaction hashes are unique in a single chain, and
- non-coinbase transaction hashes are unique in a single chain, because they recursively commit to unique inputs.
So the required parallel verification conditions are satisfied.
Script verification
To verify scripts, a script verifier requests the relevant UTXOs from the
state service and waits for all of them to resolve, or fails verification
with a timeout error. Currently, we outsource script verification to
zcash_consensus, which does FFI into the same C++ code as zcashd uses.
We need to ensure this code is thread-safe.
Database implementation
Implementing the state request correctly requires considering two sets of behaviors:
- behaviors related to the state’s external API (a
Bufferedtower::Service); - behaviors related to the state’s internal implementation (using
rocksdb).
Making this distinction helps us to ensure we don’t accidentally leak
“internal” behaviors into “external” behaviors, which would violate
encapsulation and make it more difficult to replace rocksdb.
In the first category, our state is presented to the rest of the application
as a Buffered tower::Service. The Buffer wrapper allows shared access
to a service using an actor model, moving the service to be shared into a
worker task and passing messages to it over an multi-producer single-consumer
(mpsc) channel. The worker task receives messages and makes Service::calls.
The Service::call method returns a Future, and the service is allowed to
decide how much work it wants to do synchronously (in call) and how much
work it wants to do asynchronously (in the Future it returns).
This means that our external API ensures that the state service sees a linearized sequence of state requests, although the exact ordering is unpredictable when there are multiple senders making requests.
Because the state service has exclusive access to the rocksdb database, and the
state service sees a linearized sequence of state requests, we have an easy
way to opt in to asynchronous database access. We can perform rocksdb operations
synchronously in the Service::call, waiting for them to complete, and be
sure that all future requests will see the resulting rocksdb state. Or, we can
perform rocksdb operations asynchronously in the future returned by
Service::call.
If we perform all writes synchronously and allow reads to be either synchronous or asynchronous, we ensure that writes cannot race each other. Asynchronous reads are guaranteed to read at least the state present at the time the request was processed, or a later state.
Lookup states
Now, returning to the UTXO lookup problem, we can map out the possible states with this restriction in mind. This description assumes that UTXO storage is split into disjoint sets, one in-memory (e.g., blocks after the reorg limit) and the other in rocksdb (e.g., blocks after the reorg limit). The details of this storage are not important for this design, only that the two sets are disjoint.
When the state service processes a Request::AwaitSpendableUtxo referencing
some UTXO u, there are three disjoint possibilities:
uis already contained in an in-memory block storage;uis already contained in the rocksdb UTXO set;uis not yet known to the state service.
In case 3, we need to queue u and scan all future blocks to see whether
they contain u. However, if we have a mechanism to queue u, we can
perform check 2 asynchronously, because restricting to synchronous writes
means that any async read will return the current or later state. If u was
in the rocksdb UTXO set when the request was processed, the only way that an
async read would not return u is if the UTXO were spent, in which case the
service is not required to return a response.
Lookup implementation
This behavior can be encapsulated into a PendingUtxos
structure described below.
#![allow(unused)]
fn main() {
// sketch
#[derive(Default, Debug)]
struct PendingUtxos(HashMap<OutPoint, oneshot::Sender<Utxo>>);
impl PendingUtxos {
// adds the outpoint and returns (wrapped) rx end of oneshot
// checks the spend height and restriction before sending the utxo response
// return can be converted to `Service::Future`
pub fn queue(
&mut self,
outpoint: OutPoint,
spend_height: Height,
spend_restriction: SpendRestriction,
) -> impl Future<Output=Result<Response, ...>>;
// if outpoint is a hashmap key, remove the entry and send output on the channel
pub fn respond(&mut self, outpoint: OutPoint, output: transparent::Output);
/// check the list of pending UTXO requests against the supplied `utxos`
pub fn check_against(&mut self, utxos: &HashMap<transparent::OutPoint, Utxo>);
// scans the hashmap and removes any entries with closed senders
pub fn prune(&mut self);
}
}
The state service should maintain an Arc<Mutex<PendingUtxos>>, used as follows:
- In
Service::call(Request::AwaitSpendableUtxo { outpoint: u, .. }, the service should:
- call
PendingUtxos::queue(u)to get a futurefto return to the caller; - spawn a task that does a rocksdb lookup for
u, callingPendingUtxos::respond(u, output)if present; - check the in-memory storage for
u, callingPendingUtxos::respond(u, output)if present; - return
fto the caller (it may already be ready). The common case is thatureferences an old spendable UTXO, so spawning the lookup task first means that we don’t wait to check in-memory storage forubefore starting the rocksdb lookup.
- In
f, the future returned byPendingUtxos::queue(u), the service should check that theUtxois spendable before returning it:
- if
Utxo.from_coinbaseis false, return the utxo; - if
Utxo.from_coinbaseis true, check that:spend_restrictionisAllShieldedOutputs, andspend_heightis greater than or equal toMIN_TRANSPARENT_COINBASE_MATURITYplus theUtxo.height,- if both checks pass, return the utxo.
- if any check fails, drop the utxo, and let the request timeout.
- In
Service::call(Request::CommitBlock(block, ..)), the service should:
- check for double-spends of each UTXO in the block, and
- do any other transactional checks before committing a block as normal.
Because the
AwaitSpendableUtxorequest is informational, there’s no need to do the transactional checks before matching against pending UTXO requests, and doing so upfront can run expensive verification earlier than needed.
- In
Service::poll_ready(), the service should callPendingUtxos::prune()at least some of the time. This is required because when a consumer uses a timeout layer, the cancelled requests should be flushed from the queue to avoid a resource leak. However, doing this on every call will result in us spending a bunch of time iterating over the hashmap.
Drawbacks
One drawback of this design is that we may have to wait on a lock. However, the critical section basically amounts to a hash lookup and a channel send, so I don’t think that we’re likely to run into problems with long contended periods, and it’s unlikely that we would get a deadlock.
Rationale and alternatives
High-level design rationale is inline with the design sketch. One low-level
option would be to avoid encapsulating behavior in the PendingUtxos and
just have an Arc<Hashmap<..>>, so that the lock only protects the hashmap
lookup and not sending through the channel. But I think the current design is
cleaner and the cost is probably not too large.
Unresolved questions
- We need to pick a timeout for UTXO lookup. This should be long enough to account for the fact that we may start verifying blocks before all of their ancestors are downloaded.
State Updates
Status (Zebra 7.0.0): Implemented.
Code: the state service, write task, and non-finalized state.
- The in-memory/finalized boundary follows a 1,000-block rollback window, not 100.
- Initialization is async; callers add buffering.
- It returns a boxed read-write service and a separate
ReadStateService. - It also returns
LatestChainTipandChainTipChange. - A separate task processes queued writes sequentially, not
Service::call. - The commit requests are
CommitSemanticallyVerifiedBlockandCommitCheckpointVerifiedBlock. - Successful commits return
Response::Committed, notResponse::Added. - The
AwaitUtxorequest does not check spend restrictions. - Equal-work chains prefer the first-received tip, then tip hash.
- Feature Name: state_updates
- Start Date: 2020-08-14
- Design PR: https://github.com/ZcashFoundation/zebra/pull/902
- Zebra Issue: https://github.com/ZcashFoundation/zebra/issues/1049
Summary
Zebra manages chain state in the zebra-state crate, which allows state
queries via asynchronous RPC (in the form of a Tower service). The state
system is responsible for contextual verification in the sense of RFC2,
checking that new blocks are consistent with the existing chain state before
committing them. This RFC describes how the state is represented internally,
and how state updates are performed.
Motivation
We need to be able to access and modify the chain state, and we want to have a description of how this happens and what guarantees are provided by the state service.
Definitions
-
state data: Any data the state service uses to represent chain state.
-
structural/semantic/contextual verification: as defined in RFC2.
-
block chain: A sequence of valid blocks linked by inclusion of the previous block hash in the subsequent block. Chains are rooted at the genesis block and extend to a tip.
-
chain state: The state of the ledger after application of a particular sequence of blocks (state transitions).
-
block work: The approximate amount of work required for a miner to generate a block hash that passes the difficulty filter. The number of block header attempts and the mining time are proportional to the work value. Numerically higher work values represent longer processing times.
-
cumulative work: The sum of the block work of all blocks in a chain, from genesis to the chain tip.
-
best chain: The chain with the greatest cumulative work. This chain represents the consensus state of the Zcash network and transactions.
-
side chain: A chain which is not contained in the best chain. Side chains are pruned at the reorg limit, when they are no longer connected to the finalized state.
-
chain reorganization: Occurs when a new best chain is found and the previous best chain becomes a side chain.
-
reorg limit: The longest reorganization accepted by
zcashd, 100 blocks. -
orphaned block: A block which is no longer included in the best chain.
-
non-finalized state: State data corresponding to blocks above the reorg limit. This data can change in the event of a chain reorg.
-
finalized state: State data corresponding to blocks below the reorg limit. This data cannot change in the event of a chain reorg.
-
non-finalized tips: The highest blocks in each non-finalized chain. These tips might be at different heights.
-
finalized tip: The highest block in the finalized state. The tip of the best chain is usually 100 blocks (the reorg limit) above the finalized tip. But it can be lower during the initial sync, and after a chain reorganization, if the new best chain is at a lower height.
-
relevant chain: The relevant chain for a block starts at the previous block, and extends back to genesis.
-
relevant tip: The tip of the relevant chain.
Guide-level explanation
The zebra-state crate provides an implementation of the chain state storage
logic in a Zcash consensus node. Its main responsibility is to store chain
state, validating new blocks against the existing chain state in the process,
and to allow later querying of said chain state. zebra-state provides this
interface via a tower::Service based on the actor model with a
request/response interface for passing messages back and forth between the
state service and the rest of the application.
The main entry point for the zebra-state crate is the init function. This
function takes a zebra_state::Config and constructs a new state service,
which it returns wrapped by a tower::Buffer. This service is then interacted
with via the tower::Service trait.
#![allow(unused)]
fn main() {
use tower::{Service, ServiceExt};
let state = zebra_state::on_disk::init(state_config, network);
let request = zebra_state::Request::BlockLocator;
let response = state.ready_and().await?.call(request).await?;
assert!(matches!(response, zebra_state::Response::BlockLocator(_)));
}
Note: The tower::Service API requires that ready is always called
exactly once before each call. It is up to users of the zebra state service
to uphold this contract.
The tower::Buffer wrapper is Cloneable, allowing shared access to a common state service. This allows different tasks to share access to the chain state.
The set of operations supported by zebra-state are encoded in its Request
enum. This enum has one variant for each supported operation.
#![allow(unused)]
fn main() {
pub enum Request {
CommitBlock {
block: Arc<Block>,
},
CommitFinalizedBlock {
block: Arc<Block>,
},
Depth(Hash),
Tip,
BlockLocator,
Transaction(Hash),
Block(HashOrHeight),
// .. some variants omitted
}
}
zebra-state breaks down its requests into two categories and provides
different guarantees for each category: requests that modify the state, and
requests that do not. Requests that update the state are guaranteed to run
sequentially and will never race against each other. Requests that read state
are done asynchronously and are guaranteed to read at least the state present
at the time the request was processed by the service, or a later state
present at the time the request future is executed. The state service avoids
race conditions between the read state and the written state by doing all
contextual verification internally.
Reference-level explanation
State Components
Zcash (as implemented by zcashd) differs from Bitcoin in its treatment of
transaction finality. If a new best chain is detected that does not extend
the previous best chain, blocks at the end of the previous best chain become
orphaned (no longer included in the best chain). Their state updates are
therefore no longer included in the best chain’s chain state. The process of
rolling back orphaned blocks and applying new blocks is called a chain
reorganization. Bitcoin allows chain reorganizations of arbitrary depth,
while zcashd limits chain reorganizations to 100 blocks. (In zcashd, the
new best chain must be a side-chain that forked within 100 blocks of the tip
of the current best chain.)
This difference means that in Bitcoin, chain state only has probabilistic finality, while in Zcash, chain state is final once it is beyond the reorg limit. To simplify our implementation, we split the representation of the state data at the finality boundary provided by the reorg limit.
State data from blocks above the reorg limit (non-finalized state) is
stored in-memory and handles multiple chains. State data from blocks below
the reorg limit (finalized state) is stored persistently using rocksdb and
only tracks a single chain. This allows a simplification of our state
handling, because only finalized data is persistent and the logic for
finalized data handles less invariants.
One downside of this design is that restarting the node loses the last 100 blocks, but node restarts are relatively infrequent and a short re-sync is cheap relative to the cost of additional implementation complexity.
Another downside of this design is that we do not achieve exactly the same
behavior as zcashd in the event of a 51% attack: zcashd limits each chain
reorganization to 100 blocks, but permits multiple reorgs, while Zebra limits
all chain reorgs to 100 blocks. In the event of a successful 51% attack on
Zcash, this could be resolved by wiping the rocksdb state and re-syncing the new
chain, but in this scenario there are worse problems.
Service Interface
The state is accessed asynchronously through a Tower service interface. Determining what guarantees the state service can and should provide to the rest of the application requires considering two sets of behaviors:
- behaviors related to the state’s external API (a
Bufferedtower::Service); - behaviors related to the state’s internal implementation (using
rocksdb).
Making this distinction helps us to ensure we don’t accidentally leak
“internal” behaviors into “external” behaviors, which would violate
encapsulation and make it more difficult to replace rocksdb.
In the first category, our state is presented to the rest of the application
as a Buffered tower::Service. The Buffer wrapper allows shared access
to a service using an actor model, moving the service to be shared into a
worker task and passing messages to it over an multi-producer single-consumer
(mpsc) channel. The worker task receives messages and makes Service::calls.
The Service::call method returns a Future, and the service is allowed to
decide how much work it wants to do synchronously (in call) and how much
work it wants to do asynchronously (in the Future it returns).
This means that our external API ensures that the state service sees a linearized sequence of state requests, although the exact ordering is unpredictable when there are multiple senders making requests.
Because the state service has exclusive access to the rocksdb database, and the
state service sees a linearized sequence of state requests, we have an easy
way to opt in to asynchronous database access. We can perform rocksdb operations
synchronously in the Service::call, waiting for them to complete, and be
sure that all future requests will see the resulting rocksdb state. Or, we can
perform rocksdb operations asynchronously in the future returned by
Service::call.
If we perform all writes synchronously and allow reads to be either synchronous or asynchronous, we ensure that writes cannot race each other. Asynchronous reads are guaranteed to read at least the state present at the time the request was processed, or a later state.
Summary
-
rocksdb reads may be done synchronously (in
call) or asynchronously (in theFuture), depending on the context; -
rocksdb writes must be done synchronously (in
call)
In-memory data structures
At a high level, the in-memory data structures store a collection of chains, each rooted at the highest finalized block. Each chain consists of a map from heights to blocks. Chains are stored using an ordered map from cumulative work to chains, so that the map ordering is the ordering of worst to best chains.
The Chain type
The Chain type represents a chain of blocks. Each block represents an
incremental state update, and the Chain type caches the cumulative state
update from its root to its tip.
The Chain type is used to represent the non-finalized portion of a complete
chain of blocks rooted at the genesis block. The parent block of the root of
a Chain is the tip of the finalized portion of the chain. As an exception, the finalized
portion of the chain is initially empty, until the genesis block has been finalized.
The Chain type supports several operations to manipulate chains, push,
pop_root, and fork. push is the most fundamental operation and handles
contextual validation of chains as they are extended. pop_root is provided
for finalization, and is how we move blocks from the non-finalized portion of
the state to the finalized portion. fork on the other hand handles creating
new chains for push when new blocks arrive whose parent isn’t a tip of an
existing chain.
Note: The Chain type’s API is only designed to handle non-finalized
data. The genesis block and all pre canopy blocks are always considered to
be finalized blocks and should not be handled via the Chain type through
CommitBlock. They should instead be committed directly to the finalized
state with CommitFinalizedBlock. This is particularly important with the
genesis block since the Chain will panic if used while the finalized state
is completely empty.
The Chain type is defined by the following struct and API:
#![allow(unused)]
fn main() {
#[derive(Debug, Clone)]
pub struct Chain {
// The function `eq_internal_state` must be updated every time a field is added to [`Chain`].
/// The configured network for this chain.
network: Network,
/// The contextually valid blocks which form this non-finalized partial chain, in height order.
pub(crate) blocks: BTreeMap<block::Height, ContextuallyValidBlock>,
/// An index of block heights for each block hash in `blocks`.
pub height_by_hash: HashMap<block::Hash, block::Height>,
/// An index of [`TransactionLocation`]s for each transaction hash in `blocks`.
pub tx_loc_by_hash: HashMap<transaction::Hash, TransactionLocation>,
/// The [`transparent::Utxo`]s created by `blocks`.
///
/// Note that these UTXOs may not be unspent.
/// Outputs can be spent by later transactions or blocks in the chain.
//
// TODO: replace OutPoint with OutputLocation?
pub(crate) created_utxos: HashMap<transparent::OutPoint, transparent::OrderedUtxo>,
/// The [`transparent::OutPoint`]s spent by `blocks`,
/// including those created by earlier transactions or blocks in the chain.
pub(crate) spent_utxos: HashSet<transparent::OutPoint>,
/// The Sprout note commitment tree of the tip of this [`Chain`],
/// including all finalized notes, and the non-finalized notes in this chain.
pub(super) sprout_note_commitment_tree: sprout::tree::NoteCommitmentTree,
/// The Sprout note commitment tree for each anchor.
/// This is required for interstitial states.
pub(crate) sprout_trees_by_anchor:
HashMap<sprout::tree::Root, sprout::tree::NoteCommitmentTree>,
/// The Sapling note commitment tree of the tip of this [`Chain`],
/// including all finalized notes, and the non-finalized notes in this chain.
pub(super) sapling_note_commitment_tree: sapling::tree::NoteCommitmentTree,
/// The Sapling note commitment tree for each height.
pub(crate) sapling_trees_by_height: BTreeMap<block::Height, sapling::tree::NoteCommitmentTree>,
/// The Orchard note commitment tree of the tip of this [`Chain`],
/// including all finalized notes, and the non-finalized notes in this chain.
pub(super) orchard_note_commitment_tree: orchard::tree::NoteCommitmentTree,
/// The Orchard note commitment tree for each height.
pub(crate) orchard_trees_by_height: BTreeMap<block::Height, orchard::tree::NoteCommitmentTree>,
/// The ZIP-221 history tree of the tip of this [`Chain`],
/// including all finalized blocks, and the non-finalized `blocks` in this chain.
pub(crate) history_tree: HistoryTree,
/// The Sprout anchors created by `blocks`.
pub(crate) sprout_anchors: MultiSet<sprout::tree::Root>,
/// The Sprout anchors created by each block in `blocks`.
pub(crate) sprout_anchors_by_height: BTreeMap<block::Height, sprout::tree::Root>,
/// The Sapling anchors created by `blocks`.
pub(crate) sapling_anchors: MultiSet<sapling::tree::Root>,
/// The Sapling anchors created by each block in `blocks`.
pub(crate) sapling_anchors_by_height: BTreeMap<block::Height, sapling::tree::Root>,
/// The Orchard anchors created by `blocks`.
pub(crate) orchard_anchors: MultiSet<orchard::tree::Root>,
/// The Orchard anchors created by each block in `blocks`.
pub(crate) orchard_anchors_by_height: BTreeMap<block::Height, orchard::tree::Root>,
/// The Sprout nullifiers revealed by `blocks`.
pub(super) sprout_nullifiers: HashSet<sprout::Nullifier>,
/// The Sapling nullifiers revealed by `blocks`.
pub(super) sapling_nullifiers: HashSet<sapling::Nullifier>,
/// The Orchard nullifiers revealed by `blocks`.
pub(super) orchard_nullifiers: HashSet<orchard::Nullifier>,
/// Partial transparent address index data from `blocks`.
pub(super) partial_transparent_transfers: HashMap<transparent::Address, TransparentTransfers>,
/// The cumulative work represented by `blocks`.
///
/// Since the best chain is determined by the largest cumulative work,
/// the work represented by finalized blocks can be ignored,
/// because they are common to all non-finalized chains.
pub(super) partial_cumulative_work: PartialCumulativeWork,
/// The chain value pool balances of the tip of this [`Chain`],
/// including the block value pool changes from all finalized blocks,
/// and the non-finalized blocks in this chain.
///
/// When a new chain is created from the finalized tip,
/// it is initialized with the finalized tip chain value pool balances.
pub(crate) chain_value_pools: ValueBalance<NonNegative>,
}
}
pub fn push(&mut self, block: Arc<Block>)
Push a block into a chain as the new tip
-
Update cumulative data members
- Add the block’s hash to
height_by_hash - Add work to
self.partial_cumulative_work - For each
transactioninblock- Add key:
transaction.hashand value:(height, tx_index)totx_loc_by_hash - Add created utxos to
self.created_utxos - Add spent utxos to
self.spent_utxos - Add nullifiers to the appropriate
self.<version>_nullifiers
- Add key:
- Add the block’s hash to
-
Add block to
self.blocks
pub fn pop_root(&mut self) -> Arc<Block>
Remove the lowest height block of the non-finalized portion of a chain.
-
Remove the lowest height block from
self.blocks -
Update cumulative data members
- Remove the block’s hash from
self.height_by_hash - Subtract work from
self.partial_cumulative_work - For each
transactioninblock- Remove
transaction.hashfromtx_loc_by_hash - Remove created utxos from
self.created_utxos - Remove spent utxos from
self.spent_utxos - Remove the nullifiers from the appropriate
self.<version>_nullifiers
- Remove
- Remove the block’s hash from
-
Return the block
pub fn fork(&self, new_tip: block::Hash) -> Option<Self>
Fork a chain at the block with the given hash, if it is part of this chain.
-
If
selfdoes not containnew_tipreturnNone -
Clone self as
forked -
While the tip of
forkedis not equal tonew_tip- call
forked.pop_tip()and discard the old tip
- call
-
Return
forked
fn pop_tip(&mut self)
Remove the highest height block of the non-finalized portion of a chain.
-
Remove the highest height
blockfromself.blocks -
Update cumulative data members
- Remove the corresponding hash from
self.height_by_hash - Subtract work from
self.partial_cumulative_work - for each
transactioninblock- remove
transaction.hashfromtx_loc_by_hash - Remove created utxos from
self.created_utxos - Remove spent utxos from
self.spent_utxos - Remove the nullifiers from the appropriate
self.<version>_nullifiers
- remove
- Remove the corresponding hash from
Ord
The Chain type implements Ord for reorganizing chains. First chains are
compared by their partial_cumulative_work. Ties are then broken by
comparing block::Hashes of the tips of each chain. (This tie-breaker means
that all Chains in the NonFinalizedState must have at least one block.)
Note: Unlike zcashd, Zebra does not use block arrival times as a
tie-breaker for the best tip. Since Zebra downloads blocks in parallel,
download times are not guaranteed to be unique. Using the block::Hash
provides a consistent tip order. (As a side-effect, the tip order is also
consistent after a node restart, and between nodes.)
Default
The Chain type implements Default for constructing new chains whose
parent block is the tip of the finalized state. This implementation should be
handled by #[derive(Default)].
- initialise cumulative data members
- Construct an empty
self.blocks,height_by_hash,tx_loc_by_hash,self.created_utxos,self.spent_utxos,self.<version>_anchors,self.<version>_nullifiers - Zero
self.partial_cumulative_work
- Construct an empty
Note: The ChainState can be empty after a restart, because the
non-finalized state is empty.
NonFinalizedState Type
The NonFinalizedState type represents the set of all non-finalized state.
It consists of a set of non-finalized but verified chains and a set of
unverified blocks which are waiting for the full context needed to verify
them to become available.
NonFinalizedState is defined by the following structure and API:
#![allow(unused)]
fn main() {
/// The state of the chains in memory, including queued blocks.
#[derive(Debug, Default)]
pub struct NonFinalizedState {
/// Verified, non-finalized chains.
chain_set: BTreeSet<Chain>,
/// Blocks awaiting their parent blocks for contextual verification.
contextual_queue: QueuedBlocks,
}
}
pub fn finalize(&mut self) -> Arc<Block>
Finalize the lowest height block in the non-finalized portion of the best chain and updates all side chains to match.
-
Extract the best chain from
self.chain_setintobest_chain -
Extract the rest of the chains into a
side_chainstemporary variable, so they can be mutated -
Remove the lowest height block from the best chain with
let finalized_block = best_chain.pop_root(); -
Add
best_chainback toself.chain_setifbest_chainis not empty -
For each remaining
chaininside_chains- remove the lowest height block from
chain - If that block is equal to
finalized_blockandchainis not empty addchainback toself.chain_set - Else, drop
chain
- remove the lowest height block from
-
Return
finalized_block
fn commit_block(&mut self, block: Arc<Block>)
Commit block to the non-finalized state.
-
If the block is a pre-Canopy block, or the canopy activation block, panic.
-
If any chains tip hash equal
block.header.previous_block_hashremove that chain fromself.chain_set -
Else Find the first chain that contains
block.parentand fork it withblock.parentas the new tiplet fork = self.chain_set.iter().find_map(|chain| chain.fork(block.parent));
-
Else panic, this should be unreachable because
commit_blockis only called whenblockis ready to be committed. -
Push
blockintoparent_chain -
Insert
parent_chainintoself.chain_set
pub(super) fn commit_new_chain(&mut self, block: Arc<Block>)
Construct a new chain starting with block.
-
Construct a new empty chain
-
pushblockinto that new chain -
Insert the new chain into
self.chain_set
The QueuedBlocks type
The queued blocks type represents the non-finalized blocks that were committed before their parent blocks were. It is responsible for tracking which blocks are queued by their parent so they can be committed immediately after the parent is committed. It also tracks blocks by their height so they can be discarded if they ever end up below the reorg limit.
NonFinalizedState is defined by the following structure and API:
#![allow(unused)]
fn main() {
/// A queue of blocks, awaiting the arrival of parent blocks.
#[derive(Debug, Default)]
struct QueuedBlocks {
/// Blocks awaiting their parent blocks for contextual verification.
blocks: HashMap<block::Hash, QueuedBlock>,
/// Hashes from `queued_blocks`, indexed by parent hash.
by_parent: HashMap<block::Hash, Vec<block::Hash>>,
/// Hashes from `queued_blocks`, indexed by block height.
by_height: BTreeMap<block::Height, Vec<block::Hash>>,
}
}
pub fn queue(&mut self, new: QueuedBlock)
Add a block to the queue of blocks waiting for their requisite context to become available.
-
extract the
parent_hash,new_hash, andnew_heightfromnew.block -
Add
newtoself.blocksusingnew_hashas the key -
Add
new_hashto the set of hashes inself.by_parent.entry(parent_hash).or_default() -
Add
new_hashto the set of hashes inself.by_height.entry(new_height).or_default()
pub fn dequeue_children(&mut self, parent: block::Hash) -> Vec<QueuedBlock>
Dequeue the set of blocks waiting on parent.
-
Remove the set of hashes waiting on
parentfromself.by_parent -
Remove and collect each block in that set of hashes from
self.blocksasqueued_children -
For each
blockinqueued_childrenremove the associatedblock.hashfromself.by_height -
Return
queued_children
pub fn prune_by_height(&mut self, finalized_height: block::Height)
Prune all queued blocks whose height are less than or equal to
finalized_height.
-
Split the
by_heightlist at the finalized height, removing all heights that are belowfinalized_height -
for each hash in the removed values of
by_height- remove the corresponding block from
self.blocks - remove the block’s hash from the list of blocks waiting on
block.header.previous_block_hashfromself.by_parent
- remove the corresponding block from
Summary
-
Chainrepresents the non-finalized portion of a single chain -
NonFinalizedStaterepresents the non-finalized portion of all chains -
QueuedBlocksrepresents all unverified blocks that are waiting for context to be available.
The state service uses the following entry points:
-
commit_blockwhen it receives new blocks. -
finalizeto prevent chains inNonFinalizedStatefrom growing beyond the reorg limit. -
FinalizedState.queue_and_commit_finalized_blocks on the blocks returned by
finalize, to commit those finalized blocks to disk.
Committing non-finalized blocks
New non-finalized blocks are committed as follows:
pub(super) fn queue_and_commit_non_finalized_blocks(&mut self, new: Arc<Block>) -> tokio::sync::oneshot::Receiver<block::Hash>
-
If a duplicate block hash exists in a non-finalized chain, or the finalized chain, it has already been successfully verified:
- create a new oneshot channel
- immediately send
Err(DuplicateBlockHash)drop the sender - return the receiver
-
If a duplicate block hash exists in the queue:
- Find the
QueuedBlockfor that existing duplicate block - create a new channel for the new request
- replace the old sender in
queued_blockwith the new sender - send
Err(DuplicateBlockHash)through the old sender channel - continue to use the new receiver
- Find the
-
Else create a
QueuedBlockforblock:- Create a
tokio::sync::oneshotchannel - Use that channel to create a
QueuedBlockforblock - Add
blocktoself.queued_blocks - continue to use the new receiver
- Create a
-
If
block.header.previous_block_hashis not present in the finalized or non-finalized state:- Return the receiver for the block’s channel
-
Else iteratively attempt to process queued blocks by their parent hash starting with
block.header.previous_block_hash -
While there are recently committed parent hashes to process
- Dequeue all blocks waiting on
parentwithlet queued_children = self.queued_blocks.dequeue_children(parent); - for each queued
block- Run contextual validation on
block- contextual validation should check that the block height is equal to the previous block height plus 1. This check will reject blocks with invalid heights.
- If the block fails contextual validation send the result to the associated channel
- Else if the block’s previous hash is the finalized tip add to the
non-finalized state with
self.mem.commit_new_chain(block) - Else add the new block to an existing non-finalized chain or new fork
with
self.mem.commit_block(block); - Send
Ok(hash)over the associated channel to indicate the block was successfully committed - Add
block.hashto the set of recently committed parent hashes to process
- Run contextual validation on
- Dequeue all blocks waiting on
-
While the length of the non-finalized portion of the best chain is greater than the reorg limit
- Remove the lowest height block from the non-finalized state with
self.mem.finalize(); - Commit that block to the finalized state with
self.disk.commit_finalized_direct(finalized);
- Remove the lowest height block from the non-finalized state with
-
Prune orphaned blocks from
self.queued_blockswithself.queued_blocks.prune_by_height(finalized_height); -
Return the receiver for the block’s channel
rocksdb data structures
The current database format is documented in Upgrading the State Database.
Committing finalized blocks
If the parent block is not committed, add the block to an internal queue for future processing. Otherwise, commit the block described below, then commit any queued children. (Although the checkpointer generates verified blocks in order when it completes a checkpoint, the blocks are committed in the response futures, so they may arrive out of order).
Committing a block to the rocksdb state should be implemented as a wrapper around
a function also called by Request::CommitBlock,
which should:
pub(super) fn queue_and_commit_finalized_blocks(&mut self, queued_block: QueuedBlock)
- Obtain the highest entry of
hash_by_heightas(old_height, old_tip). Check thatblock’s parent hash isold_tipand its height isold_height+1, or panic. This check is performed as defense-in-depth to prevent database corruption, but it is the caller’s responsibility (e.g. the zebra-state service’s responsibility) to commit finalized blocks in order.
The genesis block does not have a parent block. For genesis blocks,
check that block’s parent hash is null (all zeroes) and its height is 0.
-
Insert the block and transaction data into the relevant column families.
-
If the block is a genesis block, skip any transaction updates.
(Due to a bug in zcashd, genesis block anchors and transactions are ignored during validation.)
-
Update the block anchors, history tree, and chain value pools.
-
Iterate over the enumerated transactions in the block. For each transaction, update the relevant column families.
Note: The Sprout and Sapling anchors are the roots of the Sprout and
Sapling note commitment trees that have already been calculated for the last
transaction(s) in the block that have JoinSplits in the Sprout case and/or
Spend/Output descriptions in the Sapling case. These should be passed as
fields in the Commit*Block requests.
Due to the coinbase maturity rules, the Sprout root is the empty root for the first 100 blocks. (These rules are already implemented in contextual validation and the anchor calculations.)
Hypothetically, if Sapling were activated from genesis, the specification requires
a Sapling anchor, but zcashd would ignore that anchor.
These updates can be performed in a batch or without necessarily iterating over all transactions, if the data is available by other means; they’re specified this way for clarity.
Accessing previous blocks for contextual validation
The state service performs contextual validation of blocks received via the
CommitBlock request. Since CommitBlock is synchronous, contextual validation
must also be performed synchronously.
The relevant chain for a block starts at its previous block, and follows the chain of previous blocks back to the genesis block.
Relevant chain iterator
The relevant chain can be retrieved from the state service as follows:
- if the previous block is the finalized tip:
- get recent blocks from the finalized state
- if the previous block is in the non-finalized state:
- get recent blocks from the relevant chain, then
- get recent blocks from the finalized state, if required
The relevant chain can start at any non-finalized block, or at the finalized tip.
Relevant chain implementation
The relevant chain is implemented as a StateService iterator, which returns
Arc<Block>s.
The chain iterator implements ExactSizeIterator, so Zebra can efficiently
assert that the relevant chain contains enough blocks to perform each contextual
validation check.
#![allow(unused)]
fn main() {
impl StateService {
/// Return an iterator over the relevant chain of the block identified by
/// `hash`.
///
/// The block identified by `hash` is included in the chain of blocks yielded
/// by the iterator.
pub fn chain(&self, hash: block::Hash) -> Iter<'_> { ... }
}
impl Iterator for Iter<'_> {
type Item = Arc<Block>;
...
}
impl ExactSizeIterator for Iter<'_> { ... }
impl FusedIterator for Iter<'_> {}
}
For further details, see PR 1271.
Request / Response API
The state API is provided by a pair of Request/Response enums. Each
Request variant corresponds to particular Response variants, and it’s
fine (and encouraged) for caller code to unwrap the expected variants with
unreachable! on the unexpected variants. This is slightly inconvenient but
it means that we have a unified state interface with unified backpressure.
This API includes both write and read calls. Spotting Commit requests in
code review should not be a problem, but in the future, if we need to
restrict access to write calls, we could implement a wrapper service that
rejects these, and export “read” and “write” frontends to the same inner service.
Request::CommitBlock
#![allow(unused)]
fn main() {
CommitBlock {
block: Arc<Block>,
sprout_anchor: sprout::tree::Root,
sapling_anchor: sapling::tree::Root,
}
}
Performs contextual validation of the given block, committing it to the state
if successful. Returns Response::Added(block::Hash) with the hash of
the newly committed block or an error.
Request::CommitFinalizedBlock
#![allow(unused)]
fn main() {
CommitFinalizedBlock {
block: Arc<Block>,
sprout_anchor: sprout::tree::Root,
sapling_anchor: sapling::tree::Root,
}
}
Commits a finalized block to the rocksdb state, skipping contextual validation.
This is exposed for use in checkpointing, which produces in-order finalized
blocks. Returns Response::Added(block::Hash) with the hash of the
committed block if successful.
Request::Depth(block::Hash)
Computes the depth in the best chain of the block identified by the given hash, returning
Response::Depth(Some(depth))if the block is in the best chain;Response::Depth(None)otherwise.
Implemented by querying:
- (non-finalized) the
height_by_hashmap in the best chain, and - (finalized) the
height_by_hashtree
Request::Tip
Returns Response::Tip(block::Hash) with the current best chain tip.
Implemented by querying:
- (non-finalized) the highest height block in the best chain
- (finalized) the highest height block in the
hash_by_heighttree, if thenon-finalizedstate is empty
Request::BlockLocator
Returns Response::BlockLocator(Vec<block::Hash>) with hashes starting from
the current chain tip and reaching backwards towards the genesis block. The
first hash is the best chain tip. The last hash is the tip of the finalized
portion of the state. If the finalized and non-finalized states are both
empty, the block locator is also empty.
This can be used by the sync component to request hashes of subsequent blocks.
Implemented by querying:
- (non-finalized) the
hash_by_heightmap in the best chain - (finalized) the
hash_by_heighttree.
Request::Transaction(transaction::Hash)
Returns
-
Response::Transaction(Some(Transaction))if the transaction identified by the given hash is contained in the state; -
Response::Transaction(None)if the transaction identified by the given hash is not contained in the state.
Implemented by querying:
- (non-finalized) the
tx_loc_by_hashmap (to get the block that contains the transaction) of each chain starting with the best chain, and then find block that chain’sblocks(to get the block containing the transaction data) - (finalized) the
tx_loc_by_hashtree (to get the block that contains the transaction) and thenblock_header_by_heighttree (to get the block containing the transaction data), if the transaction is not in any non-finalized chain
Request::Block(block::Hash)
Returns
-
Response::Block(Some(Arc<Block>))if the block identified by the given hash is contained in the state; -
Response::Block(None)if the block identified by the given hash is not contained in the state;
Implemented by querying:
- (non-finalized) the
height_by_hashof each chain starting with the best chain, then find block that chain’sblocks(to get the block data) - (finalized) the
height_by_hashtree (to get the block height) and then theblock_header_by_heighttree (to get the block data), if the block is not in any non-finalized chain
Request::AwaitSpendableUtxo { outpoint: OutPoint, spend_height: Height, spend_restriction: SpendRestriction }
Returns
Response::SpendableUtxo(transparent::Output)
Implemented by querying:
- (non-finalized) if any
ChainscontainOutPointin theircreated_utxos, return theUtxoforOutPoint; - (finalized) else if
OutPointis inutxos_by_outpoint, return theUtxoforOutPoint; - else wait for
OutPointto be created as described in RFC0004;
Then validating:
- check the transparent coinbase spend restrictions specified in RFC0004;
- if the restrictions are satisfied, return the response;
- if the spend is invalid, drop the request (and the caller will time out).
Drawbacks
-
Restarts can cause
zebradto redownload up to the last one hundred blocks it verified in the best chain, and potentially some recent side-chain blocks. -
The service interface puts some extra responsibility on callers to ensure it is used correctly and does not verify the usage is correct at compile time.
-
the service API is verbose and requires manually unwrapping enums
-
We do not handle reorgs the same way
zcashddoes, and could in theory need to delete our entire on disk state and resync the chain in some pathological reorg cases. -
testnet rollbacks are infrequent, but possible, due to bugs in testnet releases. Each testnet rollback will require additional state service code.
Contextual difficulty validation
Status (Zebra 7.0.0): Implemented.
Code: difficulty calculations, contextual checks, and upgrade parameters.
- The 17-block window and 75-second post-Blossom spacing apply only before NU7.
- From NU7, ZIP 218 sets a 102-block window and 25-second spacing.
- The median span remains 11 blocks; difficulty context grows from 28 to 113.
- Bounded vectors and
pow_adjustment_block_spanreplace fixed-size arrays. - Short histories are supported; quotient/remainder sums avoid overflow.
- Regtest uses fixed difficulty.
- The rollback window is 1,000 blocks, not 100.
- Testnet’s minimum-difficulty gap changes from 6 to 18 spacings at NU7, retaining 450 seconds after Blossom.
- The hash conversion uses little-endian raw bytes, per the difficulty rule.
- Feature Name: contextual_difficulty_validation
- Start Date: 2020-11-02
- Design PR: (not recorded in the original RFC)
- Zebra Issue: ZcashFoundation/zebra#1036
Summary
Zcash nodes use a Proof of Work algorithm to reach consensus on the best chain. Valid blocks must reach a difficulty threshold, which is adjusted after every block. The difficulty adjustment calculations depend on the difficulties and times of recent blocks. So Zebra performs contextual validation RFC2 of difficulty adjustments as part of committing blocks to the state.
Motivation
The Zcash block difficulty adjustment is one of the core Zcash consensus rules. Zebra must implement this consensus rule to make sure that its cached chain state is consistent with the consensus of Zcash nodes.
Difficulty adjustment is also a significant part of Zcash’s security guarantees. It ensures that the network continues to resist takeover attacks, even as the number of Zcash miners grows.
Difficulty adjustment also ensures that blocks are regularly spaced, which allows users to create and finalise transactions with short, consistent delays. These predictable delays contribute to Zcash’s usability.
Definitions
Difficulty:
-
hash difficulty: An arbitrary ranking of blocks, based on their hashes. Defined as the hash of the block, interpreted as a big-endian 256-bit number. Numerically smaller difficulties are harder to generate.
-
difficulty threshold: The easiest valid hash difficulty for a block. Numerically lower thresholds are harder to satisfy.
-
difficulty filter: A block passes the difficulty filter if the hash difficulty is less than or equal to the difficulty threshold (based on the block’s difficulty field).
-
block work: The approximate amount of work required for a miner to generate a block hash that passes the difficulty filter. The number of block header attempts and the mining time are proportional to the work value. Numerically higher work values represent longer processing times.
-
averaging window: The 17 most recent blocks in the relevant chain.
-
median block span: The 11 most recent blocks from a chosen tip, typically the relevant tip.
-
target spacing: 150 seconds per block before Blossom activation, 75 seconds per block from Blossom activation onwards.
-
adjusted difficulty: After each block is mined, the difficulty threshold of the next block is adjusted, to keep the block gap close to the target spacing.
-
mean target difficulty: The arithmetic mean of the difficulty thresholds of the blocks in the averaging window.
-
median timespan: The average number of seconds taken to generate the blocks in the averaging window. Calculated using the difference of median block spans in and after the averaging window, then damped and bounded.
-
target timespan: The target spacing for an averaging window’s worth of blocks.
Consensus:
-
consensus rule: A protocol rule which all nodes must apply consistently, so they can converge on the same chain fork.
-
structural/semantic/contextual verification: as defined in RFC2.
State:
-
block chain: A sequence of valid blocks linked by inclusion of the previous block hash in the subsequent block. Chains are rooted at the genesis block and extend to a tip.
-
relevant chain: The relevant chain for a block starts at the previous block, and extends back to genesis.
-
relevant tip: The tip of the relevant chain.
-
non-finalized state: State data corresponding to blocks above the reorg limit. This data can change in the event of a chain reorg.
-
finalized state: State data corresponding to blocks below the reorg limit. This data cannot change in the event of a chain reorg.
-
non-finalized tips: The highest blocks in each non-finalized chain. These tips might be at different heights.
-
finalized tip: The highest block in the finalized state. The tip of the best chain is usually 100 blocks (the reorg limit) above the finalized tip. But it can be lower during the initial sync, and after a chain reorganization, if the new best chain is at a lower height.
Guide-level explanation
Zcash’s difficulty consensus rules are similar to Bitcoin.
Each block contains a difficulty threshold in its header. The hash of the
block header must be less than this difficulty threshold. (When interpreted
as a 256-bit integer in big-endian byte order.) This context-free semantic
verification check is performed by the BlockVerifier.
After each block, the difficulty threshold is adjusted so that the block gap is close to the target spacing. On average, harder blocks take longer to mine, and easier blocks take less time.
The adjusted difficulty for the next block is calculated using the difficulty thresholds and times of recent blocks. Zcash uses the most recent 28 blocks in the relevant chain in its difficulty adjustment calculations.
The difficulty adjustment calculations adjust the mean target difficulty, based on the difference between the median timespan and the target timespan. If the median timespan is less than the target timespan, the next block is harder to mine.
The StateService calculates the adjusted difficulty using the context from the
relevant chain. The difficulty contextual verification check ensures that the
difficulty threshold of the next block is equal to the adjusted difficulty
for its relevant chain.
State service interface changes
Contextual validation accesses recent blocks. So we modify the internal state service interface to provide an abstraction for accessing recent blocks.
The relevant chain
The relevant chain consists of the ancestors of a block, starting with its parent block, and extending back to the genesis block.
In Zebra, recent blocks are part of the non-finalized state, which can contain multiple chains. Past the reorganization limit, Zebra commits a single chain to the finalized state.
The relevant chain can start at any block in the non-finalized state, or at the finalized tip. See RFC5 for details.
Contextual validation design
Contextual validation is performed synchronously by the state service, as soon as the state has:
- received the semantically valid next block (via
CommitBlock), and - committed the previous block.
The difficulty adjustment check calculates the correct adjusted difficulty
threshold value for a candidate block, and ensures that the block’s
difficulty_threshold field is equal to that value.
This check is implemented as follows:
Difficulty adjustment
The block difficulty threshold is adjusted by scaling the mean target difficulty by the median timespan.
On Testnet, if a long time has elapsed since the previous block, the difficulty adjustment is modified to allow minimum-difficulty blocks.
Mean target difficulty
The mean target difficulty is the arithmetic mean of the difficulty
thresholds of the PoWAveragingWindow (17) most recent blocks in the relevant
chain.
Zcash uses block difficulty thresholds in its difficulty adjustment calculations. (Block hashes are not used for difficulty adjustment.)
Median timespan
The average number of seconds taken to generate the 17 blocks in the averaging window.
The median timespan is calculated by taking the difference of the median times for:
- the relevant tip: the
PoWMedianBlockSpan(11) most recent blocks, and - the 11 blocks after the 17-block
PoWAveragingWindow: that is, blocks 18-28 behind the relevant tip.
The median timespan is damped by the PoWDampingFactor, and bounded by
PoWMaxAdjustDown and PoWMaxAdjustUp.
Test network minimum difficulty blocks
If there is a large gap after a Testnet block, the next block becomes a minimum
difficulty block. Testnet minimum difficulty blocks have their
difficulty_threshold set to the minimum difficulty for Testnet.
Block difficulty threshold
The block difficulty threshold for the next block is calculated by scaling the mean target difficulty by the ratio between the median timespan and the averaging window timespan.
The result of this calculation is limited by ToCompact(PoWLimit(network)), a
per-network minimum block difficulty. This minimum difficulty is also used when
a Testnet block’s time gap exceeds the minimum difficulty gap.
Reference-level explanation
Contextual validation
Contextual validation is implemented in
StateService::check_contextual_validity, which calls a separate function for
each contextual validity check.
In Zebra, contextual validation starts after Canopy activation, so we can assume that the relevant chain contains at least 28 blocks on Mainnet and Testnet. (And panic if this assumption does not hold at runtime.)
Fundamental data types
Zebra is free to implement its difficulty calculations in any way that produces
equivalent results to zcashd and the Zcash specification.
Difficulty
In Zcash block headers, difficulty thresholds are stored as a “compact” nBits
value, which uses a custom 32-bit floating-point encoding. Zebra calls this type
CompactDifficulty.
In Zcash, difficulty threshold calculations are performed using unsigned 256-bit
integers. Rust has no standard u256 type, but there are a number of crates
available which implement the required operations on 256-bit integers. Zebra
abstracts over the chosen u256 implementation using its ExpandedDifficulty
type.
Time
In Zcash, time values are unsigned 32-bit integers. But the difficulty adjustment
calculations include time subtractions which could overflow an unsigned type, so
they are performed using signed 64-bit integers in zcashd.
Zebra parses the header.time field into a DateTime<Utc>. Conveniently, the
chrono::DateTime<_>::timestamp() function returns i64 values. So Zebra can do
its signed time calculations using i64 values internally.
Note: i32 is an unsuitable type for signed time calculations. It is
theoretically possible for the time gap between blocks to be larger than
i32::MAX, because those times are provided by miners. Even if the median time
gap is that large, the bounds and minimum difficulty in Zcash’s difficulty
adjustment algorithm will preserve a reasonable difficulty threshold. So Zebra
must support this edge case.
Consensus-Critical Operations
The order of operations and overflow semantics for 256-bit integers can be consensus-critical.
For example:
- dividing before multiplying discards lower-order bits, but
- multiplying before dividing can cause overflow.
Zebra’s implementation should try to match zcashd’s order of operations and overflow handling as closely as possible.
Difficulty adjustment check
The difficulty adjustment check calculates the correct difficulty threshold
value for a candidate block, and ensures that the block’s
difficulty_threshold field is equal to that value.
Context data type
The difficulty adjustment functions use a context consisting of the difficulties and times from the previous 28 blocks in the relevant chain.
These functions also use the candidate block’s height and network.
To make these functions more ergonomic, we create a AdjustedDifficulty
type, and implement the difficulty adjustment calculations as methods on that
type.
#![allow(unused)]
fn main() {
/// The averaging window for difficulty threshold arithmetic mean calculations.
///
/// `PoWAveragingWindow` in the Zcash specification.
pub const POW_AVERAGING_WINDOW: usize = 17;
/// The median block span for time median calculations.
///
/// `PoWMedianBlockSpan` in the Zcash specification.
pub const POW_MEDIAN_BLOCK_SPAN: usize = 11;
/// Contains the context needed to calculate the adjusted difficulty for a block.
struct AdjustedDifficulty {
candidate_time: DateTime<Utc>,
candidate_height: block::Height,
network: Network,
relevant_difficulty_thresholds: [CompactDifficulty; POW_AVERAGING_WINDOW + POW_MEDIAN_BLOCK_SPAN],
relevant_times: [DateTime<Utc>; POW_AVERAGING_WINDOW + POW_MEDIAN_BLOCK_SPAN],
}
}
We implement some initialiser methods on AdjustedDifficulty for convenience.
We might want to validate downloaded headers in future, so we include a
new_from_header initialiser.
#![allow(unused)]
fn main() {
/// Initialise and return a new `AdjustedDifficulty` using a `candidate_block`,
/// `network`, and a `context`.
///
/// The `context` contains the previous
/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) `difficulty_threshold`s and
/// `time`s from the relevant chain for `candidate_block`, in reverse height
/// order, starting with the previous block.
///
/// Note that the `time`s might not be in reverse chronological order, because
/// block times are supplied by miners.
///
/// Panics:
/// If the `context` contains fewer than 28 items.
pub fn new_from_block<C>(candidate_block: &Block,
network: Network,
context: C)
-> AdjustedDifficulty
where
C: IntoIterator<Item = (CompactDifficulty, DateTime<Utc>)>,
{ ... }
/// Initialise and return a new `AdjustedDifficulty` using a
/// `candidate_header`, `previous_block_height`, `network`, and a `context`.
///
/// Designed for use when validating block headers, where the full block has not
/// been downloaded yet.
///
/// See `new_from_block` for detailed information about the `context`.
///
/// Panics:
/// If the context contains fewer than 28 items.
pub fn new_from_header<C>(candidate_header: &block::Header,
previous_block_height: block::Height,
network: Network,
context: C)
-> AdjustedDifficulty
where
C: IntoIterator<Item = (CompactDifficulty, DateTime<Utc>)>,
{ ... }
}
Memory usage note
Copying CompactDifficulty values into the AdjustedDifficulty struct uses
less memory than borrowing those values. CompactDifficulty values are 32 bits,
but pointers are 64-bit on most modern machines. (And since they all come from
different blocks, we need a pointer to each individual value.)
Borrowing DateTime<Utc> values might use slightly less memory than copying
them - but that depends on the exact way that Rust stores associated types
derived from a generic argument.
In any case, the overall size of each AdjustedDifficulty is only a few
hundred bytes. If it turns up in profiles, we can look at borrowing the block
header data.
Difficulty adjustment check implementation
The difficulty adjustment check ensures that the
candidate_difficulty_threshold is equal to the difficulty_threshold value
calculated using AdjustedDifficulty::adjusted_difficulty_threshold.
We implement this function:
#![allow(unused)]
fn main() {
/// Validate the `difficulty_threshold` from a candidate block's header, based
/// on an `expected_difficulty` for that block.
///
/// Uses `expected_difficulty` to calculate the expected `ToCompact(Threshold())`
/// value, then compares that value to the `difficulty_threshold`. Returns
/// `Ok(())` if the values are equal.
pub fn difficulty_threshold_is_valid(difficulty_threshold: CompactDifficulty,
expected_difficulty: AdjustedDifficulty)
-> Result<(), ValidateContextError> { ... }
}
Mean target difficulty calculation
The mean target difficulty is the arithmetic mean of the difficulty
thresholds of the PoWAveragingWindow (17) most recent blocks in the relevant
chain.
We implement this method on AdjustedDifficulty:
#![allow(unused)]
fn main() {
/// Calculate the arithmetic mean of the averaging window thresholds: the
/// expanded `difficulty_threshold`s from the previous `PoWAveragingWindow` (17)
/// blocks in the relevant chain.
///
/// Implements `MeanTarget` from the Zcash specification.
fn mean_target_difficulty(&self) -> ExpandedDifficulty { ... }
}
Implementation notes
Since the PoWLimits are 2^251 − 1 for Testnet, and 2^243 − 1 for Mainnet,
the sum of these difficulty thresholds will be less than or equal to
(2^251 − 1)*17 = 2^255 + 2^251 - 17. Therefore, this calculation can not
overflow a u256 value. So the function is infallible.
In Zebra, contextual validation starts after Canopy activation, so we can assume
that the relevant chain contains at least 17 blocks. Therefore, the PoWLimit
case of MeanTarget() in the Zcash specification is unreachable.
Median timespan calculation
The median timespan is the difference of the median times for:
- the relevant tip: the
PoWMedianBlockSpan(11) most recent blocks, and - the 11 blocks after the 17-block
PoWAveragingWindow: that is, blocks 18-28 behind the relevant tip.
(The median timespan is known as the ActualTimespan in the Zcash specification,
but this terminology is confusing, because it is a difference of medians, rather
than any “actual” elapsed time.)
Zebra implements the median timespan using the following methods on
AdjustedDifficulty:
#![allow(unused)]
fn main() {
/// Calculate the bounded median timespan. The median timespan is the
/// difference of medians of the timespan times, which are the `time`s from
/// the previous `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the
/// relevant chain.
///
/// Uses the candidate block's `height' and `network` to calculate the
/// `AveragingWindowTimespan` for that block.
///
/// The median timespan is damped by the `PoWDampingFactor`, and bounded by
/// `PoWMaxAdjustDown` and `PoWMaxAdjustUp`.
///
/// Implements `ActualTimespanBounded` from the Zcash specification.
///
/// Note: This calculation only uses `PoWMedianBlockSpan` (11) times at the
/// start and end of the timespan times. timespan times `[11..=16]` are ignored.
fn median_timespan_bounded(&self) -> Duration { ... }
/// Calculate the median timespan. The median timespan is the difference of
/// medians of the timespan times, which are the `time`s from the previous
/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain.
///
/// Implements `ActualTimespan` from the Zcash specification.
///
/// See `median_timespan_bounded` for details.
fn median_timespan(&self) -> Duration { ... }
/// Calculate the median of the `median_block_span_times`: the `time`s from a
/// slice of `PoWMedianBlockSpan` (11) blocks in the relevant chain.
///
/// Implements `MedianTime` from the Zcash specification.
fn median_time(mut median_block_span_times: [DateTime<Utc>; POW_MEDIAN_BLOCK_SPAN])
-> DateTime<Utc> { ... }
}
Zebra implements the AveragingWindowTimespan using the following methods on
NetworkUpgrade:
#![allow(unused)]
fn main() {
impl NetworkUpgrade {
/// Returns the `AveragingWindowTimespan` for the network upgrade.
pub fn averaging_window_timespan(&self) -> Duration { ... }
/// Returns the `AveragingWindowTimespan` for `network` and `height`.
pub fn averaging_window_timespan_for_height(network: Network,
height: block::Height)
-> Duration { ... }
}
}
Implementation notes
In Zebra, contextual validation starts after Canopy activation, so we can assume that the relevant chain contains at least 28 blocks. Therefore:
max(0, height − PoWMedianBlockSpan)in theMedianTime()calculation simplifies toheight − PoWMedianBlockSpan, and- there is always an odd number of blocks in
MedianTime(), so the median is always the exact middle of the sequence.
Therefore, the function is infallible.
Test network minimum difficulty calculation
A block is a Testnet minimum difficulty block if:
- the block is a Testnet block,
- the block’s height is 299188 or greater, and
- the time gap from the previous block is greater than the Testnet minimum difficulty gap, which is 6 times the target spacing for the block’s height. (The target spacing was halved from the Blossom network upgrade onwards.)
The difficulty adjustment is modified for Testnet minimum difficulty blocks as follows:
- the difficulty threshold in the block header is set to the Testnet minimum
difficulty threshold,
ToCompact(PoWLimit(network)).
Since the new difficulty changes the block header, Testnet blocks can only satisfy one of the alternate difficulty adjustment rules:
- if the time gap is less than or equal to the Testnet minimum difficulty gap: the difficulty threshold is calculated using the default difficulty adjustment rule,
- if the time gap is greater than the Testnet minimum difficulty gap: the difficulty threshold is the Testnet minimum difficulty threshold.
See ZIP-208 for details.
Note: some older versions of ZIPs 205 and 208 incorrectly said that:
- the time gap threshold uses an “at least” check (it is strictly greater than),
- the minimum difficulty threshold value was
PoWLimit(it isToCompact(PoWLimit)), - the
difficulty_threshold(nBits) field is not modified in Testnet minimum difficulty blocks (the field is modified), and - the Testnet minimum difficulty value is not used to calculate future difficulty adjustments (the modified value is used in future adjustments).
ZIP 205 and 208 were fixed on 14 November 2020, see ZIP PR 417 and ZIP commit 806076c for details.
Test network minimum difficulty implementation
The Testnet minimum difficulty calculation uses the existing
NetworkUpgrade::minimum_difficulty_spacing_for_height function to calculate the
minimum difficulty gap.
We implement this method on NetworkUpgrade:
#![allow(unused)]
fn main() {
/// Returns true if the gap between `block_time` and `previous_block_time` is
/// greater than the Testnet minimum difficulty time gap. This time gap
/// depends on the `network` and `block_height`.
///
/// Returns false on Mainnet, when `block_height` is less than the minimum
/// difficulty start height, and when the time gap is too small.
///
/// `block_time` can be less than, equal to, or greater than
/// `previous_block_time`, because block times are provided by miners.
///
/// Implements the Testnet minimum difficulty adjustment from ZIPs 205 and 208.
///
/// Spec Note: Some parts of ZIPs 205 and 208 previously specified an incorrect
/// check for the time gap. This function implements the correct "greater than"
/// check.
pub fn is_testnet_min_difficulty_block(
network: Network,
block_height: block::Height,
block_time: DateTime<Utc>,
previous_block_time: DateTime<Utc>,
) -> bool { ... }
}
Implementation notes
In Zcash, the Testnet minimum difficulty rule starts at block 299188, and in Zebra, contextual validation starts after Canopy activation. So we can assume that there is always a previous block.
Therefore, this function is infallible.
Block difficulty threshold calculation
The block difficulty threshold for the next block is calculated by scaling the mean target difficulty by the ratio between the median timespan and the averaging window timespan.
The result of the scaled threshold calculation is limited by
ToCompact(PoWLimit(network)), a per-network minimum block difficulty. This
minimum difficulty is also used when a Testnet block’s time gap exceeds the
minimum difficulty gap. We use the existing
ExpandedDifficulty::target_difficulty_limit function to calculate the value of
ToCompact(PoWLimit(network)).
In Zebra, contextual validation starts after Canopy activation, so the genesis
case of Threshold() in the Zcash specification is unreachable.
Block difficulty threshold implementation
We implement these methods on AdjustedDifficulty:
#![allow(unused)]
fn main() {
/// Calculate the expected `difficulty_threshold` for a candidate block, based
/// on the `candidate_time`, `candidate_height`, `network`, and the
/// `difficulty_threshold`s and `time`s from the previous
/// `PoWAveragingWindow + PoWMedianBlockSpan` (28) blocks in the relevant chain.
///
/// Implements `ThresholdBits` from the Zcash specification, and the Testnet
/// minimum difficulty adjustment from ZIPs 205 and 208.
pub fn expected_difficulty_threshold(&self) -> CompactDifficulty { ... }
/// Calculate the `difficulty_threshold` for a candidate block, based on the
/// `candidate_height`, `network`, and the relevant `difficulty_threshold`s and
/// `time`s.
///
/// See `expected_difficulty_threshold` for details.
///
/// Implements `ThresholdBits` from the Zcash specification. (Which excludes the
/// Testnet minimum difficulty adjustment.)
fn threshold_bits(&self) -> CompactDifficulty { ... }
}
Implementation notes
Since:
- the
PoWLimits are2^251 − 1for Testnet, and2^243 − 1for Mainnet, - the
ActualTimespanBoundedcan be at mostMaxActualTimespan, which isfloor(PoWAveragingWindow * PoWTargetSpacing * (1 + PoWMaxAdjustDown))orfloor(17 * 150 * (1 + 32/100)) = 3366, AveragingWindowTimespanis at most17 * 150 = 2250, andMeanTargetis at mostPoWLimit, …
The maximum scaled value inside the Threshold() calculation is:
floor(PoWLimit / 2250) * 3366, which equalsfloor((2^251 − 1) / 2250) * 3366, which equals(2^251 − 1) * 132/100,- which is less than
2^252.
Therefore, this calculation can not overflow a u256 value. (And even if it did
overflow, it would be constrained to a valid value by the PoWLimit minimum.)
Note that the multiplication by ActualTimespanBounded must happen after the
division by AveragingWindowTimespan. Performing the multiplication first
could overflow.
If implemented in this way, the function is infallible.
zcashd truncates the MeanTarget after the mean calculation, and
after dividing by AveragingWindowTimespan. But as long as there is no overflow,
this is equivalent to the single truncation of the final result in the Zcash
specification. However, Zebra should follow the order of operations in zcashd,
and use repeated divisions, because that can’t overflow. See the relevant
comment in the zcashd source code.
Module Structure
The structs and functions in this RFC are implemented in a new
zebra_state::service::check::difficulty module.
This module has two entry points:
DifficultyAdjustment::new_from_blockdifficulty_threshold_is_valid
These entry points are both called from
StateService::check_contextual_validity.
Test Plan
Explain how the feature will be tested, including:
- tests for consensus-critical functionality
- existing test vectors, if available
- Zcash blockchain block test vectors (specify the network upgrade, feature, or block height and network)
- property testing or fuzzing
The tests should cover:
- positive cases: make sure the feature accepts valid inputs
- using block test vectors for each network upgrade provides some coverage of valid inputs
- negative cases: make sure the feature rejects invalid inputs
- make sure there is a test case for each error condition in the code
- if there are lots of potential errors, prioritise:
- consensus-critical errors
- security-critical errors, and
- likely errors
- edge cases: make sure that boundary conditions are correctly handled
Drawbacks
Why should we not do this?
Alternate consensus parameters
Any alternate consensus parameters or regtest mode would have to respect the constraints set by this design.
In particular:
- the
PoWLimitmust be less than or equal to(2^256 - 1) / PoWAveragingWindow(approximately2^251) to avoid overflow, - the
PoWAveragingWindowandPoWMedianBlockSpanare fixed by function argument types (at least until Rust gets stable const generics), and - the design eliminates a significant number of edge cases by assuming that difficulty adjustments aren’t
validated for the first
PoWAveragingWindow + PoWMedianBlockSpan(28) blocks in the chain.
Rationale and alternatives
Is this design a good basis for later designs or implementations?
The design includes specific methods for a future header-only validation design.
What other designs have been considered and what is the rationale for not choosing them?
A previous version of the RFC did not have the AdjustedDifficulty struct and
methods. That design was easy to misuse, because each function had a complicated
argument list.
What is the impact of not doing this?
Zebra could accept invalid, low-difficulty blocks from arbitrary miners. That would be a security issue.
Prior art
zcashd- the Zcash specification
- Bitcoin
Unresolved questions
-
What parts of the design do you expect to resolve through the implementation of this feature before stabilization?
- Guide-level examples
- Reference-level examples
- Corner case examples
- Testing
-
What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC?
- Monitoring and maintenance
Future possibilities
Reusing the relevant chain API in other contextual checks
The relevant chain iterator can be reused to implement other contextual validation checks.
For example, responding to peer requests for block locators, which means
implementing relevant chain hash queries as a StateService request
Header-only difficulty adjustment validation
Implementing header-only difficulty adjustment validation as a StateService request.
Caching difficulty calculations
Difficulty calculations use u256 could be a bit expensive, particularly if we
get a flood of low-difficulty blocks. To reduce the impact of this kind of DoS,
we could cache the value returned by threshold_bits for each block in the
non-finalized state, and for the finalized tip. This value could be used to
quickly calculate the difficulties for any child blocks of these blocks.
There’s no need to persist this cache, or pre-fill it. (Minimum-difficulty
Testnet blocks don’t call threshold_bits, and some side-chain blocks will
never have a next block.)
This caching is only worth implementing if these calculations show up in zebrad
profiles.
Treestate
Status (Zebra 7.0.0): Implemented.
Code: shielded storage, contextual checks, and block commitments.
- Ironwood adds trees, anchors, and nullifiers in separate column families.
- Ironwood reuses Orchard tree types.
- Finalized Sapling/Orchard/Ironwood trees are stored by height on change.
- Sprout keeps its latest tree and an anchor-to-tree index.
- Heartwood’s activation block has an all-zero reserved commitment.
- Later Heartwood and Canopy blocks commit to chain history, not the Sapling root.
- From NU5, the commitment combines chain history and transaction authorization data.
- Database writes insert or overwrite; no deletion-event tracker forbids re-insertion.
- Feature Name: treestate
- Start Date: 2020-08-31
- Design PR: ZcashFoundation/zebra#983
- Zebra Issue: ZcashFoundation/zebra#958
Summary
To validate blocks involving shielded transactions, we have to check the computed treestate from the included transactions against the block header metadata (for Sapling and Orchard) or previously finalized state (for Sprout). This document describes how we compute and manage that data, assuming a finalized state service as described in the State Updates RFC.
Motivation
Block validation requires checking that the treestate of the block (consisting of the note commitment tree and nullifier set) is consistent with the metadata we have in the block header (the root of the note commitment tree) or previously finalized state (for Sprout).
Definitions
Common Definitions
Many terms used here are defined in the Zcash Protocol Specification
notes: Represents a value bound to a shielded payment address (public key) which is spendable by the recipient who holds the spending key corresponding to a given shielded payment address.
nullifiers: A value that prevents double-spending of a shielded payment.
Revealed by Spend descriptions when its associated Note is spent.
nullifier set: The set of unique Nullifiers revealed by any Transactions
within a Block. Nullifiers are enforced to be unique within a valid block chain
by committing to previous treestates in Spend descriptions, in order to prevent
double-spends.
note commitments: Pedersen commitment to the values consisting a Note. One
should not be able to construct a Note from its commitment.
note commitment tree: An incremental Merkle tree of fixed depth used to
store NoteCommitments that JoinSplit transfers or Spend transfers produce. It
is used to express the existence of value and the capability to spend it. It is
not the job of this tree to protect against double-spending, as it is
append-only: that’s what the Nullifier set is for.
note position: The index of a NoteCommitment at the leafmost layer,
counting leftmost to rightmost. The position in the tree is determined by the
order of transactions in the block.
root: The layer 0 node of a Merkle tree.
anchor: A Merkle tree root of a NoteCommitment tree. It uniquely
identifies a NoteCommitment tree state given the assumed security properties
of the Merkle tree’s hash function. Since the Nullifier set is always updated
together with the NoteCommitment tree, this also identifies a particular state
of the associated Nullifier set.
Sprout Definitions
joinsplit: A shielded transfer that can spend Sprout Notes and transparent
value, and create new Sprout Notes and transparent value, in one Groth16 proof
statement.
Sapling Definitions
spend descriptions: A shielded Sapling transfer that spends a Note. Includes
an anchor of some previous Block’s NoteCommitment tree.
output descriptions: A shielded Sapling transfer that creates a
Note. Includes the u-coordinate of the NoteCommitment itself.
Orchard Definitions
action descriptions: A shielded Orchard transfer that spends and/or creates a
Note. Does not include an anchor, because that is encoded once in the
anchorOrchard field of a V5 Transaction.
Guide-level explanation
Common Processing for All Protocols
As Blocks are validated, the NoteCommitments revealed by all the transactions
within that block are used to construct NoteCommitmentTrees, with the
NoteCommitments aligned in their note positions in the bottom layer of the
Sprout or Sapling tree from the left-most leaf to the right-most in
Transaction order in the Block. So the Sprout NoteCommitments revealed by
the first JoinSplit<Groth16> in a block would take note position 0 in the Sprout
note commitment tree, for example. Once all the transactions in a block are
parsed and the notes for each tree collected in their appropriate positions, the
root of each tree is computed. While the trees are being built, the respective
block nullifier sets are updated in memory as note nullifiers are revealed. If
the rest of the block is validated according to consensus rules, that root is
committed to its own data structure via our state service (Sprout anchors,
Sapling anchors). Sapling block validation includes comparing the specified
FinalSaplingRoot in its block header to the root of the Sapling NoteCommitment
tree that we have just computed to make sure they match.
Sprout Processing
For Sprout, we must compute/update interstitial NoteCommitmentTrees between
JoinSplits that may reference an earlier one’s root as its anchor. If we do
this at the transaction layer, we can iterate through all the JoinSplits and
compute the Sprout NoteCommitmentTree and nullifier set similar to how we do
the Sapling ones as described below, but at each state change (ie,
per-JoinSplit) we note the root and cache it for lookup later. As the
JoinSplits are validated without context, we check for its specified anchor
amongst the interstitial roots we’ve already calculated (according to the spec,
these interstitial roots don’t have to be finalized or the result of an
independently validated JoinSplit, they just must refer to any prior JoinSplit
root in the same transaction). So we only have to wait for our previous root to
be computed via any of our candidates, which in the worst case is waiting for
all of them to be computed for the last JoinSplit. If our JoinSplits defined
root pops out, that JoinSplit passes that check.
Sapling Processing
As the transactions within a block are parsed, Sapling shielded transactions
including Spend descriptions and Output descriptions describe the spending and
creation of Zcash Sapling notes. Spend descriptions specify an anchor, which
references a previous NoteCommitment tree root. This is a previous block’s anchor
as defined in their block header. This is convenient because we can query our state
service for previously finalized Sapling block anchors, and if they are found, then
that consensus check
has been satisfied and the Spend description can be validated independently.
For Sapling, at the block layer, we can iterate over all the transactions in
order and if they have Spends and/or Outputs, we update our Nullifer set for
the block as nullifiers are revealed in Spend descriptions, and update our note
commitment tree as NoteCommitments are revealed in Output descriptions, adding
them as leaves in positions according to their order as they appear transaction
to transaction, output to output, in the block. This can be done independent of
the transaction validations. When the Sapling transactions are all validated,
the NoteCommitmentTree root should be computed: this is the anchor for this
block.
Anchor Validation Across Network Upgrades
For Sapling and Blossom blocks, we need to check that this root matches
the RootHash bytes in this block’s header, as the FinalSaplingRoot. Once all
other consensus and validation checks are done, this will be saved down to our
finalized state to our sapling_anchors set, making it available for lookup by
other Sapling descriptions in future transactions.
In Heartwood and Canopy, the rules for final Sapling roots are modified to support empty blocks by allowing an empty subtree hash instead of requiring the root to match the previous block’s final Sapling root when there are no Sapling transactions.
In NU5, the rules are further extended to include Orchard note commitment trees,
with similar logic applied to the anchorOrchard field in V5 transactions.
Orchard Processing
For Orchard, similar to Sapling, action descriptions can spend and create notes.
The anchor is specified at the transaction level in the anchorOrchard field of
a V5 transaction. The process follows similar steps to Sapling for validation and
inclusion in blocks.
Block Finalization
To finalize the block, the Sprout, Sapling, and Orchard treestates are the ones resulting from the last transaction in the block, and determines the respective anchors that will be associated with this block as we commit it to our finalized state. The nullifiers revealed in the block will be merged with the existing ones in our finalized state (ie, it should strictly grow over time).
State Management
Orchard
- There is a single copy of the latest Orchard Note Commitment Tree for the finalized tip.
- When finalizing a block, the finalized tip is updated with a serialization of the latest Orchard Note Commitment Tree. (The previous tree should be deleted as part of the same database transaction.)
- Each non-finalized chain gets its own copy of the Orchard note commitment tree, cloned from the note commitment tree of the finalized tip or fork root.
- When a block is added to a non-finalized chain tip, the Orchard note commitment tree is updated with the note commitments from that block.
- When a block is rolled back from a non-finalized chain tip, the Orchard tree state is restored to its previous state before the block was added. This involves either keeping a reference to the previous state or recalculating from the fork point.
Sapling
- There is a single copy of the latest Sapling Note Commitment Tree for the finalized tip.
- When finalizing a block, the finalized tip is updated with a serialization of the Sapling Note Commitment Tree. (The previous tree should be deleted as part of the same database transaction.)
- Each non-finalized chain gets its own copy of the Sapling note commitment tree, cloned from the note commitment tree of the finalized tip or fork root.
- When a block is added to a non-finalized chain tip, the Sapling note commitment tree is updated with the note commitments from that block.
- When a block is rolled back from a non-finalized chain tip, the Sapling tree state is restored to its previous state, similar to the Orchard process. This involves either maintaining a history of tree states or recalculating from the fork point.
Sprout
- Every finalized block stores a separate copy of the Sprout note commitment tree (😿), as of that block.
- When finalizing a block, the Sprout note commitment tree for that block is stored in the state. (The trees for previous blocks also remain in the state.)
- Every block in each non-finalized chain gets its own copy of the Sprout note commitment tree. The initial tree is cloned from the note commitment tree of the finalized tip or fork root.
- When a block is added to a non-finalized chain tip, the Sprout note commitment tree is cloned, then updated with the note commitments from that block.
- When a block is rolled back from a non-finalized chain tip, the trees for each block are deleted, along with that block.
We can’t just compute a fresh tree with just the note commitments within a block, we are adding them to the tree referenced by the anchor, but we cannot update that tree with just the anchor, we need the ‘frontier’ nodes and leaves of the incremental merkle tree.
Reference-level explanation
The implementation involves several key components:
-
Incremental Merkle Trees: We use the
incrementalmerkletreecrate to implement the note commitment trees for each shielded pool. -
Nullifier Storage: We maintain nullifier sets in RocksDB to efficiently check for duplicates.
-
Tree State Management:
- For finalized blocks, we store the tree states in RocksDB.
- For non-finalized chains, we keep tree states in memory.
-
Anchor Verification:
- For Sprout: we check anchors against our stored Sprout tree roots.
- For Sapling: we compare the computed root against the block header’s
FinalSaplingRoot. - For Orchard: we validate the
anchorOrchardfield in V5 transactions.
-
Re-insertion Prevention: Our implementation should prevent re-inserts of keys that have been deleted from the database, as this could lead to inconsistencies. The state service tracks deletion events and validates insertion operations accordingly.
Drawbacks
-
Storage Requirements: Storing separate tree states (especially for Sprout) requires significant disk space.
-
Performance Impact: Computing and verifying tree states can be computationally expensive, potentially affecting sync performance.
-
Implementation Complexity: Managing multiple tree states across different protocols adds complexity to the codebase.
-
Fork Handling: Maintaining correct tree states during chain reorganizations requires careful handling.
Rationale and alternatives
We chose this approach because:
-
Protocol Compatibility: Our implementation follows the Zcash protocol specification requirements for handling note commitment trees and anchors.
-
Performance Optimization: By caching tree states, we avoid recomputing them for every validation operation.
-
Memory Efficiency: For non-finalized chains, we only keep necessary tree states in memory.
-
Scalability: The design scales with chain growth by efficiently managing storage requirements.
Alternative approaches considered:
-
Recompute Trees On-Demand: Instead of storing tree states, recompute them when needed. This would save storage but significantly impact performance.
-
Single Tree State: Maintain only the latest tree state and recompute for historical blocks. This would simplify implementation but make historical validation harder.
-
Full History Storage: Store complete tree states for all blocks. This would optimize validation speed but require excessive storage.
Prior art
-
Zcashd: Uses similar concepts but with differences in implementation details, particularly around storage and concurrency.
-
Lightwalletd: Provides a simplified approach to tree state management focused on scanning rather than full validation.
-
Incrementalmerkletree Crate: Our implementation leverages this existing Rust crate for efficient tree management.
Unresolved questions
-
Optimization Opportunities: Are there further optimizations we can make to reduce storage requirements while maintaining performance?
-
Root Storage: Should we store the
Roothash insprout_note_commitment_tree, and use it to look up the complete tree state when needed? -
Re-insertion Prevention: What’s the most efficient approach to prevent re-inserts of deleted keys?
-
Concurrency Model: How do we best handle concurrent access to tree states during parallel validation?
Future possibilities
-
Pruning Strategies: Implement advanced pruning strategies for historical tree states to reduce storage requirements.
-
Parallelization: Further optimize tree state updates for parallel processing.
-
Checkpoint Verification: Use tree states for efficient checkpoint-based verification.
-
Light Client Support: Leverage tree states to support Zebra-based light clients with efficient proof verification.
-
State Storage Optimization: Investigate more efficient serialization formats and storage mechanisms for tree states.
Zebra client
Status (Zebra 7.0.0): Superseded.
Code: zebra-rpc and chain-tip notifications.
- The
zebra-client/zebra-clicrates and private scanning database are absent. - Wallets use external tools: Zallet or lightwalletd.
- Clients submit serialized transactions through
sendrawtransaction, notsendTransaction. - Tip APIs:
LatestChainTipandChainTipChange, notStateNotifications.
- Feature Name: (
zebra-client) - Start Date: (2020-10-14)
- Design PR: ZcashFoundation/zebra#1163
- Zebra Issue: (not recorded in the original RFC)
Summary
The zebra-client crate handles client functionality. Client functionality
is defined as all functionality related to a particular user’s private data,
in contrast to the other full node functionality which handles public chain
state. This includes:
- note and key management;
- transaction generation;
- a client component for
zebradthat handles block chain scanning, with appropriate side-channel protections; - an RPC endpoint for
zebradthat allows access to the client component; - Rust library code that implements basic wallet functionality;
- a
zebra-clibinary that wraps the wallet library and RPC queries in a command-line interface.
Client functionality is restricted to transparent and Sapling shielded transactions; Sprout shielded transactions are not supported. (Users should migrate to Sapling).
Motivation
We want to allow users to efficiently and securely send and receive funds via Zebra. One challenge unique to Zcash is block chain scanning: because shielded transactions reveal no metadata about the sender or receiver, users must scan the block chain for relevant transactions using viewing keys. This means that unlike a transparent blockchain with public transactions, a full node must have online access to viewing keys to scan the chain. This creates the risk of a privacy leak, because the node should not reveal which viewing keys it has access to.
Block chain scanning requires a mechanism that allows users to manage and store key material. This mechanism should also provide basic wallet functionality, so that users can send and receive funds without requiring third-party software.
To protect user privacy, this and all secret-dependent functionality should
be strongly isolated from the rest of the node implementation. Care should be
taken to protect against side channels that could reveal information about
viewing keys. To make this isolation easier, all secret-dependent
functionality is provided only by the zebra-client crate.
Definitions
-
client functionality: all functionality related to a particular user’s private data, in contrast to other full node functionality which handles public chain state.
-
block chain scanning: the process of scanning the block chain for relevant transactions using a viewing key, as described in §4.19 of the protocol specification.
-
viewing key: Sapling shielded addresses support viewing keys, which represent the capability to decrypt transactions, as described in §3.1 and §4.2.2 of the protocol specification.
-
task: In this document, task refers specifically to a Tokio task. In brief, a task is a light weight, non-blocking unit of execution (green thread), similar to a Goroutine or Erlang process. Tasks execute independently and are scheduled co-operatively using explicit yield points. Tasks are executed on the Tokio runtime, which can either be single- or multi-threaded.
Guide-level explanation
There are two main parts of this functionality. The first is a Client
component running as part of zebrad, and the second is a zebra-cli
command-line tool.
The Client component is responsible for blockchain scanning. It maintains
its own distinct sled database, which stores the viewing keys it uses to
scan as well as the results of scanning. When a new block is added to the
chain state, the Client component is notified asynchronously using a
channel. For each Sapling shielded transaction in the block, the component
attempts to perform trial decryption of that transaction’s notes using each
registered viewing key, as described in §4.19. If successful,
decrypted notes are saved to the database.
The PING/REJECT attack demonstrates the importance of
decoupling execution of normal node operations from secret-dependent
operations. Zebra’s network stack already makes it immune to those particular
attacks, because each peer connection is executed in a different task.
However, to eliminate this entire class of vulnerability, we execute the
Client component in its own task, decoupled from the rest of the node
functionality. In fact, each viewing key’s scanning is performed
independently, as described in more detail below, with an analysis of
potential side-channels.
The second part is the zebra-cli command-line tool, which provides basic
wallet functionality. This tool manages spending keys and addresses, and
communicates with the Client component in zebrad to provide basic wallet
functionality. Specifically, zebra-cli uses a distinct RPC endpoint to load
viewing keys into zebrad and to query the results of block chain scanning.
zebra-cli can then use the results of those queries to generate transactions
and submit them to the network using zebrad.
This design upholds the principle of least authority by separating key
material required for spending funds from the key material required for block
chain scanning. This allows compartmentalization. For instance, a user could
in principle run zebrad on a cloud VPS with only their viewing keys and
store their spending keys on a laptop, or a user could run zebrad on a
local machine and store their spending keys in a hardware wallet. Both of
these use cases would require some additional tooling support, but are
possible with this design.
Reference-level explanation
State notifications
We want a way to subscribe to updates from the state system via a channel. For the purposes of this RFC, these changes are in-flight, but in the future, these could be used for a push-based RPC mechanism.
Subscribers can subscribe to all state change notifications as they come in.
Currently the zebra_state::init() method returns a BoxService that allows you to
make requests to the chain state. Instead, we would return a (BoxService, StateNotifications) tuple, where StateNotifications is a new structure initially
defined as:
#![allow(unused)]
fn main() {
#[non_exhaustive]
pub struct StateNotifications {
pub new_blocks: tokio::sync::watch::Receiver<Arc<Block>>,
}
}
Instead of making repeated polling requests to a state service to look for any new blocks, this channel will push new blocks to a consumer as they come in, for the consumer to use or discard at their discretion. This will be used by the client component described below. This will also be needed for gossiping blocks to other peers, as they are validated.
Online client component
This component maintains its own Sled tree. See RFC#0005 for more details on Sled.
We use the following Sled trees:
| Tree | Keys | Values |
|---|---|---|
viewing_keys | IncomingViewingKey | String |
height_by_key | IncomingViewingKey | BE32(height) |
received_set_by_key | IncomingViewingKey | ? |
spend_set_by_key | IncomingViewingKey | ? |
nullifier_map_by_key | IncomingViewingKey | ? |
See https://zips.z.cash/protocol/protocol.pdf#saplingscan
Zcash structures are encoded using ZcashSerialize/ZcashDeserialize.
This component runs inside zebrad. After incoming viewing keys are registered, it holds onto them in order to do blockchain scanning. The component keeps track of where it’s scanned to (TODO: per key?). Runs in its own separate task, in case it crashes, it’s not noticeable, and executes independently (but in the same process) of the normal node operation.
In the case of the client component that needs to do blockchain scanning and trial decryption, every valid block with non-coinbase transactions will need to be checked and its transactions trial-decrypted with registered incoming viewing keys to see if any notes have been received by the key’s owner and if any notes have already been spent elsewhere.
RPC’s
A specific set of privileged RPC endpoints:
- Allows registering of incoming viewing keys with zebrad in order to do blockchain scanning
- Allows querying of the results of that scanning, to get wallet balance, etc
- Not authenticated to start (see ‘Future possibilities’)
- Users can control access by controlling access to the privileged endpoint (ie via a firewall)
Support for sending tx’s via non-privileged RPC endpoints, or via Stolon:
- sendTransaction: once you author a transaction you can gossip it via any Zcash node, not just a specific instance of zebrad
Wallet functionality
- Holds on to your spending keys so you can author transactions
- Uses RPC methods to query the online client component inside zebrad about wallet balances
CLI binary
- zebra-cli talks to the subcomponent running in zebrad
- (can use servo/bincode to communicate with zebrad)
- via the privileged (and possibly the unprivileged) RPC endpoints
- can use cap-std to restrict filesystem and network access for zebra-client. See https://github.com/ZcashFoundation/zebra/issues/2340
- can use the tui crate to render a terminal UI
Task isolation in Tokio
- TODO: fill in
- cooperative multitasking is fine, IF you cooperate
- lots of tasks
Module Structure
zebra-client ( currently and empty stub) zebra-cli (does not exist yet) zebra-rfc? (exists as an empty stub, we way have zebra-cli communicate with zebra-client inside zebrad via an RPC method any/or a private IPC layer)
Test Plan
Drawbacks
Supporting a wallet assumes risk. Effort required to implement wallet functionality.
- need to responsibly handle secret key material;
- currently we only handle public data.
Rationale and alternatives
-
why have a separate RPC endpoint?
- extra endpoints are cheap
- allows segmentation by capability
- alternative is error-prone after-the-fact ACLs like Tor control port filters
-
What is the impact of not doing this?
- We can’t send money with zebra alone.
- rely on third party wallet software to send funds with zebra
- we need to provide basic functionality within zebra’s trust boundary, rather than forcing users to additionally trust 3p software
- there are great 3p wallets, we want to integrate with them, just don’t want to rely on them
-
What about the light client protocol?
- does not address this use case, has different trust model (private lookup, no scanning)
- we want our first client that interacts with zebrad to not have a long startup time, which a lightclient implementation would require
- zebra-cli should be within the same trust and privacy boundary as the zebrad node it is interacting with
- light client protocol as currently implemented requires stack assumptions such as protobufs and a hardcoded lightserver to talk to
-
What about having one database per key?
- easy to reliably delete or backup all data related to a single key
- might use slightly more space/CPU
- slightly harder to delete all the keys
Unresolved questions
- wait to fill this in until doing the detailed writeup.
Future possibilities
-
BlazeSync algorithm for fast syncing, like Zecwallet
-
mandatory sweeps for legacy keys
- blazingly fast wallet startup, to match
zebrad’s blazingly fast sync - generate unified address from a new seed phrase (or one provided by the user)
- user can just backup seed phrase rather than a set of private keys
- handles arbitrary keys from
zcashdand other wallets, even if they weren’t generated from a seed phrase handles Sprout funds withoutzebra-clienthaving to support Sprout balances- startup is incredibly fast
- sweep takes a few minutes to be confirmed
- scanning the entire chain could take hours
- if we know when the seed phrase was created, we can skip millions of blocks during scanning
- sweeps can also be initiated by the user for non-linkability / performance / refresh
- sweeps should handle the “block reward recipient” case where there are a lot of small outputs
- initial release could support mandatory sweeps, and future releases could support legacy keys
- blazingly fast wallet startup, to match
-
split
Clientcomponent into subprocess - this helps somewhat but the benefit is reduced by our preexisting memory safety, thanks to Rust - not meaningful without other isolation (need to restrictzebradfrom accessing viewing keys on disk, etc) - could use cap-std to restrict filesystem and network access for zebra-client. See https://github.com/ZcashFoundation/zebra/issues/2340 - instead of process isolation, maybe you actually want the Light Client Protocol, or something similar? -
hardware wallet integration for
zebra-cli- having
zebra-cliallows us to do this - much higher security ROI than subprocess
- very cool future feature
- having
-
authenticate queries for a particular viewing key by proving knowledge of the viewing key (requires crypto). this could allow public access to the client endpoint
-
Use Unified Addresses only, no legacy addrs.
Version 5 transactions
Status (Zebra 7.0.0): Superseded.
Code: zebra-chain::transaction and serialization.
- Versions 5 and 6 remain supported.
- The wrapper holds
zcash_primitives::transaction::Transaction, not an enum. - Production serialization delegates to
zcash_primitives. - Zebra’s Sapling/Orchard/Ironwood codecs require tests or
proptest-impl. - Version 6 supports Orchard and Ironwood from NU6.3.
- Input/output-presence checks remain in semantic verification, not the parser.
- Feature Name:
v5_transaction - Start Date: 2021-03-11
- Design PR: ZcashFoundation/zebra#1886
- Zebra Issue: ZcashFoundation/zebra#1863
Summary
Network Upgrade number 5 (NU5) introduces a new transaction type (transaction version 5). This document is a proposed design for implementing such a transaction version.
Motivation
The Zebra software wants to be a protocol compatible Zcash implementation. One of the tasks to do this includes the support of the new version 5 transactions that will be implemented in Network Upgrade 5 (NU5).
Definitions
NU5- the 5th Zcash network upgrade, counting from theOverwinterupgrade as upgrade zero.Orchard- a new shielded pool introduced inNU5.Sapling- a new shielded pool introduced in the 1st network upgrade. (Saplingis also the name of that network upgrade, but this RFC is focused on theSaplingshielded pool.)orchard data- Data types needed to support orchard transactions.sapling data- Data types needed to support sapling transactions.orchard transaction version- Transactions that support orchard data. Currently only V5.sapling transaction version- Transactions that support sapling data. Currently V4 and V5 but the data is implemented differently in them.
Guide-level explanation
V5 transactions are described by the protocol in the second table of Transaction Encoding and Consensus.
All of the changes proposed in this document are only to the zebra-chain crate.
To highlight changes most of the document comments from the code snippets in the reference section were removed.
Sapling Changes Overview
V4 and V5 transactions both support sapling, but the underlying data structures are different. So we need to make the sapling data types generic over the V4 and V5 structures.
In V4, anchors are per-spend, but in V5, they are per-transaction. In V5, the shared anchor is only present if there is at least one spend.
For consistency, we also move some fields into the ShieldedData type, and rename some fields and types.
Orchard Additions Overview
V5 transactions are the only ones that will support orchard transactions with Orchard data types.
Orchard uses Halo2Proofs with corresponding signature type changes. Each Orchard Action contains a spend and an output. Placeholder values are substituted for unused spends and outputs.
Other Transaction V5 Changes
V5 transactions split Spends, Outputs, and AuthorizedActions into multiple arrays,
with a single CompactSize count before the first array. We add new
zcash_deserialize_external_count and zcash_serialize_external_count utility functions,
which make it easier to serialize and deserialize these arrays correctly.
The order of some of the fields changed from V4 to V5. For example the lock_time and
expiry_height were moved above the transparent inputs and outputs.
The serialized field order and field splits are in the V5 transaction section in the NU5 spec. (Currently, the V5 spec is on a separate page after the V1-V4 specs.)
Zebra’s structs sometimes use a different order from the spec. We combine fields that occur together, to make it impossible to represent structurally invalid Zcash data.
In general:
- Zebra enums and structs put fields in serialized order.
- Composite structs and emnum variants are ordered based on last data deserialized for the composite.
Reference-level explanation
Sapling Changes
We know by protocol (2nd table of Transaction Encoding and Consensus) that V5 transactions will support sapling data however we also know by protocol that spends (Spend Description Encoding and Consensus, See †) and outputs (Output Description Encoding and Consensus, See †) fields change from V4 to V5.
ShieldedData is currently defined and implemented in zebra-chain/src/transaction/shielded_data.rs. As this is Sapling specific we propose to move this file to zebra-chain/src/sapling/shielded_data.rs.
Changes to V4 Transactions
Here we have the proposed changes for V4 transactions:
- make
sapling_shielded_datause thePerSpendAnchoranchor variant - rename
shielded_datatosapling_shielded_data - move
value_balanceinto thesapling::ShieldedDatatype - order fields based on the last data deserialized for each field
#![allow(unused)]
fn main() {
enum Transaction::V4 {
inputs: Vec<transparent::Input>,
outputs: Vec<transparent::Output>,
lock_time: LockTime,
expiry_height: block::Height,
joinsplit_data: Option<JoinSplitData<Groth16Proof>>,
sapling_shielded_data: Option<sapling::ShieldedData<PerSpendAnchor>>,
}
}
The following types have ZcashSerialize and ZcashDeserialize implementations,
because they can be serialized into a single byte vector:
transparent::Inputtransparent::OutputLockTimeblock::HeightOption<JoinSplitData<Groth16Proof>>
Note: Option<sapling::ShieldedData<PerSpendAnchor>> does not have serialize or deserialize implementations,
because the binding signature is after the joinsplits. Its serialization and deserialization is handled as
part of Transaction::V4.
Anchor Variants
We add an AnchorVariant generic type trait, because V4 transactions have a per-Spend anchor, but V5 transactions have a shared anchor. This trait can be added to sapling/shielded_data.rs:
#![allow(unused)]
fn main() {
struct PerSpendAnchor {}
struct SharedAnchor {}
/// This field is not present in this transaction version.
struct FieldNotPresent;
impl AnchorVariant for PerSpendAnchor {
type Shared = FieldNotPresent;
type PerSpend = sapling::tree::Root;
}
impl AnchorVariant for SharedAnchor {
type Shared = sapling::tree::Root;
type PerSpend = FieldNotPresent;
}
trait AnchorVariant {
type Shared;
type PerSpend;
}
}
Changes to Sapling ShieldedData
We use AnchorVariant in ShieldedData to model the anchor differences between V4 and V5:
- in V4, there is a per-spend anchor
- in V5, there is a shared anchor, which is only present when there are spends
If there are no spends and no outputs:
- in v4, the value_balance is fixed to zero
- in v5, the value balance field is not present
- in both versions, the binding_sig field is not present
#![allow(unused)]
fn main() {
/// ShieldedData ensures that value_balance and binding_sig are only present when
/// there is at least one spend or output.
struct sapling::ShieldedData<AnchorV: AnchorVariant> {
value_balance: Amount,
transfers: sapling::TransferData<AnchorV>,
binding_sig: redjubjub::Signature<Binding>,
}
/// TransferData ensures that:
/// * there is at least one spend or output, and
/// * the shared anchor is only present when there are spends
enum sapling::TransferData<AnchorV: AnchorVariant> {
/// In Transaction::V5, if there are any spends,
/// there must also be a shared spend anchor.
SpendsAndMaybeOutputs {
shared_anchor: AnchorV::Shared,
spends: AtLeastOne<Spend<AnchorV>>,
maybe_outputs: Vec<Output>,
}
/// If there are no spends, there must not be a shared
/// anchor.
JustOutputs {
outputs: AtLeastOne<Output>,
}
}
}
The AtLeastOne type is a vector wrapper which always contains at least one
element. For more details, see its documentation.
Some of these fields are in a different order to the serialized data, see the V4 and V5 transaction specs for details.
The following types have ZcashSerialize and ZcashDeserialize implementations,
because they can be serialized into a single byte vector:
Amountsapling::tree::Rootredjubjub::Signature<Binding>
Adding V5 Sapling Spend
Sapling spend code is located at zebra-chain/src/sapling/spend.rs.
We use AnchorVariant to model the anchor differences between V4 and V5.
And we create a struct for serializing V5 transaction spends:
#![allow(unused)]
fn main() {
struct Spend<AnchorV: AnchorVariant> {
cv: commitment::ValueCommitment,
per_spend_anchor: AnchorV::PerSpend,
nullifier: note::Nullifier,
rk: redjubjub::VerificationKeyBytes<SpendAuth>,
// This field is stored in a separate array in v5 transactions, see:
// https://zips.z.cash/protocol/nu5.pdf#txnencodingandconsensus
// parse using `zcash_deserialize_external_count` and `zcash_serialize_external_count`
zkproof: Groth16Proof,
// This fields is stored in another separate array in v5 transactions
spend_auth_sig: redjubjub::Signature<SpendAuth>,
}
/// The serialization prefix fields of a `Spend` in Transaction V5.
///
/// In `V5` transactions, spends are split into multiple arrays, so the prefix,
/// proof, and signature must be serialised and deserialized separately.
///
/// Serialized as `SpendDescriptionV5` in [protocol specification §7.3].
struct SpendPrefixInTransactionV5 {
cv: commitment::ValueCommitment,
nullifier: note::Nullifier,
rk: redjubjub::VerificationKeyBytes<SpendAuth>,
}
}
The following types have ZcashSerialize and ZcashDeserialize implementations,
because they can be serialized into a single byte vector:
Spend<PerSpendAnchor>(moved from the pre-RFCSpend)SpendPrefixInTransactionV5(new)Groth16Proofredjubjub::Signature<redjubjub::SpendAuth>(new - for v5 spend auth sig arrays)
Note: Spend<SharedAnchor> does not have serialize and deserialize implementations.
It must be split using into_v5_parts before serialization, and
recombined using from_v5_parts after deserialization.
These convenience methods convert between Spend<SharedAnchor> and its v5 parts:
SpendPrefixInTransactionV5, the spend proof, and the spend auth signature.
Changes to Sapling Output
In Zcash the Sapling output fields are the same for V4 and V5 transactions,
so the Output struct is unchanged. However, V4 and V5 transactions serialize
outputs differently, so we create additional structs for serializing outputs in
each transaction version.
The output code is located at zebra-chain/src/sapling/output.rs:
#![allow(unused)]
fn main() {
struct Output {
cv: commitment::ValueCommitment,
cm_u: jubjub::Fq,
ephemeral_key: keys::EphemeralPublicKey,
enc_ciphertext: note::EncryptedNote,
out_ciphertext: note::WrappedNoteKey,
// This field is stored in a separate array in v5 transactions, see:
// https://zips.z.cash/protocol/nu5.pdf#txnencodingandconsensus
// parse using `zcash_deserialize_external_count` and `zcash_serialize_external_count`
zkproof: Groth16Proof,
}
/// Wrapper for `Output` serialization in a `V4` transaction.
struct OutputInTransactionV4(pub Output);
/// The serialization prefix fields of an `Output` in Transaction V5.
///
/// In `V5` transactions, spends are split into multiple arrays, so the prefix
/// and proof must be serialised and deserialized separately.
///
/// Serialized as `OutputDescriptionV5` in [protocol specification §7.3].
struct OutputPrefixInTransactionV5 {
cv: commitment::ValueCommitment,
cm_u: jubjub::Fq,
ephemeral_key: keys::EphemeralPublicKey,
enc_ciphertext: note::EncryptedNote,
out_ciphertext: note::WrappedNoteKey,
}
}
The following fields have ZcashSerialize and ZcashDeserialize implementations,
because they can be serialized into a single byte vector:
OutputInTransactionV4(moved fromOutput)OutputPrefixInTransactionV5(new)Groth16Proof
Note: The serialize and deserialize implementations on Output are moved to
OutputInTransactionV4. In v4 transactions, outputs must be wrapped using
into_v4 before serialization, and unwrapped using
from_v4 after deserialization. In transaction v5, outputs
must be split using into_v5_parts before serialization, and
recombined using from_v5_parts after deserialization.
These convenience methods convert Output to:
- its v4 serialization wrapper
OutputInTransactionV4, and - its v5 parts:
OutputPrefixInTransactionV5and the output proof.
Adding V5 Transactions
Now lets see how the V5 transaction is specified in the protocol, this is the second table of Transaction Encoding and Consensus and how are we going to represent it based in the above changes for Sapling fields and the new Orchard fields.
We propose the following representation for transaction V5 in Zebra:
#![allow(unused)]
fn main() {
enum Transaction::V5 {
lock_time: LockTime,
expiry_height: block::Height,
inputs: Vec<transparent::Input>,
outputs: Vec<transparent::Output>,
sapling_shielded_data: Option<sapling::ShieldedData<SharedAnchor>>,
orchard_shielded_data: Option<orchard::ShieldedData>,
}
}
To model the V5 anchor type, sapling_shielded_data uses the SharedAnchor variant located at zebra-chain/src/transaction/sapling/shielded_data.rs.
The following fields have ZcashSerialize and ZcashDeserialize implementations,
because they can be serialized into a single byte vector:
LockTimeblock::Heighttransparent::Inputtransparent::OutputOption<sapling::ShieldedData<SharedAnchor>>(new)Option<orchard::ShieldedData>(new)
Orchard Additions
Adding Orchard ShieldedData
The new V5 structure will create a new orchard::ShieldedData type. This new type will be defined in a new zebra-chain/src/orchard/shielded_data.rs file:
#![allow(unused)]
fn main() {
struct orchard::ShieldedData {
flags: Flags,
value_balance: Amount,
shared_anchor: orchard::tree::Root,
proof: Halo2Proof,
actions: AtLeastOne<AuthorizedAction>,
binding_sig: redpallas::Signature<Binding>,
}
}
The fields are ordered based on the last data deserialized for each field.
The following types have ZcashSerialize and ZcashDeserialize implementations,
because they can be serialized into a single byte vector:
orchard::Flags(new)AmountHalo2Proof(new)redpallas::Signature<Binding>(new)
Adding Orchard AuthorizedAction
In V5 transactions, there is one SpendAuth signature for every Action. To ensure that this structural rule is followed, we create an AuthorizedAction type in orchard/shielded_data.rs:
#![allow(unused)]
fn main() {
/// An authorized action description.
///
/// Every authorized Orchard `Action` must have a corresponding `SpendAuth` signature.
struct orchard::AuthorizedAction {
action: Action,
// This field is stored in a separate array in v5 transactions, see:
// https://zips.z.cash/protocol/nu5.pdf#txnencodingandconsensus
// parse using `zcash_deserialize_external_count` and `zcash_serialize_external_count`
spend_auth_sig: redpallas::Signature<SpendAuth>,
}
}
Where Action is defined as Action definition.
The following types have ZcashSerialize and ZcashDeserialize implementations,
because they can be serialized into a single byte vector:
Action(new)redpallas::Signature<SpendAuth>(new)
Note: AuthorizedAction does not have serialize and deserialize implementations.
It must be split using into_parts before serialization, and
recombined using from_parts after deserialization.
These convenience methods convert between AuthorizedAction and its parts:
Action and the spend auth signature.
Adding Orchard Flags
Finally, in the V5 transaction we have a new orchard::Flags type. This is a bitfield type defined as:
#![allow(unused)]
fn main() {
bitflags! {
/// Per-Transaction flags for Orchard.
///
/// The spend and output flags are passed to the `Halo2Proof` verifier, which verifies
/// the relevant note spending and creation consensus rules.
struct orchard::Flags: u8 {
/// Enable spending non-zero valued Orchard notes.
///
/// "the `enableSpendsOrchard` flag, if present, MUST be 0 for coinbase transactions"
const ENABLE_SPENDS = 0b00000001;
/// Enable creating new non-zero valued Orchard notes.
const ENABLE_OUTPUTS = 0b00000010;
// Reserved, zeros (bits 2 .. 7)
}
}
}
This type is also defined in orchard/shielded_data.rs.
Note: A consensus rule was added to the protocol specification stating that:
In a version 5 transaction, the reserved bits 2..7 of the flagsOrchard field MUST be zero.
Test Plan
- All renamed, modified and new types should serialize and deserialize.
- The full V4 and V5 transactions should serialize and deserialize.
- Prop test strategies for V4 and V5 will be updated and created.
- Before NU5 activation on testnet, test on the following test vectors:
- Hand-crafted Orchard-only, Orchard/Sapling, Orchard/Transparent, and Orchard/Sapling/Transparent transactions based on the spec
- “Fake” Sapling-only and Sapling/Transparent transactions based on the existing test vectors, converted from V4 to V5 format
- We can write a test utility function to automatically do these conversions
- An empty transaction, with no Orchard, Sapling, or Transparent data
- A v5 transaction with no spends, but some outputs, to test the shared anchor serialization rule
- Any available
zcashdtest vectors
- After NU5 activation on testnet:
- Add test vectors using the testnet activation block and 2 more post-activation blocks
- After NU5 activation on mainnet:
- Add test vectors using the mainnet activation block and 2 more post-activation blocks
Security
To avoid parsing memory exhaustion attacks, we will make the following changes across all Transaction, ShieldedData, Spend and Output variants, V1 through to V5:
- Check cardinality consensus rules at parse time, before deserializing any
Vecs- In general, Zcash requires that each transaction has at least one Transparent/Sprout/Sapling/Orchard transfer, this rule is not currently encoded in our data structures (it is only checked during semantic verification)
- Stop parsing as soon as the first error is detected
These changes should be made in a later pull request, see #1917 for details.
Async Rust in Zebra
Status (Zebra 7.0.0): Partially implemented.
Code: the Tokio component, consensus router, and batch-control crate.
- CPU-bound crypto uses Rayon with oneshot responses; Tokio runs async tasks and blocking I/O.
- The
ChainVerifierexample corresponds toBlockVerifierRouter. - The removed client does not justify current buffer bounds.
- The
IS_SHUTTING_DOWNatomic still usesSeqCst; the proposed replacement is absent.
- Feature Name: async_rust_in_zebra
- Start Date: 2021-03-30
- Design PR: ZcashFoundation/zebra#1965
- Zebra Issue: ZcashFoundation/zebra#1593
Summary
Zebra programmers need to carefully write async code so it doesn’t deadlock or hang.
This is particularly important for poll, select, Buffer, Batch, and Mutex.
Zebra executes concurrent tasks using async Rust, with the tokio executor.
At a higher level, Zebra also uses tower::Services,
tower::Buffers,
and our own tower-batch-control
implementation.
Motivation
Like all concurrent codebases, Zebra needs to obey certain constraints to avoid hangs. Unfortunately, Rust’s tooling in these areas is still developing. So Zebra developers need to manually check these constraints during design, development, reviews, and testing.
Definitions
hang: a Zebra component stops making progress.constraint: a rule that Zebra must follow to preventhangs.CORRECTNESS comment: the documentation for aconstraintin Zebra’s code.task: an async task can execute code independently of other tasks, using cooperative multitasking.contention: slower execution because multiple tasks are waiting to acquire a lock, buffer/batch slot, or readiness.missed wakeup: a taskhangs because it is never scheduled for wakeup.lock: exclusive access to a shared resource. Locks stop other code from running until they are released. For example, a mutex, buffer slot, or service readiness.critical section: code that is executed while holding a lock.deadlock: ahangthat stops an async task executing code, because it is waiting for a lock, slot, or task readiness. For example: a task is waiting for a service to be ready, but the service readiness depends on that task making progress.starvationorlivelock: ahangthat executes code, but doesn’t do anything useful. For example: a loop never terminates.
Guide-level explanation
If you are designing, developing, or testing concurrent Zebra code, follow the patterns in these examples to avoid hangs.
If you are reviewing concurrent Zebra designs or code, make sure that:
- it is clear how the design or code avoids hangs
- the design or code follows the patterns in these examples (as much as possible)
- the concurrency constraints and risks are documented
The Reference section contains in-depth background information about Rust async concurrency in Zebra.
Here are some examples of concurrent designs and documentation in Zebra:
Registering Wakeups Before Returning Poll::Pending
To avoid missed wakeups, futures must schedule a wakeup before they return
Poll::Pending. For more details, see the Poll::Pending and Wakeups
section.
Zebra’s unready_service.rs uses the ready! macro to correctly handle
Poll::Pending from the inner service.
You can see some similar constraints in pull request #1954.
#![allow(unused)]
fn main() {
// CORRECTNESS
//
// The current task must be scheduled for wakeup every time we return
// `Poll::Pending`.
//
//`ready!` returns `Poll::Pending` when the service is unready, and
// the inner `poll_ready` schedules this task for wakeup.
//
// `cancel.poll` also schedules this task for wakeup if it is canceled.
let res = ready!(this
.service
.as_mut()
.expect("poll after ready")
.poll_ready(cx));
}
Futures-Aware Mutexes
To avoid hangs or slowdowns, prefer futures-aware types, particularly for complex waiting or locking code. But in some simple cases, std::sync::Mutex is more efficient. For more details, see the Futures-Aware Types section.
Zebra’s Handshake
won’t block other tasks on its thread, because it uses futures::lock::Mutex:
#![allow(unused)]
fn main() {
pub async fn negotiate_version(
peer_conn: &mut Framed<TcpStream, Codec>,
addr: &SocketAddr,
config: Config,
nonces: Arc<futures::lock::Mutex<HashSet<Nonce>>>,
user_agent: String,
our_services: PeerServices,
relay: bool,
) -> Result<(Version, PeerServices), HandshakeError> {
// Create a random nonce for this connection
let local_nonce = Nonce::default();
// # Correctness
//
// It is ok to wait for the lock here, because handshakes have a short
// timeout, and the async mutex will be released when the task times
// out.
nonces.lock().await.insert(local_nonce);
...
}
}
Zebra’s Inbound service
can’t use an async-aware mutex for its AddressBook, because the mutex is shared
with non-async code. It only holds the mutex to clone the address book, reducing
the amount of time that other tasks on its thread are blocked:
#![allow(unused)]
fn main() {
// # Correctness
//
// Briefly hold the address book threaded mutex while
// cloning the address book. Then sanitize after releasing
// the lock.
let peers = address_book.lock().unwrap().clone();
let mut peers = peers.sanitized();
}
Avoiding Deadlocks when Acquiring Buffer or Service Readiness
To avoid deadlocks, readiness and locks must be acquired in a consistent order. For more details, see the Acquiring Buffer Slots, Mutexes, or Readiness section.
Zebra’s ChainVerifier
avoids deadlocks, contention, and errors by:
- calling
poll_readybefore eachcall - acquiring buffer slots for the earlier verifier first (based on blockchain order)
- ensuring that buffers are large enough for concurrent tasks
#![allow(unused)]
fn main() {
// We acquire checkpoint readiness before block readiness, to avoid an unlikely
// hang during the checkpoint to block verifier transition. If the checkpoint and
// block verifiers are contending for the same buffer/batch, we want the checkpoint
// verifier to win, so that checkpoint verification completes, and block verification
// can start. (Buffers and batches have multiple slots, so this contention is unlikely.)
//
// The chain verifier holds one slot in each verifier, for each concurrent task.
// Therefore, any shared buffers or batches polled by these verifiers should double
// their bounds. (For example, the state service buffer.)
ready!(self
.checkpoint
.poll_ready(cx)
.map_err(VerifyChainError::Checkpoint))?;
ready!(self.block.poll_ready(cx).map_err(VerifyChainError::Block))?;
Poll::Ready(Ok(()))
}
Critical Section Compiler Errors
To avoid deadlocks or slowdowns, critical sections should be as short as possible, and they should not depend on any other tasks. For more details, see the Acquiring Buffer Slots, Mutexes, or Readiness section.
Zebra’s CandidateSet
must release a std::sync::Mutex lock before awaiting a tokio::time::Sleep
future. This ensures that the threaded mutex lock isn’t held over the await
point.
If the lock isn’t dropped, compilation fails, because the mutex lock can’t be sent between threads.
#![allow(unused)]
fn main() {
// # Correctness
//
// In this critical section, we hold the address mutex, blocking the
// current thread, and all async tasks scheduled on that thread.
//
// To avoid deadlocks, the critical section:
// - must not acquire any other locks
// - must not await any futures
//
// To avoid hangs, any computation in the critical section should
// be kept to a minimum.
let reconnect = {
let mut guard = self.address_book.lock().unwrap();
...
let reconnect = guard.reconnection_peers().next()?;
let reconnect = MetaAddr::new_reconnect(&reconnect.addr, &reconnect.services);
guard.update(reconnect);
reconnect
};
// SECURITY: rate-limit new candidate connections
sleep.await;
}
Sharing Progress between Multiple Futures
To avoid starvation and deadlocks, tasks that depend on multiple futures should make progress on all of those futures. This is particularly important for tasks that depend on their own outputs. For more details, see the Unbiased Selection section.
Zebra’s peer crawler task avoids starvation and deadlocks by:
- sharing progress between any ready futures using the
select!macro - spawning independent tasks to avoid hangs (see Acquiring Buffer Slots, Mutexes, or Readiness)
- using timeouts to avoid hangs
You can see a range of hang fixes in pull request #1950.
#![allow(unused)]
fn main() {
// CORRECTNESS
//
// To avoid hangs and starvation, the crawler must:
// - spawn a separate task for each handshake, so they can make progress
// independently (and avoid deadlocking each other)
// - use the `select!` macro for all actions, because the `select` function
// is biased towards the first ready future
loop {
let crawler_action = tokio::select! {
a = handshakes.next() => a,
a = crawl_timer.next() => a,
_ = demand_rx.next() => {
if let Some(candidate) = candidates.next().await {
// candidates.next has a short delay, and briefly holds the address
// book lock, so it shouldn't hang
DemandHandshake { candidate }
} else {
DemandCrawl
}
}
};
match crawler_action {
DemandHandshake { candidate } => {
// spawn each handshake into an independent task, so it can make
// progress independently of the crawls
let hs_join =
tokio::spawn(dial(candidate, connector.clone()));
handshakes.push(Box::pin(hs_join));
}
DemandCrawl => {
// update has timeouts, and briefly holds the address book
// lock, so it shouldn't hang
candidates.update().await?;
}
// handle handshake responses and the crawl timer
}
}
}
Prioritising Cancellation Futures
To avoid starvation, cancellation futures must take priority over other futures, if multiple futures are ready. For more details, see the Biased Selection section.
Zebra’s connection.rs avoids hangs by prioritising the cancel and timer futures over the peer receiver future. Under heavy load, the peer receiver future could always be ready with a new message, starving the cancel or timer futures.
You can see a range of hang fixes in pull request #1950.
#![allow(unused)]
fn main() {
// CORRECTNESS
//
// Currently, select prefers the first future if multiple
// futures are ready.
//
// If multiple futures are ready, we want the cancellation
// to take priority, then the timeout, then peer responses.
let cancel = future::select(tx.cancellation(), timer_ref);
match future::select(cancel, peer_rx.next()) {
...
}
}
Atomic Shutdown Flag
As of April 2021, Zebra implements some shutdown checks using an atomic bool.
Zebra’s shutdown.rs
avoids data races and missed updates by using the strongest memory
ordering (SeqCst).
We plan to replace this raw atomic code with a channel, see #1678.
#![allow(unused)]
fn main() {
/// A flag to indicate if Zebra is shutting down.
///
/// Initialized to `false` at startup.
pub static IS_SHUTTING_DOWN: AtomicBool = AtomicBool::new(false);
/// Returns true if the application is shutting down.
pub fn is_shutting_down() -> bool {
// ## Correctness:
//
// Since we're shutting down, and this is a one-time operation,
// performance is not important. So we use the strongest memory
// ordering.
// https://doc.rust-lang.org/nomicon/atomics.html#sequentially-consistent
IS_SHUTTING_DOWN.load(Ordering::SeqCst)
}
/// Sets the Zebra shutdown flag to `true`.
pub fn set_shutting_down() {
IS_SHUTTING_DOWN.store(true, Ordering::SeqCst);
}
}
Integration Testing Async Code
Sometimes, it is difficult to unit test async code, because it has complex dependencies. For more details, see the Testing Async Code section.
zebrad’s acceptance tests
run short Zebra syncs on the Zcash mainnet or testnet. These acceptance tests
make sure that zebrad can:
- sync blocks using its async block download and verification pipeline
- cancel a sync
- reload disk state after a restart
These tests were introduced in pull request #1193.
#![allow(unused)]
fn main() {
/// Test if `zebrad` can sync some larger checkpoints on mainnet.
#[test]
fn sync_large_checkpoints_mainnet() -> Result<()> {
let reuse_tempdir = sync_until(
LARGE_CHECKPOINT_TEST_HEIGHT,
Mainnet,
STOP_AT_HEIGHT_REGEX,
LARGE_CHECKPOINT_TIMEOUT,
None,
)?;
// if stopping corrupts the rocksdb database, zebrad might hang or crash here
// if stopping does not write the rocksdb database to disk, Zebra will
// sync, rather than stopping immediately at the configured height
sync_until(
(LARGE_CHECKPOINT_TEST_HEIGHT - 1).unwrap(),
Mainnet,
"previous state height is greater than the stop height",
STOP_ON_LOAD_TIMEOUT,
Some(reuse_tempdir),
)?;
Ok(())
}
}
Instrumenting Async Functions
Sometimes, it is difficult to debug async code, because there are many tasks running concurrently. For more details, see the Monitoring Async Code section.
Zebra runs instrumentation on some of its async function using tracing.
Here’s an instrumentation example from Zebra’s sync block downloader:
#![allow(unused)]
fn main() {
/// Queue a block for download and verification.
///
/// This method waits for the network to become ready, and returns an error
/// only if the network service fails. It returns immediately after queuing
/// the request.
#[instrument(level = "debug", skip(self), fields(%hash))]
pub async fn download_and_verify(&mut self, hash: block::Hash) -> Result<(), Report> {
...
}
}
Tracing and Metrics in Async Functions
Sometimes, it is difficult to monitor async code, because there are many tasks running concurrently. For more details, see the Monitoring Async Code section.
Zebra’s client requests are monitored via:
- trace and debug logs using
tracingcrate - related work spans using the
tracingcrate - counters using the
metricscrate
#![allow(unused)]
fn main() {
/// Handle an incoming client request, possibly generating outgoing messages to the
/// remote peer.
///
/// NOTE: the caller should use .instrument(msg.span) to instrument the function.
async fn handle_client_request(&mut self, req: InProgressClientRequest) {
trace!(?req.request);
let InProgressClientRequest { request, tx, span } = req;
if tx.is_canceled() {
metrics::counter!("peer.canceled", 1);
tracing::debug!("ignoring canceled request");
return;
}
...
}
}
Reference-level explanation
The reference section contains in-depth information about concurrency in Zebra:
- After an await, the rest of the Future might not be run
- Task starvation
Poll::Pendingand Wakeups- Futures-Aware Types
- Acquiring Buffer Slots, Mutexes, or Readiness
- Buffer and Batch
- Awaiting Multiple Futures
- Replacing Atomics with Channels
- Testing Async Code
- Monitoring Async Code
Most Zebra designs or code changes will only touch on one or two of these areas.
After an await, the rest of the Future might not be run
Futures can be “canceled” at any await point. Authors of futures must be aware that after an await, the code might not run. Futures might be polled to completion causing the code to work. But then many years later, the code is changed and the future might conditionally not be polled to completion which breaks things. The burden falls on the user of the future to poll to completion, and there is no way for the lib author to enforce this - they can only document this invariant.
In particular, FutureExt::now_or_never:
- drops the future, and
- doesn’t schedule the task for wakeups.
So even if the future or service passed to now_or_never is cloned,
the task won’t be awoken when it is ready again.
Task starvation
Tokio tasks are scheduled cooperatively:
a task is allowed to run until it yields, indicating to the Tokio runtime’s scheduler that it cannot currently continue executing. When a task yields, the Tokio runtime switches to executing the next task.
If a task doesn’t yield during a CPU-intensive operation, or a tight loop, it can starve other tasks on the same thread. This can cause hangs or timeouts.
There are a few different ways to avoid task starvation:
- run the operation on another thread using
spawn_blockingorblock_in_place - manually yield using
yield_now
Poll::Pending and Wakeups
When returning Poll::Pending, poll functions must ensure that the task will be woken up when it is ready to make progress.
In most cases, the poll function calls another poll function that schedules the task for wakeup.
Any code that generates a new Poll::Pending should either have:
- a
CORRECTNESScomment explaining how the task is scheduled for wakeup, or - a wakeup implementation, with tests to ensure that the wakeup functions as expected.
Note: poll functions often have a qualifier, like poll_ready or poll_next.
Futures-Aware Types
Prefer futures-aware types in complex locking or waiting code, rather than types which will block the current thread.
For example:
- Use
futures::lock::Mutexrather thanstd::sync::Mutex - Use
tokio::time::{sleep, timeout}rather thanstd::thread::sleep
Always qualify ambiguous names like Mutex and sleep, so that it is obvious
when a call will block.
If you are unable to use futures-aware types:
- block the thread for as short a time as possible
- document the correctness of each blocking call
- consider re-designing the code to use
tower::Services, or other futures-aware types
In some simple cases, std::sync::Mutex is correct and more efficient, when:
- the value behind the mutex is just data, and
- the locking behaviour is simple.
In these cases:
wrap the
Arc<Mutex<...>>in a struct that provides non-async methods for performing operations on the data within, and only lock the mutex inside these methods
For more details, see the tokio documentation.
Acquiring Buffer Slots, Mutexes, or Readiness
Ideally, buffer slots, mutexes, or readiness should be:
- acquired with one lock per critical section, and
- held for as short a time as possible.
If multiple locks are required for a critical section, acquire them in the same order any time those locks are used. If tasks acquire multiple locks in different orders, they can deadlock, each holding a lock that the other needs.
If a buffer, mutex, future or service has complex readiness dependencies,
schedule those dependencies separate tasks using tokio::spawn. Otherwise,
it might deadlock due to a dependency loop within a single executor task.
Carefully read the documentation of the channel methods you call, to check if they lock.
For example, tokio::sync::watch::Receiver::borrow
holds a read lock, so the borrowed data should always be cloned.
Use Arc for efficient clones if needed.
Never have two active watch borrow guards in the same scope, because that can cause a deadlock. The
watch::Sender may start acquiring a write lock while the first borrow guard is active but the
second one isn’t. That means that the first read lock was acquired, but the second never will be
because starting to acquire the write lock blocks any other read locks from being acquired. At the
same time, the write lock will also never finish acquiring, because it waits for all read locks to
be released, and the first read lock won’t be released before the second read lock is acquired.
In all of these cases:
- make critical sections as short as possible, and
- do not depend on other tasks or locks inside the critical section.
Acquiring Service Readiness
Note: do not call poll_ready on multiple tasks, then match against the results. Use the ready!
macro instead, to acquire service readiness in a consistent order.
Buffer and Batch
The constraints imposed by the tower::Buffer and tower::Batch implementations are:
poll_readymust be called at least once for eachcall- Once we’ve reserved a buffer slot, we always get
Poll::Readyfrom a buffer, regardless of the current readiness of the buffer or its underlying service - The
Buffer/Batchcapacity limits the number of concurrently waiting tasks. Once this limit is reached, further tasks will block, awaiting a free reservation. - Some tasks can depend on other tasks before they resolve. (For example: block validation.)
If there are task dependencies, the
Buffer/Batchcapacity must be larger than the maximum number of concurrently waiting tasks, or Zebra could deadlock (hang).
We also avoid hangs because:
- the timeouts on network messages, block downloads, and block verification will restart verification if it hangs
BufferandBatchrelease their reservations when response future is returned by the buffered/batched service, even if the returned future hangs- in general, we should move as much work into futures as possible, unless the design requires sequential
calls
- in general, we should move as much work into futures as possible, unless the design requires sequential
- larger
Buffer/Batchbounds
Buffered Services
A service should be provided wrapped in a Buffer if:
- it is a complex service
- it has multiple callers, or
- it has a single caller that calls it multiple times concurrently.
Services might also have other reasons for using a Buffer. These reasons should be documented.
Choosing Buffer Bounds
Zebra’s Buffer bounds should be set to the maximum number of concurrent requests, plus 1:
it’s advisable to set bound to be at least the maximum number of concurrent requests the
Bufferwill see https://docs.rs/tower/0.4.3/tower/buffer/struct.Buffer.html#method.new
The extra slot protects us from future changes that add an extra caller, or extra concurrency.
As a general rule, Zebra Buffers should all have at least 5 slots, because most Zebra services can
be called concurrently by:
- the sync service,
- the inbound service, and
- multiple concurrent
zebra-clientblockchain scanning tasks.
Services might also have other reasons for a larger bound. These reasons should be documented.
We should limit Buffer lengths for services whose requests or responses contain Blocks (or other large
data items, such as Transaction vectors). A long Buffer full of Blocks can significantly increase memory
usage.
For example, parsing a malicious 2 MB block can take up to 12 MB of RAM. So a 5 slot buffer can use 60 MB of RAM.
Long Buffers can also increase request latency. Latency isn’t a concern for Zebra’s core use case as a node
software, but it might be an issue if wallets, exchanges, or block explorers want to use Zebra.
Awaiting Multiple Futures
When awaiting multiple futures, Zebra can use biased or unbiased selection.
Typically, we prefer unbiased selection, so that if multiple futures are ready, they each have a chance of completing. But if one of the futures needs to take priority (for example, cancellation), you might want to use biased selection.
Unbiased Selection
The futures::select! and
tokio::select! macros select
ready arguments at random by default.
To poll a select! in order, pass biased; as the first argument to the macro.
Also consider the FuturesUnordered stream for unbiased selection of a large number
of futures. However, this macro and stream require mapping all arguments to the same
type.
Consider mapping the returned type to a custom enum with module-specific names.
Biased Selection
The futures::select
is biased towards its first argument. If the first argument is always ready, the second
argument will never be returned. (This behavior is not documented or guaranteed.) This
bias can cause starvation or hangs. Consider edge cases where queues are full, or there
are a lot of messages. If in doubt:
- put shutdown or cancel oneshots first, then timers, then other futures
- use the
select!macro to ensure fairness
Select’s bias can be useful to ensure that cancel oneshots and timers are always
executed first. Consider the select_biased! macro and FuturesOrdered stream
for guaranteed ordered selection of futures. (However, this macro and stream require
mapping all arguments to the same type.)
The futures::select Either return type is complex, particularly when nested. This
makes code hard to read and maintain. Map the Either to a custom enum.
Replacing Atomics with Channels
If you’re considering using atomics, prefer a safe, tested, portable abstraction, like tokio’s watch or oneshot channels.
In Zebra, we try to use safe abstractions, and write obviously correct code. It takes a lot of effort to write, test, and maintain low-level code. Almost all of our performance-critical code is in cryptographic libraries. And our biggest performance gains from those libraries come from async batch cryptography.
We are gradually replacing atomics with channels in Zebra.
Atomic Risks
Some atomic sizes and atomic operations are not available on some platforms. Others come with a performance penalty on some platforms.
It’s also easy to use a memory ordering that’s too weak. Future code changes might require a stronger memory ordering. But it’s hard to test for these kinds of memory ordering bugs.
Some memory ordering bugs can only be discovered on non-x86 platforms. And when they do occur,
they can be rare. x86 processors guarantee strong orderings, even for Relaxed accesses.
Since Zebra’s CI all runs on x86 (as of June 2021), our tests get AcqRel orderings, even
when we specify Relaxed. But ARM processors like the Apple M1
implement weaker memory orderings, including genuinely Relaxed access. For more details, see the hardware reordering
section of the Rust nomicon.
But if a Zebra feature requires atomics:
- use an
AtomicUsizewith the strongest memory ordering (SeqCst) - use a weaker memory ordering, with:
- a correctness comment,
- multithreaded tests with a concurrency permutation harness like loom, on x86 and ARM, and
- benchmarks to prove that the low-level code is faster.
Tokio’s watch channel uses SeqCst for reads and writes
to its internal “version” atomic. So Zebra should do the same.
Testing Async Code
Zebra’s existing acceptance and integration tests will catch most hangs and deadlocks.
Some tests are only run after merging to main. If a recently merged PR fails on main,
we revert the PR, and fix the failure.
Some concurrency bugs only happen intermittently. Zebra developers should run regular full syncs to ensure that their code doesn’t cause intermittent hangs. This is particularly important for code that modifies Zebra’s highly concurrent crates:
zebradzebra-networkzebra-statezebra-consensustower-batch-controltower-fallback
Monitoring Async Code
Zebra uses the following crates for monitoring and diagnostics:
- tracing: tracing events, spans, logging
- tracing-futures: future and async function instrumentation
- metrics with a prometheus exporter
These introspection tools are also useful during testing:
tracinglogs individual events- spans track related work through the download and verification pipeline
metricsmonitors overall progress and error rates- labels split counters or gauges into different categories (for example, by peer address)
Drawbacks
Implementing and reviewing these constraints creates extra work for developers. But concurrency bugs slow down every developer, and impact users. And diagnosing those bugs can take a lot of developer effort.
Unresolved questions
Can we catch these bugs using automated tests?
How can we diagnose these kinds of issues faster and more reliably?
- TurboWish
(also known as
tokio-console) looks really promising for task, channel, and future introspection. As of May 2020, there is an early prototype available.
Value pools
Status (Zebra 7.0.0): Implemented.
Code: ValueBalance, block accounting, and pool persistence.
- Balances include deferred funding (NU6), Ironwood (NU6.3), and the Network Sustainability Mechanism (NSM, NU7).
- The NSM balance is not a chain value pool and is excluded from issued supply.
- The remaining transaction value sums transparent, Sprout, Sapling, Orchard, and Ironwood balances.
- Serialized value balances occupy 56 bytes, not 32.
- The
tip_chain_value_poolcolumn stores the latest balance under key(). - The transaction wrapper supports versions through 6.
- Block changes use the exact parent’s pools and deferred-pool change.
- Pool accounting includes deferred funding and NSM changes.
- Coinbase subsidy and fee validation is implemented, not future work.
- Removing a tip subtracts its pool change; finalizing a root leaves total pools unchanged.
- Feature Name: value_pools
- Start Date: 2021-06-29
- Design PR: ZcashFoundation/zebra#2430
- Zebra Issue: ZcashFoundation/zebra#2152
Summary
This document describes how to verify the Zcash chain and transaction value pools in Zebra.
Motivation
In the Zcash protocol there are consensus rules that:
- prohibit negative chain value pools ZIP-209, and
- restrict the creation of new money to a specific number of coins in each coinbase transaction. Spec Section 3.4
These rules make sure that a fixed amount of Zcash is created by each block, even if there are vulnerabilities in some shielded pools.
(Checking the coins created by coinbase transactions and funding streams is out of scope for this design.)
Definitions
Transaction Value Balances
transaction value pool- The unspent input value in a transaction. Inputs add value, outputs remove value, and value balances modify value. The pool represents the sum of transparent and shielded inputs, minus the sum of transparent and shielded outputs.value balance- The change in a transaction’s value pool. There is a separate value balance for each transparent and shielded pool.transparent value balance- The change in the transaction value pool, due to transparent inputs and outputs. The sum of the UTXOs spent by transparent inputs intx_infields, minus the sum of newly created outputs intx_outfields.sprout value balance- The change in the transaction value pool, due to sprout JoinSplits. The sum of allv_sprout_newfields, minus the sum of allv_sprout_oldfields.sapling value balance- The change in the transaction value pool, due to sapling Spends and Outputs. Equal to thevalueBalanceSaplingfield.orchard value balance- The change in the transaction value pool, due to orchard Actions. Equal to thevalueBalanceOrchardfield.remaining transaction value- The unspent value in the transaction value pool. The sum of the transparent and shielded value balances in each transaction. This value is equal to thetransaction value poolafter we know the values of all the input UTXOs.coinbase transaction- A transaction which spends newly created value (coinbase), and the remaining value of other transactions in its block (miner fees). Coinbase transactions do not have any other inputs, so they can’t spend the outputs of other transactions.
Chain Value Pools
Note: chain value pools and transaction value balances have opposite signs.
chain value pool balance- The total value of unspent outputs in the chain, for each transparent and shielded pool. The sum of all block chain value pool changes in the chain. Each of the transparent, sprout, sapling, and orchard chain value pool balances must be non-negative.block chain value pool change- The change in the chain value pools caused by a block. The negative sum of all the value balances in each block.
Guide-level explanation
Transaction Value Balances
Each transaction has an individual value pool, containing its unspent input value.
Spent transparent inputs add value to this pool, and newly created transparent outputs remove value. Similarly, Sprout JoinSplits have a field that adds value to the transaction pool, and a field that removes value. These transparent and sprout values are unsigned.
Sapling and Orchard have a single signed value balance per transaction, which modifies the transaction value pool.
We need to check that each transaction’s total output value is less than or equal to the total input value. The remaining value in the transaction must not be negative. This makes sure that transactions can only spend up to their total input value. (Only coinbase transactions can create new value.)
In the spec, this is called the remaining value in the transparent transaction value pool. But in Zebra, we don’t assign this value to a specific pool. We just call it the transaction value pool.
Chain Value Pools
There is one chain value pool for transparent funds, and one for each kind of shielded transfer, containing their unspent outputs.
These value pools are updated using chain value pool changes, which are the negation of transaction value balances. (Transaction value balances use unspent input value, but chain value balances use unspent outputs.)
Each of the chain value pools can change its value with every block added to the chain. This is a state feature and Zebra handle this in the zebra-state crate. We propose to store the pool values for the finalized tip height on disk.
We need to check each chain value pool as blocks are added to the chain, to make sure that chain balances never go negative.
Summary of the implementation
- Create a new type
ValueBalancethat will containAmounts for each pool(transparent, sprout, sapling, orchard). - Create
value_pool()methods on each relevant submodule (transparent, joinsplit, sapling and orchard). - Create a
value_pool()method in transaction with all the above and in block with all the transaction value balances. - Pass the value balance of the incoming block into the state.
- Get a previously stored value balance.
- With both values check the consensus rules (constraint violations).
- Update the saved values for the new tip.
Reference-level explanation
Consensus rules
Shielded Chain Value Pools
Consensus rules:
If any of the “Sprout chain value pool balance”, “Sapling chain value pool balance”, or “Orchard chain value pool balance” would become negative in the block chain created as a result of accepting a block, then all nodes MUST reject the block as invalid.
Nodes MAY relay transactions even if one or more of them cannot be mined due to the aforementioned restriction.
https://zips.z.cash/zip-0209#specification
Transparent Transaction Value Pool & Remaining Value
The unspent input value in a transaction: the sum of the transaction value balances.
Consensus rules:
Transparent inputs to a transaction insert value into a transparent transaction value pool associated with the transaction, and transparent outputs remove value from this pool.
As in Bitcoin, the remaining value in the transparent transaction value pool of a non-coinbase transaction is available to miners as a fee. The remaining value in the transparent transaction value pool of a coinbase transaction is destroyed.
The remaining value in the transparent transaction value pool MUST be nonnegative.
https://zips.z.cash/protocol/protocol.pdf#transactions
In Zebra, the remaining value in non-coinbase transactions is not assigned to any particular pool, until a miner spends it as part of a coinbase output.
Sprout Chain Value Pool
Consensus rules:
Each JoinSplit transfer can be seen, from the perspective of the transparent transaction value pool, as an input and an output simultaneously.
v_sprout_old takes value from the transparent transaction value pool and v_sprout_new adds value to the transparent transaction value pool . As a result, v_sprout_old is treated like an output value, whereas v_sprout_new is treated like an input value.
As defined in ZIP-209, the Sprout chain value pool balance for a given block chain is the sum of all v_sprout_old field values for transactions in the block chain, minus the sum of all v_sprout_new fields values for transactions in the block chain.
If the Sprout chain value pool balance would become negative in the block chain created as a result of accepting a block, then all nodes MUST reject the block as invalid.
https://zips.z.cash/protocol/protocol.pdf#joinsplitbalance
Sapling Chain Value Pool
Consensus rules:
A positive Sapling balancing value takes value from the Sapling transaction value pool and adds it to the transparent transaction value pool. A negative Sapling balancing value does the reverse. As a result, positive vbalanceSapling is treated like an input to the transparent transaction value pool, whereas negative vbalanceSapling is treated like an output from that pool.
As defined in ZIP-209, the Sapling chain value pool balance for a given block chain is the negation of the sum of all valueBalanceSapling field values for transactions in the block chain.
If the Sapling chain value pool balance would become negative in the block chain created as a result of accepting a block, then all nodes MUST reject the block as invalid.
https://zips.z.cash/protocol/protocol.pdf#saplingbalance
Orchard Chain Value Pool
Consensus rules:
Orchard introduces Action transfers, each of which can optionally perform a spend, and optionally perform an output. Similarly to Sapling, the net value of Orchard spends minus outputs in a transaction is called the Orchard balancing value, measured in zatoshi as a signed integer vbalanceOrchard.
vbalanceOrchard is encoded in a transaction as the field valueBalanceOrchard. If a transaction has no Action descriptions, vbalanceOrchard is implicitly zero. Transaction fields are described in § 7.1 ‘Transaction Encoding and Consensus’ on p. 116.
A positive Orchard balancing value takes value from the Orchard transaction value pool and adds it to the transparent transaction value pool. A negative Orchard balancing value does the reverse. As a result, positive vbalanceOrchard is treated like an input to the transparent transaction value pool, whereas negative vbalanceOrchard is treated like an output from that pool.
Similarly to the Sapling chain value pool balance defined in ZIP-209, the Orchard chain value pool balance for a given block chain is the negation of the sum of all valueBalanceOrchard field values for transactions in the block chain.
If the Orchard chain value pool balance would become negative in the block chain created as a result of accepting a block , then all nodes MUST reject the block as invalid.
https://zips.z.cash/protocol/protocol.pdf#orchardbalance
Transparent Chain Value Pool
Consensus rule:
Transfers of transparent value work essentially as in Bitcoin
https://zips.z.cash/protocol/protocol.pdf#overview
There is no explicit Zcash consensus rule that the transparent chain value pool balance must be non-negative. But an equivalent rule must be enforced by Zcash implementations, so that each block only creates a fixed amount of coins.
Specifically, this rule can be derived from other consensus rules:
- a transparent output must have a non-negative value,
- a transparent input can only spend an unspent transparent output,
- so, there must be a non-negative remaining value in the transparent transaction value pool.
Some of these consensus rules are derived from Bitcoin, so they may not be documented in the Zcash Specification.
Coinbase Transactions
In this design, we assume that all coinbase outputs are valid, to avoid checking the newly created coinbase value, and the miner fees.
The coinbase value and miner fee rules will be checked as part of a future design.
Exceptions and Edge Cases
Value pools and value balances include the value of all unspent outputs, regardless of whether they can actually be spent.
For example:
- transparent outputs which have unsatisfiable lock scripts
- shielded outputs which have invalid private keys
However, some value is not part of any output:
- if created value or miner fees are not spent in a coinbase transaction, they are destroyed
- since coinbase transaction output values are rounded to the nearest zatoshi, any fractional part of miner-controlled or funding stream outputs is destroyed by rounding
Therefore:
- the total of all chain value pools will always be strictly less than
MAX_MONEY, and - the current total of all chain value pools will always be less than or equal to the number of coins created in coinbase transactions.
These properties are implied by other consensus rules, and do not need to be checked separately.
Proposed Implementation
Create a new ValueBalance type
- Code will be located in a new file:
zebra-chain/src/value_balance.rs. - Supported operators apply to all the
Amounts inside the type:+,-,+=,-=,sum(). - Implementation of the above operators are similar to the ones implemented for
Amount<C>inzebra-chain/src/amount.rs. In particular, we want to return aResulton them so we can error when a constraint is violated. - We will use
Defaultto represent a totally emptyValueBalance, this is the state of all pools at the genesis block.
#![allow(unused)]
fn main() {
#[serde(bound = "C: Constraint")]
struct ValueBalance<C = NegativeAllowed> {
transparent: Amount<C>,
sprout: Amount<C>,
sapling: Amount<C>,
orchard: Amount<C>,
}
impl ValueBalance {
/// [Consensus rule]: The remaining value in the transparent transaction value pool MUST be nonnegative.
///
/// This rule applies to Block and Mempool transactions.
///
/// [Consensus rule]: https://zips.z.cash/protocol/protocol.pdf#transactions
fn remaining_transaction_value(&self) -> Result<Amount<NonNegative>, Err> {
// This rule checks the transparent value balance minus the sum of the sprout, sapling, and orchard
// value balances in a transaction is nonnegative
self.transparent - [self.sprout + self.sapling + self.orchard].sum()
}
}
impl Add for Result<ValueBalance<C>>
where
C: Constraint,
{
}
impl Sub for Result<ValueBalance<C>>
where
C: Constraint,
{
}
impl AddAssign for Result<ValueBalance<C>>
where
C: Constraint,
{
}
impl SubAssign for Result<ValueBalance<C>>
where
C: Constraint,
{
}
impl Sum for Result<ValueBalance<C>>
where
C: Constraint,
{
}
impl Default for ValueBalance<C>
where
C: Constraint,
{
}
}
Create a method in Transaction that returns ValueBalance<NegativeAllowed> for the transaction
We first add value_balance() methods in all the modules we need and use them to get the value balance for the whole transaction.
Create a method in Input that returns ValueBalance<NegativeAllowed>
- Method location is at
zebra-chain/src/transparent.rs. - Method need
utxos, this information is available inverify_transparent_inputs_and_outputs. - If the utxos are not available in the block or state, verification will timeout and return an error
#![allow(unused)]
fn main() {
impl Input {
fn value_balance(&self, utxos: &HashMap<OutPoint, Utxo>) -> ValueBalance<NegativeAllowed> {
}
}
}
Create a method in Output that returns ValueBalance<NegativeAllowed>
- Method location is at
zebra-chain/src/transparent.rs.
#![allow(unused)]
fn main() {
impl Output {
fn value_balance(&self) -> ValueBalance<NegativeAllowed> {
}
}
}
Create a method in JoinSplitData that returns ValueBalance<NegativeAllowed>
- Method location is at
zebra-chain/src/transaction/joinsplit.rs
#![allow(unused)]
fn main() {
pub fn value_balance(&self) -> ValueBalance<NegativeAllowed> {
}
}
Create a method in sapling::ShieldedData that returns ValueBalance<NegativeAllowed>
- Method location is at
zebra-chain/src/transaction/sapling/shielded_data.rs
#![allow(unused)]
fn main() {
pub fn value_balance(&self) -> ValueBalance<NegativeAllowed> {
}
}
Create a method in orchard::ShieldedData that returns ValueBalance<NegativeAllowed>
- Method location is at
zebra-chain/src/transaction/orchard/shielded_data.rs
#![allow(unused)]
fn main() {
pub fn value_balance(&self) -> ValueBalance<NegativeAllowed> {
}
}
Create the Transaction method
- Method location:
zebra-chain/src/transaction.rs - Method will use all the
value_balances()we created until now.
#![allow(unused)]
fn main() {
/// utxos must contain the utxos of every input in the transaction,
/// including UTXOs created by earlier transactions in this block.
pub fn value_balance(&self, utxos: &HashMap<transparent::OutPoint, Utxo>) -> ValueBalance<NegativeAllowed> {
}
}
Create a method in Block that returns ValueBalance<NegativeAllowed> for the block
- Method location is at
zebra-chain/src/block.rs. - Method will make use of
Transaction::value_balancemethod created before.
#![allow(unused)]
fn main() {
/// utxos must contain the utxos of every input in the transaction,
/// including UTXOs created by a transaction in this block,
/// then spent by a later transaction that's also in this block.
pub fn value_balance(&self, utxos: &HashMap<transparent::OutPoint, Utxo>) -> ValueBalance<NegativeAllowed> {
self.transactions()
.map(Transaction::value_balance)
.sum()
.expect("Each block should have at least one coinbase transaction")
}
}
Check the remaining transaction value consensus rule
- Do the check in
zebra-consensus/src/transaction.rs - Make the check part of the basic checks
#![allow(unused)]
fn main() {
..
// Check the remaining transaction value consensus rule:
tx.value_balance().remaining_transaction_value()?;
..
}
Pass the value balance for this block from the consensus into the state
- Add a new field into
PreparedBlocklocated atzebra-state/src/request.rs, this is theNonFinalizedsection of the state.
#![allow(unused)]
fn main() {
pub struct PreparedBlock {
..
/// The value balances for each pool for this block.
pub block_value_balance: ValuePool<NegativeAllowed>,
}
}
- In
zebra-consensus/src/block.rspass the value balance to the zebra-state:
#![allow(unused)]
fn main() {
let block_value_balance = block.value_balance();
let prepared_block = zs::PreparedBlock {
..
block_value_balance,
};
}
Add a value pool into the state Chain struct
- This is the value pool for the non finalized part of the blockchain.
- Location of the
Chainstructure where the pool field will be added:zebra-state/src/service/non_finalized_state/chain.rs
#![allow(unused)]
fn main() {
pub struct Chain {
..
/// The chain value pool balance at the tip of this chain.
value_pool: ValueBalance<NonNegative>,
}
}
- Add a new argument
finalized_tip_value_balanceto thecommit_new_chain()method located in the same file. - Pass the new argument to the Chain in:
#![allow(unused)]
fn main() {
let mut chain = Chain::new(finalized_tip_history_tree, finalized_tip_value_balance);
}
Note: We don’t need to pass the finalized tip value balance into the commit_block() method.
Check the consensus rules when the chain is updated or reversed
- Location:
zebra-state/src/service/non_finalized_state/chain.rs
#![allow(unused)]
fn main() {
impl UpdateWith<ValueBalance<NegativeAllowed>> for Chain {
fn update_chain_state_with(&mut self, value_balance: &ValueBalance<NegativeAllowed>) -> Result<(), Err> {
self.value_pool = (self.value_pool + value_balance)?;
Ok(())
}
fn revert_chain_state_with(&mut self, value_balance: &ValueBalance<NegativeAllowed>) -> Result<(), Err> {
self.value_pool = (self.value_pool + value_balance)?;
Ok(())
}
}
}
Changes to finalized state
The state service will call commit_new_chain(). We need to pass the value pool from the disk into this function.
#![allow(unused)]
fn main() {
self.mem
.commit_new_chain(prepared, self.disk.history_tree(), self.disk.get_pool())?;
}
We now detail what is needed in order to have the get_pool() method available.
Serialization of ValueBalance<C>
In order to save ValueBalance into the disk database we must implement IntoDisk and FromDisk for ValueBalance and for Amount:
#![allow(unused)]
fn main() {
impl IntoDisk for ValueBalance<C> {
type Bytes = [u8; 32];
fn as_bytes(&self) -> Self::Bytes {
[self.transparent.to_bytes(), self.sprout.to_bytes(),
self.sapling.to_bytes(), self.orchard.to_bytes()].concat()
}
}
impl FromDisk for ValueBalance<C> {
fn from_bytes(bytes: impl AsRef<[u8]>) -> Self {
let array = bytes.as_ref().try_into().unwrap();
ValueBalance {
transparent: Amount::from_bytes(array[0..8]).try_into().unwrap()
sprout: Amount::from_bytes(array[8..16]).try_into().unwrap()
sapling: Amount::from_bytes(array[16..24]).try_into().unwrap()
orchard: Amount::from_bytes(array[24..32]).try_into().unwrap()
}
}
}
impl IntoDisk for Amount {
type Bytes = [u8; 8];
fn as_bytes(&self) -> Self::Bytes {
self.to_bytes()
}
}
impl FromDisk for Amount {
fn from_bytes(bytes: impl AsRef<[u8]>) -> Self {
let array = bytes.as_ref().try_into().unwrap();
Amount::from_bytes(array)
}
}
}
The above code is going to need a Amount::from_bytes new method.
Add a from_bytes method in Amount
- Method location is at
zebra-chain/src/amount.rs - A
to_bytes()method already exist, placefrom_bytes()right after it.
#![allow(unused)]
fn main() {
/// From little endian byte array
pub fn from_bytes(&self, bytes: [u8; 8]) -> Self {
let amount = i64::from_le_bytes(bytes).try_into().unwrap();
Self(amount, PhantomData)
}
}
Changes to zebra-state/src/request.rs
Add a new field to FinalizedState:
#![allow(unused)]
fn main() {
pub struct FinalizedBlock {
..
/// The value balance for transparent, sprout, sapling and orchard
/// inside all the transactions of this block.
pub(crate) block_value_balance: ValueBalance<NegativeAllowed>,
}
}
Populate it when PreparedBlock is converted into FinalizedBlock:
#![allow(unused)]
fn main() {
impl From<PreparedBlock> for FinalizedBlock {
fn from(prepared: PreparedBlock) -> Self {
let PreparedBlock {
..
block_value_balance,
} = prepared;
Self {
..
block_value_balance,
}
}
}
}
Changes to zebra-state/src/service/finalized_state.rs
First we add a column of type ValueBalance that will contain Amounts for all the pools: transparent, sprout, sapling, orchard:
#![allow(unused)]
fn main() {
rocksdb::ColumnFamilyDescriptor::new("tip_chain_value_pool", db_options.clone()),
}
At block commit(commit_finalized_direct()) we create the handle for the new column:
#![allow(unused)]
fn main() {
let tip_chain_value_pool = self.db.cf_handle("tip_chain_value_pool").unwrap();
}
Next we save each tip value pool into the field for each upcoming block except for the genesis block:
#![allow(unused)]
fn main() {
// Consensus rule: The block height of the genesis block is 0
// https://zips.z.cash/protocol/protocol.pdf#blockchain
if height == block::Height(0) {
batch.zs_insert(tip_chain_value_pool, height, ValueBalance::default());
} else {
let current_pool = self.current_value_pool();
batch.zs_insert(tip_chain_value_pool, height, (current_pool + finalized.block_value_balance)?);
}
}
The current_value_pool() function will get the stored value of the pool at the tip as follows:
#![allow(unused)]
fn main() {
pub fn current_value_pool(&self) -> ValuePool<NonNegative> {
self.db.cf_handle("tip_chain_value_pool")
}
}
Test Plan
Unit tests
- Create a transaction that has a negative remaining value.
- Test that the transaction fails the verification in
Transaction::value_balance() - To avoid passing the utxo we can have
0as the amount of the transparent pool and some negative shielded pool.
- Test that the transaction fails the verification in
Prop tests
- Create a chain strategy that ends up with a valid value balance for all the pools (transparent, sprout, sapling, orchard)
- Test that the amounts are all added to disk.
- Add new blocks that will make each pool became negative.
- Test for constraint violations in the value balances for each case.
- Failures should be at
update_chain_state_with().
- Test consensus rules success and failures in
revert_chain_state_with()- TODO: how?
- serialize and deserialize
ValueBalanceusingIntoDiskandFromDisk
Manual tests
- Zebra must sync up to tip computing all value balances and never breaking the value pool rules.
Future Work
Add an extra state request to verify the speculative chain balance after applying a Mempool transaction. (This is out of scope for our current NU5 and mempool work.)
Note: The chain value pool balance rules apply to Block transactions, but they are optional for Mempool transactions:
Nodes MAY relay transactions even if one or more of them cannot be mined due to the aforementioned restriction.
https://zips.z.cash/zip-0209#specification
Since Zebra does chain value pool balance validation in the state, we want to skip verifying the speculative chain balance of Mempool transactions.
API reference
-
docs.rs documents the latest published release of each crate.
-
The internal documentation documents
main, including private items; every push tomainregenerates it. Build it locally with:cargo doc --no-deps --workspace --all-features --document-private-items -
JSON-RPC server documents the RPC interface; a running node also describes its methods through the
rpc.discovermethod (OpenRPC).