Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Zebra continuous delivery

The deployment wrapper builds a container image and passes its immutable Google Artifact Registry (GAR) identity, repository@sha256:digest, to the reusable deployment workflow. The source commit is a separate, full 40-character SHA input.

See the workflow inventory, deployment runbook, and naming rationale.

Triggers and environments

TriggerGitHub environmentDeployment
Matching push to mainstageMainnet and Testnet in all three zones
Matching push to nextnextMainnet and Testnet in all three zones
release: released with a tag starting with vprodMainnet and Testnet in all three zones
Manual dispatchSelected dev, next, or prodOne selected network and zone
Matching repository-owned PRdevImage build only; no node deployment

Push and PR path filters include Rust source, text files, Cargo manifests and lockfiles, .cargo/config.toml, Clippy configuration, docker/**, .dockerignore, and the three build/deployment workflow files listed in the wrapper. These filters are independent of .github/path-filters.yml. A documentation-only push does not deploy nodes. Fork PRs skip the wrapper’s image build.

The released activity covers stable publication and promotion of a prerelease to a stable release. Prerelease publication does not trigger this workflow. A release whose tag marks a pre-release from next, any pre-release version other than a release candidate, never deploys, even after promotion. Manual dispatch still uses the selected environment. Docker Hub publication belongs to release automation, not this deployment wrapper. See the release process before publishing a release or dispatching a production deployment.

Deployment jobs use environment-scoped machine type, subnet, region, logging, and Sentry variables. Authentication uses the GCP workflow credentials.

Image publication

The image builder publishes to us-docker.pkg.dev/<GCP_PROJECT>/zebra/<image_name>. The GCP_PROJECT variable comes from the selected GitHub environment.

CallerImage nameDockerfile target
release-binaries.ymlzebraruntime
zfnd-deploy-nodes-gcp.ymlzebradruntime
zfnd-ci-integration-tests-gcp.ymlValue of CI_IMAGE_NAMEtests

Docker Hub publication additionally requires publish_to_dockerhub=true. Only the release-binaries caller enables it, for every published release and prerelease. Nonrelease builds publish only to GAR, not Docker Hub or GitHub Container Registry (GHCR); the builder has no GHCR destination.

The tag recipes produce these development tags; matching conditions combine:

Ref or eventTag
Pull-request refpr-<number>
Branch ref, including manual dispatch from a branchSanitized branch name
Prerelease tag from next (any prerelease except -rc)edge
Scheduled runnightly, the default type=schedule pattern
Nonrelease event with a commit SHAsha-<first-seven-characters>

The pinned metadata action defines these defaults. The workflow disables SHA tags for release events in both registries; PR, branch-push, scheduled, and manual runs keep them for CI. For version tags and latest, see Docker images.

Zonal instance groups

A managed instance group (MIG) maintains the requested VM configuration. Zebra uses one zonal MIG per environment, branch identity, network, and zone. Each new MIG has one instance and one persistent chain-state disk. The supported zones are us-east1-b, us-east1-c, and us-east1-d. Resource names use lowercase mainnet or testnet and the final zone letter.

EnvironmentProjectMIGState disk
prodzfnd-prod-zebrazebrad-${network}-${zone-letter}zebrad-cache-${network}-${zone-letter}
stagezfnd-dev-zebrazebrad-main-${network}-${zone-letter}zebrad-cache-main-${network}-${zone-letter}
nextzfnd-dev-zebrazebrad-next-${network}-${zone-letter}zebrad-cache-next-${network}-${zone-letter}
devzfnd-dev-zebrazebrad-${ref-slug}-${network}-${zone-letter}zebrad-cache-${ref-slug}-${network}-${zone-letter}

Manual dispatch selects the source ref independently of the destination. Stable destinations replace their MIGs even from a feature branch; check the dispatch targets.

Update mechanics

The reusable workflow validates its image, source SHA, environment, network list, and zone list. It then ensures the network health checks exist and resolves cached state separately for each network when requested.

Each deploy-nodes matrix job:

  1. Rejects a disk held by an instance belonging to another MIG.
  2. Reuses the named state disk, or creates a 400 GB pd-balanced disk from a matching cache image. A missing disk without a cache image is an error; the workflow does not create an empty disk.
  3. Creates or reuses an instance template whose name includes a digest of its configuration. The template runs the immutable container image on a Spot VM using cos-cloud/cos-stable and mounts the state disk at /home/zebra/.cache/zebra.
  4. Creates a size-one MIG if absent. The disk’s stateful policy preserves it across VM replacement but deletes it on permanent instance deletion.
  5. Updates an existing MIG with --replacement-method=recreate, --max-surge=0, and --max-unavailable=1.
  6. Waits for the requested template, then optionally for group stability; see deployment verification.

A per-MIG concurrency group covers both waits and does not cancel an active deployment. Matrix fail-fast is disabled. The six network/zone jobs run independently: the workflow does not serialize zones. The one-instance unavailability limit applies within each zonal MIG, not across a network’s three MIGs. Use single-zone dispatches when the operation requires other zones to remain untouched.

Manual dispatches omit the stability wait. Check application health before serving clients.

State and addresses

Push deployments request cached images for new disks. Release deployments disable cache lookup and require pre-existing disks. Manual dispatch defaults to need_cached_disk=true and cached_disk_type=tip; need_cached_disk=false means reuse a disk already present, not initialize an empty one. Existing disks do not require a matching image. The runbook covers cache-image lookup and manual bootstrapping.

Stable deployments use reserved external IPs. Other feature-branch dev deployments use --no-address and do not set ZEBRA_NETWORK__EXTERNAL_ADDR; outbound connectivity uses the configured subnet.