GCP Deployment Operations

Operational procedures for the GCP Continuous Delivery pipeline. Architectural rationale: ADR 0006. High-level model: Continuous Delivery.

Two GCP projects (zfnd-prod-zebra, zfnd-dev-zebra) in us-east1, zones b, c, d. Every MIG, instance, and disk carries environment, network, zone, created_by, github_ref, and github_sha labels. The environment label is standardized as dev, stage, next, or prod. Use created_by as the kind discriminator: release, push, or workflow_dispatch.

The recipes below assume gh and gcloud are authenticated, and use these shell defaults when scoped to dev:

P=zfnd-dev-zebra

For production, set P=zfnd-prod-zebra instead.

Zonal MIGs use --zone (not --region). One MIG per zone per network. MIG names end with the zone letter (-b, -c, -d).

Deploy success has two stages

Each deploy-nodes matrix cell reports two sequential signals:

• Template rollout waits up to 10 minutes for the requested template to reach the zonal MIG. A failure means the deployment infrastructure did not converge.
• Application health, when enabled, then waits up to 90 minutes for the zonal MIG to reach HEALTHY. A failure means the node took longer than that to establish peers and catch up to chain tip; on-call action is usually “wait, or investigate why sync is slow”.

The per-MIG concurrency lock covers both stages, so another run cannot replace the template while health is being verified. Either failure uses failure-issue with label ci-fail/release.

Quick reference

PR deploys

A PR deploy creates one zonal MIG zebrad-${branch}-${network}-${zone-letter} in zfnd-dev-zebra, bootstrapped from the latest matching cache image (preferring images from the same branch, falling back to main, then any branch). Lives until you reap it.

Run one

gh workflow run zfnd-deploy-nodes-gcp.yml -R ZcashFoundation/zebra \
  --ref my-branch \
  -f network=Mainnet \
  -f zone=us-east1-b \
  -f environment=dev \
  -f need_cached_disk=true \
  -f cached_disk_type=tip

Find yours

gcloud compute instance-groups managed list --project $P \
  --filter="labels.created_by=workflow_dispatch AND labels.github_ref=my-branch-slug"

Spare from cleanup

Apply one of two labels:

• keep_until=YYYY-MM-DD — self-expiring; eligible for reaping after the date passes.
• delete_protection=true — indefinite; requires manual removal before reaping.

Warning: the daily cleanup workflow (zfnd-delete-gcp-resources.yml) sweeps PR-deploy resources by age and name and does not honor these labels. They only take effect with the label-aware sweep recipe below, so don’t rely on a label alone to protect a PR deploy from the daily sweep.

Apply to the instance and disk now if you need protection immediately (labels propagate to new instances on the next template swap):

MIG=zebrad-my-branch-mainnet-b
ZONE=us-east1-b
INSTANCE=$(gcloud compute instance-groups managed list-instances "$MIG" \
  --zone "$ZONE" --project $P --format='value(instance.basename())' | head -1)
gcloud compute instances add-labels "$INSTANCE" --zone "$ZONE" --project $P \
  --labels="keep_until=2026-05-01"
gcloud compute disks add-labels "zebrad-cache-my-branch-mainnet-b" \
  --zone "$ZONE" --project $P \
  --labels="keep_until=2026-05-01"

Reap one

The stateful policy auto-delete=on-permanent-instance-deletion removes the disk with the MIG.

gcloud compute instance-groups managed delete "$MIG" --zone "$ZONE" --project $P --quiet

Sweep expired

TODAY=$(date +%Y-%m-%d)
for line in $(gcloud compute instance-groups managed list --project $P \
    --filter="labels.created_by=workflow_dispatch" \
    --format='csv[no-heading](name,zone.basename(),labels.keep_until,labels.delete_protection)' \
    | awk -F, -v today="$TODAY" '$4 == "true" || $3 == "" { next } $3 < today { print $1 "|" $2 }'); do
  MIG="${line%|*}"
  ZONE="${line#*|}"
  echo "Reaping $MIG in $ZONE"
  gcloud compute instance-groups managed delete "$MIG" --zone "$ZONE" --project $P --quiet
done

Review the candidate list before piping into delete.

Diagnose a stuck MIG

MIG=zebrad-main-mainnet-b
ZONE=us-east1-b

gcloud compute instance-groups managed describe "$MIG" --zone "$ZONE" --project $P \
  --format="value(currentActions,status.isStable,status.versionTarget.isReached)"
gcloud compute instance-groups managed list-instances "$MIG" --zone "$ZONE" --project $P \
  --format="table(NAME,STATUS,HEALTH_STATE,LAST_ERROR)"
gcloud compute instance-groups managed list-errors "$MIG" --zone "$ZONE" --project $P --limit=10

When list-errors reports RESOURCE_IN_USE_BY_ANOTHER_RESOURCE on a stateful disk, find the squatter:

DISK=zebrad-cache-main-mainnet-b
gcloud compute disks describe "$DISK" --zone "$ZONE" --project $P \
  --format="value(users.basename())"

The squatter is another MIG’s instance. Reap that MIG (recipe above).

Recover a corrupted cache disk

When Zebra crash-loops on a zonal disk, recover that one zone from the last good cache image. Sister zones keep serving.

NET=mainnet; ZONE=us-east1-b; ZONE_LETTER=b
MIG=zebrad-main-${NET}-${ZONE_LETTER}
DISK=zebrad-cache-main-${NET}-${ZONE_LETTER}
TS=$(date +%Y%m%d-%H%M)

# 1. Drain: preserve the disk via auto-delete=never, then scale to 0
INSTANCE=$(gcloud compute instance-groups managed list-instances "$MIG" \
  --zone "$ZONE" --project $P --format='value(name)' | head -1)
gcloud compute instance-groups managed instance-configs update "$MIG" \
  --zone "$ZONE" --project $P --instance "$INSTANCE" \
  --stateful-disk "device-name=${DISK},auto-delete=never"
gcloud compute instance-groups managed resize "$MIG" --size 0 --zone "$ZONE" --project $P

# 2. Snapshot the corrupted disk for forensics, then delete it
gcloud compute snapshots create "${DISK}-corrupted-${TS}" \
  --source-disk "$DISK" --source-disk-zone "$ZONE" --project $P
gcloud compute disks delete "$DISK" --zone "$ZONE" --project $P --quiet

# 3. Redeploy via workflow_dispatch for this single zone
gh workflow run zfnd-deploy-nodes-gcp.yml -R ZcashFoundation/zebra \
  -f network=Mainnet -f zone="$ZONE" -f environment=dev \
  -f need_cached_disk=true -f cached_disk_type=tip

For production: set P=zfnd-prod-zebra, MIG=zebrad-${NET}-${ZONE_LETTER}, DISK=zebrad-cache-${NET}-${ZONE_LETTER}. Production has no automatic cache image; restore from a recent operator-taken snapshot.

Bootstrap a disk without a cache image

A deploy seeds a new zone’s disk from a cache image that matches the commit’s state format. When none exists, for example after a major format change on next, zones that have a disk still roll out, and a zone without one fails. Create that disk from the newest zebrad-cache-main-* image, named and sized like the other disks, then rerun the deploy. If Zebra can restore the image’s older major format, it migrates that state into the new format. Otherwise it syncs the new format from genesis.

DB-format-version-break release

When zebra-state/src/constants.rs::DATABASE_FORMAT_VERSION changes in a backwards-incompatible way, the in-place rolling swap fails because the new Zebra can’t read the old RocksDB. Do it one zone at a time with snapshot-based handoff:

P=zfnd-prod-zebra; NET=mainnet; TS=$(date +%Y%m%d-%H%M)

for Z in b c d; do
  MIG=zebrad-${NET}-${Z}; DISK=zebrad-cache-${NET}-${Z}; ZONE=us-east1-${Z}

  # Snapshot before touching
  gcloud compute snapshots create "${DISK}-pre-major-${TS}" \
    --source-disk "$DISK" --source-disk-zone "$ZONE" --project $P

  # Drain, preserving disk
  INSTANCE=$(gcloud compute instance-groups managed list-instances "$MIG" \
    --zone "$ZONE" --project $P --format='value(name)' | head -1)
  gcloud compute instance-groups managed instance-configs update "$MIG" \
    --zone "$ZONE" --project $P --instance "$INSTANCE" \
    --stateful-disk "device-name=${DISK},auto-delete=never"
  gcloud compute instance-groups managed resize "$MIG" --size 0 --zone "$ZONE" --project $P
done

# Publish the release; the workflow's release path recreates each zonal MIG
# with the new template, which attaches the preserved disks.

# Rollback (only if the release fails): restore disks from snapshots, deploy previous tag
for Z in b c d; do
  DISK=zebrad-cache-${NET}-${Z}; ZONE=us-east1-${Z}
  gcloud compute disks delete "$DISK" --zone "$ZONE" --project $P --quiet
  gcloud compute disks create "$DISK" --zone "$ZONE" --project $P \
    --source-snapshot "${DISK}-pre-major-${TS}"
done

Regional-to-zonal migration

One-shot procedure to convert an environment from the old regional architecture (one regional MIG per network holding three instances) to zonal (three zonal MIGs per network, one instance each). Run once per environment.

P=zfnd-prod-zebra   # or zfnd-dev-zebra
NET=mainnet
OLD_MIG=zebrad-${NET}    # for prod; use zebrad-main-${NET} for dev
OLD_DISK=zebrad-cache-${NET}    # for prod; use zebrad-cache-main-${NET} for dev
TS=$(date +%Y%m%d-%H%M)

# 1. Snapshot each zonal disk (three per network)
for Z in b c d; do
  gcloud compute snapshots create "${OLD_DISK}-${Z}-pre-zonal-${TS}" \
    --source-disk "$OLD_DISK" --source-disk-zone "us-east1-${Z}" --project $P
done

# 2. Delete the old regional MIG (disks auto-deleted; we have snapshots)
gcloud compute instance-groups managed delete "$OLD_MIG" \
  --region us-east1 --project $P --quiet

# 3. Create zonal disks from the snapshots with the new name scheme
#    Prod:    zebrad-cache-${NET}-${Z}
#    Staging: zebrad-cache-main-${NET}-${Z}
for Z in b c d; do
  NEW_DISK=zebrad-cache-${NET}-${Z}    # or zebrad-cache-main-${NET}-${Z}
  gcloud compute disks create "$NEW_DISK" \
    --zone "us-east1-${Z}" --project $P \
    --source-snapshot "${OLD_DISK}-${Z}-pre-zonal-${TS}" \
    --size 400 --type pd-balanced
done

# 4. Trigger workflow_dispatch for each (network, zone). The workflow sees
#    the zonal disks exist and attaches them; no fresh bootstrap from cache.
for Z in b c d; do
  gh workflow run zfnd-deploy-nodes-gcp.yml -R ZcashFoundation/zebra \
    -f network=Mainnet -f zone="us-east1-${Z}" \
    -f environment=$([ "$P" = "zfnd-prod-zebra" ] && echo prod || echo dev) \
    -f need_cached_disk=false
done

Repeat for Testnet. After all six (2 networks × 3 zones) zonal MIGs are HEALTHY, the migration for this environment is complete; retire the pre-zonal snapshots after a hold period.

Reference

Static IPs are externally provisioned and map deterministically to zones:

• us-east1-b → zebra-${network} (primary)
• us-east1-c → zebra-${network}-secondary
• us-east1-d → zebra-${network}-tertiary

Stage and production use these names in their own projects. next shares stage’s project, so its addresses are zebra-next-${network}, zebra-next-${network}-secondary, and zebra-next-${network}-tertiary.

The workflow assigns them via per-instance configs whenever the caller passes use_reserved_ip: always for push and release, and for a workflow_dispatch that selects prod or next, or runs from the main or next ref. Three dispatch cases to watch:

• Selecting prod updates the stable production MIG and its reserved IP for the selected network and zone, from whatever ref you dispatched.
• Selecting next updates the next MIG and its reserved IP the same way, from whatever ref you dispatched.
• Selecting dev from the main or next ref resolves to the main- or next- prefix, so it updates the stage or next MIG and its reserved address — not a branch-private one.

Every other dev workflow_dispatch gets no external IP at all: the instance template passes --no-address, so those instances reach the network outbound-only and never advertise a ZEBRA_NETWORK__EXTERNAL_ADDR. Reserve new addresses manually with gcloud compute addresses create before adding capacity.

Assignment only happens when the MIG is first created. If instance-configs create fails, or the address was not reserved at the time (the step warns and continues), a later rerun will not repair the binding — the MIG already exists, so the step is skipped — while the template may still carry ZEBRA_NETWORK__EXTERNAL_ADDR. Repair it by hand with gcloud compute instance-groups managed instance-configs create.

Cache images are produced by zfnd-deploy-integration-tests-gcp.yml’s create-state-image job. Naming pattern: {prefix}-{branch}-{sha}-v{state-version}-{network}-{tip|checkpoint}[-u]-{HHMMSS}. One image per network seeds all three zones. Lookup priority: current branch, then main, then any branch; most recent first. List recent:

gcloud compute images list --project zfnd-dev-zebra \
  --filter="name~^zebrad-cache-.*-v27-mainnet-tip" \
  --sort-by=~creationTimestamp --limit=5

Sentry releases use the name deployed nodes report, for example 7.0.0-rc.0+git.6d1e414. After a stage or prod deploy succeeds, the sentry-release-name job reads that name from the deployed image with zebrad --version. The sentry-release job then creates the release in the zfnd/zebrad Sentry project, associates the commits since the previous successful deploy to that environment through Sentry’s GitHub integration, and records a deploy. It needs the SENTRY_AUTH_TOKEN repository secret, an organization token. Without the secret, the job skips its steps and records nothing. Dev and next deploys are not recorded.

Daily cleanup (zfnd-delete-gcp-resources.yml) sweeps old instances, templates, disks, and cache images by age and name. It does not honor keep_until or delete_protection labels on PR deploys; use the sweep recipe for label-aware control. The eighteen stable cache disks (zebrad-cache-{mainnet,testnet}-{b,c,d} in production, zebrad-cache-main-{mainnet,testnet}-{b,c,d} in stage, and zebrad-cache-next-{mainnet,testnet}-{b,c,d} in next) are never eligible because GCP refuses to delete attached disks; if any stable disk ever shows up unattached, that is itself an incident.