> For the complete documentation index, see [llms.txt](https://valargroup.gitbook.io/shielded-vote-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://valargroup.gitbook.io/shielded-vote-docs/operations/snapshot-bumps.md).

# Bumping the PIR fleet to a new snapshot height

This runbook covers the end-to-end procedure for moving the entire PIR serving fleet (currently `pir-primary` + `pir-backup`) from one Orchard nullifier-tree snapshot height to a new one.

The Phase 3 redesign (`vote-nullifier-pir/nf-server` ≥ v0.1.x) makes this a three-step sequence anchored on a single source of truth: [`token-holder-voting-config/voting-config.json`](https://github.com/valargroup/token-holder-voting-config/blob/main/voting-config.json)’s `snapshot_height` field. Every PIR replica self-bootstraps from the published bucket on startup keyed by that height; the operator's job is to (1) publish a snapshot at the new height, (2) bump the config, and (3) trigger a rolling restart.

## When to bump

Bumps are scheduled — not opportunistic. The chain pins on-chain `snapshot_height` per voting round when a new round is created, and **every PIR replica must be serving a snapshot ≥ that height before the round opens**. The wall-clock window is "after the previous round closes, before the next round's snapshot height is committed on-chain".

Coordinate with whoever creates rounds (the EA / chain admin) so the new published snapshot exists before they sign the round-creation message.

## Preconditions

* All PIR replicas are healthy on the **current** snapshot:

  ```
  curl https://pir-primary.<domain>/root | jq .
  curl https://pir-backup.<domain>/root  | jq .
  ```

  Both should report the current `voting-config.snapshot_height`.
* The publisher host (`PIR_BACKUP_HOST`, by convention the backup replica) has up-to-date `nf-server` ingest data and the `nf-server` binary.
* You have merge access to [`token-holder-voting-config`](https://github.com/valargroup/token-holder-voting-config) and `workflow_dispatch` access to [`vote-nullifier-pir`](https://github.com/valargroup/vote-nullifier-pir).
* You can reach the admin UI at <https://svote.valargroup.org/settings> (used for the step-5 convergence check).

## Procedure

### 1. Pick the new height

Snapshot heights are rounded down to the nearest multiple of 10 by the publisher CI (`publish-snapshot.yml`). Pick a height that:

* is at least the current Zcash chain tip minus a comfortable confirmation margin (e.g. tip − 10 blocks),
* is strictly greater than the current `snapshot_height`,
* lines up with whatever the chain admin needs for the upcoming round.

```bash
TIP=$(curl -sf https://api.blockchair.com/zcash/stats | jq -r '.data.best_block_height')
NEW_H=$(( ((TIP - 10) / 10) * 10 ))
echo "New snapshot height: $NEW_H"
```

### 2. Publish the snapshot

Trigger [`publish-snapshot.yml`](https://github.com/valargroup/vote-nullifier-pir/actions/workflows/publish-snapshot.yml) manually with the chosen height:

* GitHub UI → Actions → "Publish nullifier snapshot" → Run workflow → set `height` to `$NEW_H`.
* Or via CLI:

  ```bash
  gh workflow run publish-snapshot.yml \
      --repo valargroup/vote-nullifier-pir \
      -f height=$NEW_H
  ```

The job runs on `PIR_BACKUP_HOST` and:

1. ingests nullifiers up to `$NEW_H` from lightwalletd,
2. exports tier files into `/tmp/snap-$NEW_H/`,
3. writes a `manifest.json` with sha256s of every file,
4. uploads to `s3://shielded-vote/snapshots/$NEW_H/{tier0.bin,tier1.bin,tier2.bin,pir_root.json,manifest.json}` (manifest last — partial uploads stay invisible to clients),
5. round-trip verifies the manifest and `tier0.bin` over HTTPS.

Expect 30–90 minutes end-to-end depending on chain depth and bucket region. The job is **non-disruptive** — the publisher host's own `nullifier-query-server` keeps serving the old snapshot throughout because the build happens in `/tmp` and never touches `pir-data/`.

When it finishes, confirm the artifacts exist:

```bash
curl -sfI https://shielded-vote.nyc3.digitaloceanspaces.com/snapshots/$NEW_H/manifest.json
curl -sfI https://shielded-vote.nyc3.digitaloceanspaces.com/snapshots/$NEW_H/tier0.bin
```

### 3. Bump the published config

Open a PR against [`token-holder-voting-config/voting-config.json`](https://github.com/valargroup/token-holder-voting-config/blob/main/voting-config.json) that updates only `snapshot_height` (every other field stays as-is):

```json
{
  "config_version": 1,
  "vote_servers":   [...],
  "pir_endpoints":  [...],
  "snapshot_height": <NEW_H>,
  "supported_versions": { ... }
}
```

Merge to `main`. The CDN cache (GitHub Pages) typically refreshes within a minute; verify with:

```bash
curl -s https://valargroup.github.io/token-holder-voting-config/voting-config.json \
    | jq '.snapshot_height'
```

### 4. Roll the PIR fleet

`nf-server` reads `voting-config.snapshot_height` on startup and self-bootstraps from the bucket if its local `pir_root.json` is at a different height. So the entire bump-the-fleet step is just a rolling restart, wrapped by the [`Restart PIR fleet`](https://github.com/valargroup/vote-nullifier-pir/actions/workflows/restart.yml) workflow:

* GitHub UI → Actions → "Restart PIR fleet" → Run workflow → leave `targets=both` and `verify_height_converged=true`.
* Or via CLI:

  ```bash
  gh workflow run restart.yml \
      --repo valargroup/vote-nullifier-pir \
      -f targets=both
  ```

The workflow restarts **backup first**, polls `/health`, and verifies `nf_snapshot_served_height == nf_snapshot_expected_height` from the `/metrics` endpoint before letting the primary job start. Primary is gated on backup succeeding, so the fleet never loses both replicas at once — if backup fails to converge, primary is **not** touched and you get a red job with the systemd/journal output dumped into the log.

See the [per-host restart runbook](https://github.com/valargroup/vote-nullifier-pir/blob/main/docs/runbooks/restart-pir-fleet.md) in `vote-nullifier-pir` for inputs, failure modes, and an SSH fallback for when Actions is unavailable:

```bash
ssh pir-backup.<domain>  sudo systemctl restart nullifier-query-server
# Wait for backup to reach the new height before touching primary so we
# never have two replicas down at once.
ssh pir-primary.<domain> sudo systemctl restart nullifier-query-server
```

For each restart, the systemd unit:

1. starts `nf-server serve`,
2. fetches `voting-config.json`, sees the new `snapshot_height`,
3. compares it to local `pir_root.json` (mismatch),
4. downloads `manifest.json` + `tier*.bin` + `pir_root.json` from the bucket, verifies sha256s, swaps them into `--pir-data-dir`,
5. loads tier files into memory and starts answering PIR queries.

The bootstrap typically completes in under a minute on a same-region download from the `nyc3` Space; observed cold-starts on the production droplets run 150–170 s end-to-end because `load_serving_state` mmaps and parses \~6 GB of tier files into memory after the download. Watch progress from the workflow run page, or directly via the service log / metrics endpoint:

```bash
ssh pir-backup.<domain> "journalctl -u nullifier-query-server -f"
```

```bash
curl -s http://localhost:9100/proxy/pir-backup/metrics \
    | grep -E 'nf_snapshot_(served|expected)_height|nf_snapshot_bootstrap_'
```

### 5. Confirm convergence

Once the restart workflow is green, confirm both replicas are actually serving the new snapshot. The fastest path is the **admin UI**: open <https://svote.valargroup.org/settings> and scroll to the **Nullifier service (PIR)** card. It queries each configured endpoint's `/root` and renders a side-by-side row per endpoint with `height` / `root25` / `root29` / `num_ranges`, plus a banner if any of those disagree across replicas. Every row must show `height == $NEW_H` and identical `root25` / `root29` strings before declaring the bump complete.

> **Note:** the side-by-side / roots view is [`vote-sdk#119`](https://github.com/valargroup/vote-sdk/issues/119); until it ships, the UI shows only the currently-selected endpoint's `height` and `num_ranges`, which is enough for a quick smoke test but doesn't cross-check `root25` / `root29` between replicas. Use the shell fallback below as the authoritative check in the meantime.

**Shell fallback** (no browser, or until the admin UI enhancement lands):

```bash
for host in pir-primary pir-backup; do
    echo "=== $host ==="
    curl -s "https://$host.<domain>/root" | jq '.height, .root25, .root29'
done
```

Both replicas must report `.height == $NEW_H` and identical `.root25` / `.root29` strings before declaring the bump complete.

### 6. Acknowledge the snapshot-stale alert (if it fired)

If a bump takes longer than 30 minutes to converge — for example because the publisher run was slow, or because you bumped the config before publishing — the per-host watchdog will fire a Sentry error event tagged `alert:snapshot_stale` and route it to Slack. After convergence, the watchdog emits a follow-up info event ("snapshot height converged: gap closed after Ns") so the on-call engineer can verify the issue resolved itself.

The 30-minute window is set by `SVOTE_STALE_THRESHOLD_SECS` in `/etc/default/nf-server`. See [the deploy guide](https://github.com/valargroup/vote-nullifier-pir/blob/main/docs/deploy-setup.md#snapshot-stale-alerting) for the Sentry alert-rule configuration and a verification fire procedure.

## Failure modes

| Symptom                                                                                                            | Cause                                                                                                                           | Recovery                                                                                                                                                                                                                                                     |
| ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `nf_snapshot_bootstrap_outcomes_total{result="fell_through"}` increments and `nf_snapshot_served_height` stays old | Network failure fetching voting-config or bucket; or sha256 mismatch on a tier file (corrupted upload)                          | Re-run `publish-snapshot.yml` if the artifacts are bad. For transient network errors, simply restart the unit again — bootstrap retries from scratch on every startup.                                                                                       |
| Replica refuses to start, log shows `tier1.bin size mismatch`                                                      | Locally cached `pir-data/` is from a partial bootstrap or a different `nf-server` build                                         | `rm -rf /opt/nf-ingest/pir-data/*` and restart — bootstrap will repopulate from the bucket.                                                                                                                                                                  |
| `nf_snapshot_expected_height` is `0` after restart                                                                 | `voting-config.json` has no `snapshot_height` field, or the URL was unreachable                                                 | Verify the published config has `snapshot_height`. If unreachable, check `SVOTE_VOTING_CONFIG_URL` in `/etc/default/nf-server`.                                                                                                                              |
| Bump merged but bucket has nothing at `snapshots/<NEW_H>/`                                                         | Step 3 ran before step 2; replicas keep falling through to the old snapshot                                                     | Run step 2, then restart the replicas. The old snapshot keeps serving the whole time.                                                                                                                                                                        |
| Sentry/Slack `alert:snapshot_stale` fires for one host but the other is fine                                       | That host fell through and is still serving the old height (`nf_snapshot_served_height < nf_snapshot_expected_height`) for >30m | Check `journalctl -u nullifier-query-server` on the affected host, fix whatever blocked the bootstrap (network, missing artifact, full disk), then `systemctl restart nullifier-query-server`. The watchdog will emit a follow-up info event on convergence. |

## Rolling back

If the new snapshot is broken (wrong root, corrupted upload), the fastest rollback is to revert the `voting-config.json` PR and restart the fleet. Replicas will compare local height (now `$NEW_H`) against expected (back to old) and re-bootstrap from the bucket — provided the old `snapshots/<old_H>/` artifacts are still there (DO Spaces does not TTL them, so they remain available indefinitely unless explicitly deleted).

## Why this is safe to do during traffic

* The bucket-publish step never touches the live `pir-data/` on any replica.
* The bootstrap on each replica downloads into `.bootstrap-staging/` and only renames into place once every sha256 verifies, with `pir_root.json` last so a half-applied bootstrap is detected and retried on the next startup.
* The rolling restart keeps at least one replica online at all times, so the admin UI's PIR endpoints (`pir-primary`, `pir-backup`) continue to answer queries throughout the bump.

## Related docs

* [Admin UI → Settings → Nullifier service (PIR)](https://svote.valargroup.org/settings) — side-by-side convergence check used in step 5.
* [`vote-nullifier-pir/nf-server/src/bootstrap.rs`](https://github.com/valargroup/vote-nullifier-pir/blob/main/nf-server/src/bootstrap.rs) — bootstrap implementation.
* [`vote-nullifier-pir/.github/workflows/publish-snapshot.yml`](https://github.com/valargroup/vote-nullifier-pir/blob/main/.github/workflows/publish-snapshot.yml) — publisher CI.
* [`vote-nullifier-pir/.github/workflows/restart.yml`](https://github.com/valargroup/vote-nullifier-pir/blob/main/.github/workflows/restart.yml) — rolling-restart CI used in step 4.
* [`vote-nullifier-pir/docs/runbooks/restart-pir-fleet.md`](https://github.com/valargroup/vote-nullifier-pir/blob/main/docs/runbooks/restart-pir-fleet.md) — per-host restart runbook (inputs, failure modes, SSH fallback).
* [`vote-infrastructure/cloud-init/pir.yaml`](https://github.com/valargroup/vote-infrastructure/blob/main/cloud-init/pir.yaml) — PIR host provisioning.
* [`token-holder-voting-config/README.md`](https://github.com/valargroup/token-holder-voting-config/blob/main/README.md) — published config schema.