# Rollback Runbook

Owner: Release manager
Approvers: Engineering and operations; database owner for schema/data action; finance for money impact
Severity: SEV1 when funds, ledger, authorization, secrets, or regulated data may be wrong

Rollback restores a previously verified application state. It must not erase
financial history, replay unknown provider operations, or apply an unreviewed
database down migration.

## Triggers

- Deployment abort criteria from `docs/runbooks/deploy.md` are met.
- Health, latency, error, queue, or provider metrics exceed the release threshold.
- A security, privacy, authorization, data-integrity, or financial regression is
  confirmed or cannot be safely excluded.
- The frontend and API contract are incompatible for a critical workflow.
- A feature flag cannot contain impact without leaving inconsistent state.

## First 10 Minutes

1. Appoint the rollback decision owner and start an incident timeline in UTC.
2. Stop canary progression and unrelated deployments. Preserve logs, traces,
   request IDs, provider references, migration output, and current digests.
3. Disable only affected high-risk features when doing so prevents further harm.
4. If money or provider execution is uncertain, stop automated retries and hold
   affected items for reconciliation. Do not replay them during rollback.
5. State the rollback target: API image digest, frontend artifact, configuration,
   feature flags, and database compatibility.

## API Rollback

1. Select the last signed, tested, schema-compatible API digest from release
   evidence.
2. Verify its Cosign signature/provenance and confirm it supports the current
   database schema.
3. Use the manual CD workflow with `rollback_image` set to that immutable digest.
4. Roll back canary/green traffic first, verify readiness, then shift remaining
   traffic.
5. Do not run old startup migrations. `BANKING_RUN_MIGRATIONS` remains false unless
   the database owner approves a specific migration action.

## Frontend Rollback

1. Select the last approved frontend artifact and checksum that matches the target
   API contract.
2. Redeploy the complete immutable artifact; do not mix files from two builds.
3. Point the release/CDN alias to that version and invalidate only mutable entry
   documents when required.
4. Verify login, token refresh, customer navigation, transfer step-up, and affected
   admin workflows on desktop and mobile.
5. Record the artifact checksum, cache action, and observed client version.

The current repository's automated `rollback_image` covers the API. Production
CD must automate the frontend artifact selection/promotion described above before
go-live; until then the release manager records and executes it as a controlled
environment procedure.

## Database And Migration Decision

Prefer a backward-compatible application rollback or reviewed roll-forward repair.
A `.down.sql` file proves coverage, not that production down migration is safe.

1. Database owner classifies the migration as schema-only, additive, destructive,
   data-transforming, or externally observed.
2. Confirm whether the previous API can run against the current schema.
3. If compatible, leave the schema in place and roll back application code.
4. If incompatible, choose an approved roll-forward compatibility migration unless
   the reviewed release plan explicitly proves safe down migration.
5. Before any data/schema reversal, capture backup/PITR reference, migration plan,
   row counts/checksums, current ledger/reconciliation state, and approvals.
6. Never restore the entire database over live production merely to undo an
   application release. PITR is a disaster-recovery decision with incident and
   reconciliation ownership.
7. Never delete or rewrite journal, audit, provider-event, settlement, or regulatory
   evidence to make the old application fit.

Use `docs/runbooks/database-migration-release.md` for database execution controls
and `docs/runbooks/database-backup-restore-drill.md` for restore validation.

## Configuration And Feature Flags

- Restore the exact previous approved values from release evidence.
- Rotate a credential instead of restoring it when exposure is suspected.
- Use maker-checker for risk, routing, provider, security, card, crypto, privacy,
  or rate-limit configuration changes.
- Record actor, reason, before/after value, approval reference, and effective time.

## Post-Rollback Verification

- All API replicas and worker versions match the rollback target.
- The frontend checksum and API contract are compatible.
- Health/readiness, login/session/MFA, and affected workflows pass smoke checks.
- Provider retries and callback processing remain paused until unknown outcomes
  are resolved, then resume gradually.
- Trial balance is zero by currency; account/wallet balances match ledger; no new
  duplicate external references exist.
- Reconciliation runs cover the release window and affected providers/accounts.
- Security/privacy owner confirms containment when rollback followed an exposure.

## Communications And Closure

- Give support confirmed customer impact, affected window, safe workaround, and
  approved resolution wording.
- Use the major incident notification runbook when thresholds are met.
- Attach incident timeline, old/new/rollback digests, artifacts, config values,
  approvals, migration decision, smoke results, financial reconciliation, customer
  communications, and follow-up owners.
- Reopening traffic requires release manager, operations, and finance signoff when
  financial state was at risk.