# Deployment Runbook

Owner: Release manager
Approvers: Engineering, operations, database owner; security/compliance/finance when impacted

Use this runbook for staging, pre-production, and production deployments. Production
deployments must use immutable signed image digests and protected environment
approval. A successful HTTP deploy is not complete until financial and provider
checks pass.

## Entry Criteria

- CI passed for the exact commit: backend tests/vet, migration plan, PostgreSQL
  integration tests, frontend build/e2e/accessibility, security scans, and image
  scan.
- Release notes identify risk, migrations, feature flags, provider changes,
  verification, rollback criteria, and approvers.
- Target environment uses non-default managed secrets and approved KMS keys.
- The image digest has provenance and a valid Cosign signature.
- Migration dry-run, rollback coverage, backward compatibility, current backup,
  and restore reference are approved when the schema changes.
- On-call, finance/operations, support, dashboards, and alert routes are ready for
  the change window.
- There is no unresolved SEV1/SEV2 or unexplained ledger/reconciliation break.

## Pre-Deploy Record

Record in the release evidence package:

- release/tag, commit SHA, image digest, frontend artifact checksum
- target environment, window, release manager, deployer, and approvers
- migration-plan artifact and database approval reference
- current API and frontend version/digest
- feature flags before and after deployment
- dashboard baseline and open-alert/reconciliation summary
- rollback image/artifact and decision owner

## Deployment

1. Announce the change window and freeze unrelated production changes.
2. Verify the target environment and current deployment one final time.
3. For schema changes, run the approved migration job once. Do not let every API
   replica race migrations in pre-production or production.
4. Deploy the API by immutable digest using the approved canary or blue/green
   strategy. Keep high-risk feature flags unchanged initially.
5. Wait for startup, liveness, and readiness checks. Confirm database connectivity,
   migration version, logs, metrics, and error ratio.
6. Run authenticated smoke checks for login/session, account read, and the changed
   surfaces. Use synthetic test users and test-money routes only.
7. Deploy the frontend artifact that was built and tested against this API
   contract. Invalidate the CDN cache by versioned asset name, not global deletion.
8. Increase canary traffic in approved stages while watching latency, errors,
   saturation, auth failures, provider calls, worker backlog, and state errors.
9. Enable changed high-risk feature flags only after their owner approves smoke
   and reconciliation evidence.
10. Run post-deploy financial checks: trial balance by currency, account/wallet
    projection checks, pending settlement age, provider snapshot freshness, and
    open reconciliation breaks.

Remote API deployment contract:

```bash
APP_IMAGE=ghcr.io/OWNER/REPO/banking-api@sha256:DIGEST docker compose pull api
APP_IMAGE=ghcr.io/OWNER/REPO/banking-api@sha256:DIGEST docker compose up -d api
```

Execute this only through the approved CD environment or controlled deployment
host. Do not substitute a mutable tag in production.

## Verification

- `/healthz` and `/readyz` are healthy on every new replica.
- The public edge serves valid TLS and expected security headers.
- `/metrics` remains inaccessible from the public internet and is scrapeable from
  the observability network.
- Login, refresh, logout, MFA/step-up, and admin authorization behave as expected.
- Changed API contracts pass the production-like smoke suite.
- Provider calls/webhooks use the expected environment and credentials; no
  production provider call originates from staging.
- Error rate, p95 latency, DB connection pressure, queue depth, and worker retries
  remain within the approved release thresholds.
- Ledger trial balance is zero per currency and no new unexplained reconciliation
  break exists.
- Customer and admin frontend load without console/network failures for critical
  workflows.

## Abort And Rollback Criteria

Stop traffic progression and invoke `docs/runbooks/rollback.md` for:

- any ledger imbalance, duplicate posting, wrong balance, or unapproved movement
- unexpected migration behavior or irreversible data-quality failure
- account takeover/control bypass, secret exposure, or authorization regression
- sustained error/latency threshold breach or crash loop
- provider duplicate-risk, unknown-outcome growth, or webhook incompatibility
- critical customer workflow failure without a safe feature-flag mitigation

Feature disabling is acceptable containment but does not by itself close the
deployment. Record the residual risk and either complete rollback or obtain an
explicit roll-forward decision.

## Closure Evidence

- CD run and protected-environment approvals
- image signature/provenance verification and artifact checksums
- migration execution/dry-run/backup references
- smoke output and dashboard snapshots before/during/after
- trial balance and reconciliation result
- feature-flag values and provider environment confirmation
- incidents, rollback decisions, deviations, owners, and follow-up dates
- release manager and operations signoff