# Observability, Incident Response, And Resilience

This document maps the production observability and DORA-style resilience controls for the banking app.

## Runtime Signals

- Metrics: `/metrics`, protected by internal CIDR allowlists and reverse-proxy rules.
- Logs: redacted `slog` output with `request_id` and `trace_id`.
- Correlation: inbound `X-Request-ID`, `X-Correlation-ID` and W3C `traceparent` are preserved or generated, returned to clients and injectable into outbound provider calls through `platform/correlation`.
- Dashboards: `deploy/observability/grafana-banking-overview.json`.
- Alert rules: `deploy/observability/prometheus-rules.yml`.
- Alert routing: `deploy/observability/alertmanager.yml`.

## Dashboard Coverage

The Grafana dashboard covers:

- API health: request rate, error ratio and latency.
- DB health: PostgreSQL availability and connection pressure.
- Ledger: postings and imbalance indicator.
- Transfers: pending age and transfer status counts.
- Settlements: failed/retried/completed settlement events.
- Cards: card status and issuer webhook rate.
- KYC/AML: provider call errors and throughput.
- FX: stale rates and quote volume.
- Auth: suspicious login and auth route status.
- Provider calls: availability and error status by provider.

## Metric Contract

The app already exports HTTP, uptime and provider executor metrics. Provider adapters, SQL exporters or scheduled monitoring jobs must publish the remaining business metrics referenced by the dashboard and alert rules before live production, including:

- `banking_ledger_postings_total`
- `banking_ledger_imbalanced_currencies`
- `banking_transfers_total`
- `banking_transfers_pending_oldest_age_seconds`
- `banking_settlement_events_total`
- `banking_cards_total`
- `banking_provider_calls_total`
- `banking_provider_availability`
- `banking_auth_suspicious_logins_total`
- `banking_fx_quotes_total`
- `banking_fx_stale_rates`

## Incident Runbooks

- Provider outage: `docs/runbooks/provider-outage.md`
- Settlement break: `docs/runbooks/settlement-break.md`
- Ledger imbalance: `docs/runbooks/ledger-imbalance.md`
- Data breach: `docs/runbooks/data-breach.md`
- Auth attack: `docs/runbooks/auth-attack.md`
- Card/webhook storm: `docs/runbooks/card-webhook-storm.md`
- Major incident notifications: `docs/runbooks/major-incident-notification.md`
- Business continuity and disaster recovery: `docs/runbooks/business-continuity-disaster-recovery.md`
- Controlled deploy: `docs/runbooks/deploy.md`
- Application, frontend and database rollback: `docs/runbooks/rollback.md`
- Reconciliation break: `docs/runbooks/reconciliation-break.md`

## Production Config Gates

Preprod and production require:

- `BANKING_METRICS_ENABLED=true`
- `BANKING_STRUCTURED_LOGS_ENABLED=true`
- `BANKING_REQUEST_CORRELATION_ENABLED=true`
- `BANKING_OBSERVABILITY_DASHBOARD_PACK`
- `BANKING_OBSERVABILITY_ALERT_RULES`
- `BANKING_INCIDENT_RUNBOOK_PACK`
- `BANKING_MAJOR_INCIDENT_CONTACTS`
- `BANKING_BCDR_TEST_REFERENCE`
- `BANKING_CHAOS_TEST_PLAN`

## DORA-Style Evidence

Attach the following evidence to every major resilience drill or incident:

- incident timeline with UTC timestamps
- severity and materiality decision
- request IDs and trace IDs for sample flows
- dashboard screenshots or exported panel links
- alert firing and routing evidence
- provider/regulator/customer notification decisions
- recovery validation, including ledger and reconciliation checks
- remediation owner and due date