# Provider Outage Runbook

Owner: Operations on-call
Severity: SEV1 when customer money movement, cards, KYC/AML, custody, FX or login is blocked.

## Triggers

- `BankingProviderUnavailable`
- `BankingKYCAMLProviderErrors`
- sustained provider timeout/error spikes
- provider status page or account manager confirms degradation

## First 15 Minutes

1. Create an incident record with request IDs, trace IDs, provider, environment and first alert timestamp.
2. Freeze risky outbound retries if the provider confirms partial processing or duplicate-risk behavior.
3. Switch affected feature flags to degraded mode or disabled mode when customer harm is possible.
4. Notify provider support with correlation IDs, timestamps, idempotency keys and affected references.
5. Post internal status update to operations, compliance, support and product.

## Diagnosis

- Check API health, provider call panel, transfer/card/KYC/FX panels and recent deploys.
- Sample redacted logs by `request_id` and `trace_id`.
- Confirm whether requests failed before submission, after submission or with unknown provider outcome.
- For money movement, reconcile affected provider references before replaying.

## Recovery

- Resume traffic gradually after provider confirms stability.
- Replay only idempotent calls with approved keys and documented outcome checks.
- Run reconciliation for affected provider/account/currency ranges.
- Attach evidence: alert timeline, provider ticket, logs, reconciliation result and customer-impact estimate.

## Evidence

- alert timeline
- provider ticket and status-page references
- redacted logs with request IDs and trace IDs
- reconciliation result for affected references
- customer-impact estimate and notification decision

## Communications

- Customers: send impact, affected services, workaround and resolved notice when customer-facing impact exists.
- Providers: keep the provider ticket updated every 30 minutes for SEV1.
- Regulators: use `docs/runbooks/major-incident-notification.md` if incident thresholds are met.