# Chaos And Failure Injection Tests

Owner: Platform operations
Cadence: quarterly for staging/preprod, before major provider or infrastructure changes.

## Safety Rules

- Run only in staging or preprod with test money.
- Announce scope, start time, rollback owner and abort criteria.
- Disable tests immediately if customer-facing production, real-money providers or shared secrets are affected.

## Required Scenarios

| Scenario | Injection | Expected controls |
| --- | --- | --- |
| Provider outage | force provider HTTP timeout or return 503 in sandbox adapter | circuit breaker/degraded mode, alert, runbook, no duplicate money movement |
| Provider ambiguous result | return timeout after accepting request | hold/reconcile before replay, idempotency evidence |
| DB failover | restart or fail over staging PostgreSQL primary | `/readyz` fails then recovers, no unbalanced journals |
| DB read latency | add latency to database network path | latency alert, graceful API timeout, no partial ledger posting |
| Webhook replay storm | replay card/KYC/provider webhooks out of order | signature, idempotency and ordering checks hold |
| Logging redaction | inject payload with email, token and card-like data | logs contain request ID/trace ID and redacted sensitive values |

## Evidence

- test ticket or change reference
- start/end timestamps
- alerts fired and routes used
- dashboards screenshots or exported panels
- request IDs and trace IDs from sample flows
- abort/result decision and follow-up actions