# Reconciliation Break Runbook

Owner: Finance operations
Technical owner: Ledger/reconciliation engineering
Severity: Critical for ledger imbalance or unexplained customer-funds difference; high for provider mismatch above approved tolerance

This runbook handles breaks produced by daily reconciliation between internal
ledger/account/wallet state and provider snapshots. It complements the settlement
break runbook, which focuses on individual payment settlement outcomes.

## Triggers

- Daily reconciliation run fails or does not complete by its control deadline.
- New `critical` or `high` break appears in the reconciliation dashboard.
- Break type is `journal_entry_unbalanced`, `account_ledger_mismatch`,
  `wallet_available_mismatch`, `wallet_reserved_mismatch`,
  `provider_account_mismatch`, or `provider_snapshot_orphan`.
- Provider report/snapshot is missing, stale, duplicated, or for the wrong cutoff.
- Trial balance, end-of-day snapshot, customer display, provider balance, or finance
  report disagrees.

## First 15 Minutes

1. Create or sync a `reconciliation_break` backoffice case. Record run ID, break ID,
   provider, account/wallet, currency, expected/actual/difference, cutoff, and alert
   timestamp.
2. Set the break to `investigating`, assign finance and engineering owners, and set
   severity/SLA from the approved control matrix.
3. For ledger imbalance, wrong customer balance, or uncertain external execution,
   disable affected money movement and admin adjustments immediately.
4. Preserve provider files/payload hashes, ingestion results, EOD snapshots,
   journals, transfer/card/FX/crypto references, request/trace IDs, deploys, and
   configuration changes for the affected period.
5. Notify incident command for critical impact and payments/card/custody provider
   operations when an external balance is involved.

## Validate The Inputs

- Confirm environment, provider, legal entity/account, currency, timezone, cutoff,
  value-date convention, and report version all match.
- Verify source freshness, file/control totals, checksum/signature, duplicate
  ingestion, completeness, and opening/closing balance continuity.
- Confirm currency exponent, sign convention, fees, holds, pending items, reserved
  balance, and provider settlement timing are treated consistently.
- Re-run read-only calculations from the same immutable source data. Do not replace
  the original run or delete the original break.

## Diagnose By Break Type

| Break type | Primary comparison | Common causes |
| --- | --- | --- |
| `journal_entry_unbalanced` | Debits versus credits per journal/currency | Defective posting path, migration, manual SQL, partial transaction |
| `account_ledger_mismatch` | Account projection versus ledger-derived balance | Projection update failure, wrong account mapping, missing/repeated journal |
| `wallet_available_mismatch` | Wallet available total versus account/ledger state | Hold treatment, stale aggregate, cross-wallet ownership/mapping |
| `wallet_reserved_mismatch` | Reserved amount versus open holds/reservations | Hold release/expiry/reversal processed incorrectly |
| `provider_account_mismatch` | Latest provider snapshot versus internal provider account | Timing, fee, return, duplicate/missing submission, unknown outcome |
| `provider_snapshot_orphan` | Provider account/reference versus internal mapping | Wrong environment/entity, closed/unmapped account, malformed report |

Build a transaction-level bridge from the last matched cutoff to the broken cutoff:
opening balance, each internal posting, each provider movement, fees/returns, timing
items, and closing balance. Every line must have a stable internal or provider
reference and disposition.

## Resolution Paths

- **Timing difference:** Document expected clearing date/reference. Keep the break
  investigating until it naturally matches; do not mark resolved on assumption.
- **Missing/duplicate provider input:** Quarantine bad input, ingest through the
  idempotent approved path, and rerun reconciliation with source evidence.
- **External executed, internal missing:** Verify independently with the provider,
  then use the approved state transition and compensating journal workflow.
- **Internal posted, external not executed:** Hold retries until outcome is certain;
  use explicit reversal/refund only with finance approval.
- **Incorrect internal journal:** Never edit lines. Create an explicit linked
  reversal and corrected journal through the supported movement workflow.
- **Mapping/configuration defect:** Correct through maker-checker, rerun affected
  dates, and assess all records since the bad configuration became effective.
- **False positive:** `ignored` is allowed only for a documented, approved control
  exception with evidence and expiry/review date. It is not a shortcut for an
  unexplained difference.

`resolved` and `ignored` require a meaningful resolution note and named admin.
For high/critical breaks, require independent finance review even if the current
API records a single resolver.

## Verification Before Closure

- Corrective/reversal journals are balanced and link to the original movement.
- Account and wallet projections equal ledger-derived balances.
- Provider snapshot and internal provider account reconcile at the agreed cutoff.
- Trial balance is zero per currency and the EOD snapshot/report is regenerated.
- The break changes to `resolved` with root cause, actions, evidence references,
  approver, and prevention task; no sibling breaks remain for the same cause.
- A regression/invariant test exists for software defects.
- Customer remediation, interest/fee correction, and notification decisions are
  recorded where balances or availability were affected.

## Escalation And Communications

- Critical: incident commander, finance lead, engineering on-call, compliance, and
  executive owner immediately.
- High: finance and engineering immediately; compliance/support when customer or
  regulatory impact is possible.
- Provider mismatch: open provider ticket with cutoff, references, control totals,
  and redacted evidence; never send unrelated customer data.
- Customer messages state confirmed impact and action only. Do not speculate about
  fraud, sanctions, or regulatory reporting.
- Use `docs/runbooks/major-incident-notification.md` when materiality thresholds may
  be met and `docs/runbooks/ledger-imbalance.md` for any confirmed imbalance.

## Closure Evidence

- reconciliation run/break/backoffice case IDs and complete timeline
- immutable provider source checksum and ingestion/control-total evidence
- bridge analysis and affected customer/account/provider population
- original, reversal, and corrected journal references where applicable
- trial balance, EOD snapshot, rerun result, and reviewer approval
- customer/provider/regulatory communication decisions
- root cause, control improvement, owner, due date, and post-incident review