Every trading system eventually confronts the same question: does what we think we hold match what the venue thinks we hold? Reconciliation answers it, and it is the least glamorous component in the stack.
The problem
Orders can fill partially. Acknowledgements arrive out of order or not at all. A venue can restate a fill hours later. Any of these leaves your internal ledger disagreeing with the exchange, and the disagreement compounds silently until someone looks.
Design
We treat reconciliation as a queue of small, independently retryable jobs rather than a nightly batch. Each job compares one position against one venue snapshot and emits a discrepancy record if they disagree. Jobs that fail repeatedly move to a dead-letter queue instead of blocking the pipeline.
What we learned
Two things. First, retention matters as much as correctness: a queue that keeps every failed job forever will exhaust memory long before it exhausts patience, so failed records need a bounded lifetime. Second, a growing dead-letter queue is a signal, not a resting state — if nothing consumes it, it is just a slower way to lose data.