A model that extracts the wrong balance from a bank statement does not throw an exception. It returns a number that looks entirely reasonable, in the right format, in the expected field. Someone downstream acts on it. That is a materially worse failure than a crash, and it is the reason document automation deserves more scepticism than it usually gets.
Keep the source attached to every field
The single most valuable design decision is that extracted values carry their origin: document, page and bounding region. A reviewer can then check a number in one click instead of reopening the PDF and hunting for it.
{
field: "closing_balance",
value: 48213.55,
currency: "EGP",
confidence: 0.94,
source: { documentId: "stmt_2291", page: 3, bbox: [412, 688, 190, 22] },
method: "model"
}Without provenance you cannot build a review queue, and without a review queue you are asking users to trust a number they have no way to verify.
Rules first, models where they win
For consistent templates, a parser is more accurate, faster and free. Models earn their place on the variable long tail — different banks, different layouts, scanned pages, poor OCR. Running a model over documents that a parser handles perfectly adds cost and variance for no accuracy gain.
- Detect the template; if it is known, parse deterministically.
- Route unknown or low-quality documents to model extraction.
- Record which method produced each field, so accuracy can be measured per method.
Confidence must route, not decorate
A confidence score displayed in a tooltip is decoration. A confidence score that determines whether the record goes straight through or lands in a review queue is architecture.
Set the threshold from the cost of being wrong, not from a desire to keep the review queue short. A mis-stated closing balance on a loan application is not equivalent to a mis-categorised coffee purchase, so they should not share a threshold.
Measure accuracy on documents you did not tune on
Accuracy measured on the examples you used to build the pipeline is worthless. Hold back a set of real documents — including the ugly ones: skewed scans, poor contrast, unusual layouts — and track accuracy against that set every time the pipeline changes.
If you cannot state your extraction accuracy on a held-out set, you do not yet know whether the automation works.
Capture corrections as training data
Every correction a human makes in the review queue is labelled data. Recorded properly, it becomes the mechanism by which the pipeline improves and the review queue shrinks — which is the only sustainable version of this system, because a review queue that never gets shorter is just manual work with extra steps.
Keep the provider swappable
Model pricing and capability move fast, and data-residency rules may not permit certain providers at all. Put extraction behind an interface with a stable output contract so changing model is a configuration change rather than a rewrite. Assume you will change it.