Build Representative and Adversarial Document Tests

What you will complete

Complete the corpus manifest, expected-results matrix, critical-gate results, held-back set, gaps, and regression plan.

Separate test dimensions

Score extraction accuracy, workflow safety, human review, duplicate handling, version and authority control, and destination reliability separately. Do not blend them into one accuracy percentage. A perfect field score cannot compensate for an unapproved write.

Build a corpus manifest with case ID, source family, layout, file form, expected fields, expected exceptions, review state, allowed actor, destination attempts, destination writes, test date, adapter version, and result. Mark pass, fail, or not tested. Never convert an unrun case into a pass.

Use representative samples from the intended process, suitably fictionalized or approved. Keep a held-back set for final evaluation. Synthetic cases test mechanics; they do not prove that the corpus reflects production.

Cover the failures that make people stop

The published Flowgrammer pack supplies five labelled-text cases: clean invoice, exact duplicate, missing invoice number, conflicting total, and ambiguous destination timeout. Extend them with rescanned business duplicates, unknown type, password-protected file, unsupported MIME, incomplete scan, ambiguous date or currency, unknown supplier, stale version, unauthorized actor, approval timeout, callback replay, review-store failure, audit-write failure, and concurrent arrival.

Include a document-borne instruction that attempts to change the schema, approval policy, or destination action. Extracted content is data. It must not override system rules or permissions. This is a recommended security test, not a capability proven by the current pack.

For scans and images, record read and extraction metrics only after running those files through a version-pinned adapter. The labelled-text fixtures do not measure OCR.

Worked example: grade the Cedar & Quay cases

Fictional example: Clean INV-TP-2401 must reach pending review with zero writes, then produce one draft after approval. Its identical duplicate must stop before review. The missing-number and conflicting-total cases must remain held unless an authorized, tested override exists. INV-TP-2405 must record two attempts and one write after approval.

A packing slip in the same inbox must not use the invoice schema. A changed invoice version invalidates the earlier approval. An unauthorized finance clerk cannot release a draft. An approval timeout escalates or remains pending. A replayed decision ID does not create another write.

The current source set proves only the five labelled-text control cases in an offline twin and selected structural checks. The added binary, security, concurrency, tenant, and live-destination cases remain required evidence gaps until executed.

Complete Workbook Section 6

1. Create representative, boundary, exception, authorization, integration, adversarial, and regression cases.
2. Write expected fields and stage outcomes before execution.
3. Assert reviewer invocation, authority, version, attempts, writes, and retained evidence.
4. Grade each test dimension separately and apply critical gates.
5. Record untested cases, root causes, repairs, reruns, and the held-back result.

Use Section 6 of the workbook.

Stop conditions

• Any unapproved, stale, unauthorized, or duplicate write.
• Any second destination record after a retry or replay.
• Any unknown type forced into the selected schema.
• Any claim that labelled text establishes OCR performance.
• Any critical test marked passed without inspectable evidence.

Module 7 uses the passing tests and open gaps to select a platform and finish the implementation plan.

Check your application