"Documents Considered": provenance as a feature

Near the end of every analysis report our workflow produces, there’s a table that no one asked us to make marketable: Documents Considered. One row per source — the office action, the specification, the claims listing, each cited reference — with where it came from (uploaded, or retrieved by publication number) and whether it was actually reviewed.

The interesting row is the one that says Not reviewed.

Why we mark our own gaps

Suppose an examiner’s rejection leans on a foreign-language reference that was never uploaded and couldn’t be retrieved. A typical AI system faces a choice it usually gets wrong: leave the reference out silently, or characterize it from training-data memory. Both options produce a report that looks complete. Both are traps — the first hides a gap from the attorney who needs to know it exists; the second invents evidence.

Our workflow takes the third option: the reference appears in the report, in the analysis where it’s relevant, explicitly flagged as relied on by the examiner but not reviewed by the workflow. The gap is on the page, where the attorney signing the response can see it and decide what to do about it.

Provenance changes how you read AI output

A report with honest provenance can be reviewed in a fundamentally different way. Instead of line-by-line forensic re-verification — which erases most of the time AI was supposed to save — the reviewing attorney starts from a trustworthy map: here’s what the analysis is grounded in, here’s where its edges are. Verification effort goes where the report itself says the uncertainty is.

That’s the same philosophy behind the rest of the report’s machinery: claim language in the rejection chart is checked mechanically against the pending claims, so quotes are verbatim by construction, not by hope. Every artifact shows its sources. The system’s job is not to seem complete; it’s to be checkable.

The test worth running on any legal AI

Ask a vendor’s system to analyze a matter with one document deliberately missing, and read what comes back. If the output doesn’t tell you what’s missing, you’ve learned the most important thing about it: it optimizes for looking finished. In legal work, the finished look is the cheapest thing to fake — and provenance is the hardest.