Filling an official form without a template

The hard part is not knowing what to write

An administrative form to submit: a sworn statement, a commitment deed, a certificate. The values that go in it you already have — they are in the database. The company name, the tax identifier, the amount, the date, the signatory’s name.

The difficulty is elsewhere: where to write them. Every buying authority has its own version. The same fields, in a different order, under different labels, sometimes in a table, sometimes in running prose, often scanned crooked.

You do not have a form. You have hundreds of variants of the same form, and you do not know them in advance.

The defect: a hand-maintained table that fails without a sound

The first version did what everyone does: a lookup table mapping a canonical label to a line in the document. Sixteen keys, written by hand.

On a scan, detection read very well — 60 fields matched out of 74 on one document, 22 out of 36 on another — but it returned no geometry at all: 110 fields out of 110 with no coordinates. Position therefore came from an entirely different layer, anchoring onto detected lines. And that layer stopped at the sixteen-key table: any key not in it was dropped by a silent `continue`.

The measured result, on the two documents a client actually chose that day: 21 fields in the variant, 10 placed. 23 fields, 7 placed.

The thirteen missing keys: the contract object, the representative’s capacity, the legal form, the bank, the account type, the four amounts, the VAT rate, the account holder, the city, the social-insurance scheme. Their values were correct and available. They died on a `dict.get()`.

This is the worst kind of defect: it raises nothing, it logs nothing, and it returns a document that looks finished.

Why enlarging the table is the wrong instinct

The instinct is to add the thirteen keys. That fixes the day and reproduces the defect: a hand-maintained contract that fails closed and silent as soon as a document falls outside the known set.

A system whose coverage depends on a human-written list has an unpleasant property: its quality degrades over time without anything signalling it, because the world produces variants faster than you add entries.

The reframe fits in one sentence: stop asking “what is the label” and start asking “where is the blank”. Labels vary endlessly. The shape of a space to be filled varies far less.

Pass A — geometry

Render every page as an image, hand it to a vision model, and ask it one thing: the regions to fill, with their coordinates. Not the values. Not the labels. The blanks.

This is not a hunch, it is a benchmark. On a first small benchmark — two documents, three repetitions, ground truth being the regions actually placed in production — one model reached 96.1 %, another 88.2 %. The second systematically missed the signature block on the last page.

On a wider benchmark — eight scanned documents, six form families, 280 reference regions, two prompt phrasings — the corrected prompt rescues the weak model: it rises to 95.0 % recall and becomes both the best and the cheapest. The correction was a page-by-page sweep plus an explicit pass over end-of-document blocks — added precisely on the measured defect, not on a hunch.

The lesson is not “vision models are good”. It is: a prompt gets repaired on measured defects, never on your idea of the defects.

Pass B — the blanks with no shape

Geometry catches whatever has a recognisable shape: dotted lines, a cell, a box. It misses the rest — a blank in the middle of legal prose, a space after a colon, a value expected between two sentence fragments.

Free generation (“find every field in this document”) is unstable: measured, the same document returned zero fields, then thirteen.

What makes it stable comes down to three inversions.

The code enumerates, the model answers. The document is split into units and they are presented one by one. The model does not choose what it looks at.

One answer per unit presented, and the code checks it. It cannot silently skip one.

It copies the anchors, it does not paraphrase them, and it invents no coordinates.

Above all, the question asked is not a keyword list but a counterfactual test: can this region be replaced by a value to produce a completed form, without altering the permanent text? A marker table, by contrast, fires on running text — measured: a column header reading “Provisional bond” in a tender notice produced a false positive. The counterfactual question did not.

The metric that matters is not the one you watch

Across 56 native documents and 1,783 reference regions, the geometry pass alone covered 87.3 % of regions. A very good number. It finished 8 documents out of 46.

With the second pass: 93.7 % coverage — six points, a modest gain — and 33 documents out of 46 fully covered. Four times as many.

A client does not submit 87 % of a form. The aggregate metric rewarded a gain nobody could use, and hid the one that changed everything. Since then the headline metric is the number of finished documents, not the share of covered regions.

It is a question worth asking of any system: is my metric additive while my deliverable is binary? If so, it has been lying to you politely from the start.

What to do with the text around the blank

A blank is almost never alone. It is surrounded by notes in brackets or parentheses, and each calls for different handling. Four categories, returned by the same call that detects the blank:

The blank comes first, the note stays printed. — The note itself is the region to replace. — The note describes the expected format, not the value. — It is permanent text, nothing to fill.

Without that distinction the program writes over “(town)” without erasing it. Measured on thirteen documents. This is not cosmetic: a document made illegible at that spot is a rejected document.

Verify the render, not the intentions

This is the part I would do first if I started again.

Over one week, four serious defects surfaced: a white mask erasing superscripts, a company name placed where the signatory belonged, a value written into the block of a different legal variant, and an overprint. All four were found by eye. No counter saw them — because the counters read the geometry *declared by the program doing the writing*, and therefore inherited exactly its blind spots. Measured one day: 482 characters of prose destroyed while the counters showed zero.

The fix is blunt and simple: stop trusting what the program declares. Render the original and the deliverable as images, and compare pixels. Two measures: erosion — printed characters whose ink has vanished — and unclaimed ink — regions of added ink that no declared region covers.

Three measurement traps, all met for real.

Resolution. At 150 dots per inch, anti-aliasing counted 45 “altered” characters where there were 2 at 300. Measurement happens at 300, never lower.

The box is not the glyph. The engine returns the *line* box, not the character’s: an “r” overflowing onto the line below produced 42 false positives, stable at 300, 400 and 600 dpi. Stable — therefore credible, therefore dangerous. The comparison is now restricted to the pixel rows where the character actually has ink.

The rotated page. Text coordinates are not rotated, the image is. Without a rotation matrix everything is offset. And a page whose dimensions changed is never skipped silently: a silent skip rewards breakage, so it becomes a named violation.

Finally a doctrine that reaches beyond this module: on a scan with no text layer, erosion is not measurable — the verifier returns “unavailable”, never a favourable verdict by default. A test that cannot run must not go green.

The judge that looks, for what pixels cannot see

A pixel count does not see a legally wrong document. If the value is placed cleanly in the wrong block, the ink is declared, nothing is erased, everything is green. The document is wrong all the same.

Hence a second verifier, which receives the pages rendered as images and the list of placements, and looks for four defect classes: wrong block, value that does not match its line’s label, overprint, erased form text.

One transport detail that took a while to understand: handing the same PDF as a file rather than as an image drops geometric recall from 83.5 % to 42.7 %. The model then reads extracted text and does not *see* the page. When what you are judging is a layout, you have to send an image — the input format is not an implementation detail, it is half the result.

Detecting that you missed something, without knowing what

The most dangerous defect does not announce itself: it makes work disappear silently. Three documents from the same family stopped at exactly 24 regions. It took cross-checking an annotated reference to discover an entire table was never carried through.

The signal that would have predicted that defect with no annotated reference: four geometrically homologous rows, none of which carries a field. Whatever they contain, whatever their labels. A repeated structure that is entirely empty is an anomaly.

This module deliberately returns no composite score. The previous one did, it predicted backwards, and it was impossible to say why — precisely because it blended its components. What is exposed instead are raw, comparable quantities: how many eligible cells, how many detected, coverage by row and by column. Enough to tell a missed row from a missed column from an entirely absent table.

A single score is comfortable to display and useless to debug.