Checking every claim against the line of code that decides it

As technical lead on another company’s product, I work from a stack of documents: a product requirements doc, a whitepaper, a long glossary, design docs written days apart, and the code itself. Almost none of that material was written and validated by a person, and every sentence in it reads with the same confidence whether it describes something built, something planned or something nobody has decided yet. One of the rules I wrote into that workspace’s instructions says nothing load-bearing gets repeated to a colleague until it has been checked against the primary document, and the claims check is that rule turned into a method.

How it works

I start by writing the claims down, one line each, grouped by what they’re about. The instruction at the top of the file says every claim gets marked TRUE, FALSE or PARTLY, with the exact quote and the file and line that decides it. The sources sit beside it as local read-only copies, covering the requirements doc, the whitepaper, the glossary, the design docs, the database schema, the server and device code, and the runbook. Agents then read across all of them for each claim. A sentence in the plan gets settled by the line in the schema or the controller that does or doesn’t do what the sentence says.

The list ends with an open request: anything in the plan that directly requires a change and that my list missed. Every verdict then goes to an independent reviewer before I use it.

What comes out

The result is one page. It opens with the short answer, which says which part of the document matters for what’s being built now and which parts can wait. Next comes what the document confirms about what’s already built, then what has to change, with each item sized and placed, whether that’s a small server change, a new piece of the registry or something that needs the device. Last is where the document disagrees with itself or with the other documents, and every one of those goes back to the document’s owner to settle.

In one run the disagreements included the same lifecycle step named three different ways in different parts of the document, and a plan that ruled out building a feature in one section while listing it in the first phase in another. A sample data file used an identifier of the wrong length for the standard the spec itself relied on, and two design documents edited six days apart took opposite positions on the same component. Any one of them would have sent a build in the wrong direction.

The break-it battery

The documents are half the job, and the other half is what the software does when someone pushes on it. For the internal dashboard there, I wrote a battery that loads each main screen in three browser engines at phone, tablet and desktop widths. It gives records long unbroken names, HTML, emoji and right-to-left text to test layout and escaping, and it injects API errors and slow responses into the pages. To test an outage it cuts the dashboard’s read-only database role off from the real database instead of faking one, and it makes sign-out fail with a server error, a dropped connection and a session that has already expired. It also covers keyboard-only use, colour contrast, automated accessibility scans and performance while the machine is under load, and each round records the state of the services before it starts and checks it again afterwards.

How it runs

It runs as an agent workflow with scripts, a claims file and a one-page report format, inside a workspace that can only reach that company’s mail and files. If your plan and your codebase have drifted apart, I’d be glad to talk about running it on yours.

Next use case

The email assistant I built so it can never press Send

All use cases