Adversarial verify
Before I trust a piece of work, reviewers try to break it. Every finding is then checked against the facts, because reviewers are wrong too.
Use it When a mistake would be expensive, for example anything that guards private data.
The problem
The agent that wrote something is the worst one to check it. It reads its own work and sees what it meant. Asking it “are you sure?” gets a yes.
How it works
- A draft exists: a spec, or code.
- Reviewers get one job, which is to find what is wrong with it. They are not asked whether it is good.
- Every finding is checked against the facts: the documentation, the repo, the real system. A finding is a claim, not a truth.
- A finding that holds is fixed in the draft. A finding that does not hold is rejected, with one line that says why.
- The result is the next revision, and the list of what was accepted and rejected stays with it.
The same idea works as a test that runs every time. The workbench has a canary test: it writes a private note that holds a random word, then reads every public page and every public endpoint and fails if the word shows up anywhere.
What it costs
A full extra pass over the work, and then the checking, which is the slow part. Each finding has to be looked up, one by one.
Where it fails
Without step 3. Reviewers that are told to find problems will find problems, including ones that are not there. If every finding is accepted, the draft gets worse. On the workbench spec some suggestions were turned down after that check, each with its reason.
It also only covers what the reviewers thought to attack. A clean review is not a proof.
Where I used it
On the spec of the AI lab, before any code was written. One finding that held: the test login for agents had the same rights as me on a preview that would hold my real notes. It now sees only the test rows it made itself. One that did not hold: a request to build the packages in another order.
Used inAI lab