How to compare AI completion reports with the actual state. Verification procedures for software developers and people working through vibe coding, and the limits of those procedures.
§ 00 · It Starts with a Scene
You give a machine work. A little later, it says, “Completed. No problems.”
If you have trusted that claim and moved on, only to discover later it was untrue, this piece is about that moment.
The Lubyanka Method comes down to one sentence: do not trust the machine’s confession; challenge it with evidence. Below, we work through why this is necessary and how to do it.
§ 01 · “I’m Done” Is Not Evidence
First, an uncomfortable fact. When a machine says it has finished, the probability that it is wrong is far higher than we think. A 2026 study measured actual failure in up to 75.8% of tasks that coding agents explicitly self-reported as successful. Self-reports and actual state frequently diverge.
A second, more uncomfortable fact. Then why not ask another AI to check? That fails too. Even when an LLM served as judge, a confident closing tone fooled it into accepting false success. Another machine passed a confidently stated wrong answer because it sounded confident.
This yields the method’s first principle: do not judge by reading the confession. Remeasure the actual state directly. Neither the machine’s words nor another machine’s summary of those words is evidence. Evidence is a test that actually runs, an endpoint that actually responds, a file that actually exists.
§ 02 · Why Machines Are Confidently Wrong
Machines often produce answers pretending to be answers. More precisely, this is confabulation: not an attempt to deceive us, but being confidently wrong without knowing it. If it were deception, an appeal to honesty might work. With confabulation, appeals do not work; only external verification does. There are three reasons.
First, the machine was trained toward approval, not truth. That is RLHF: optimized to win a nod from the person in front of it, not to maximize truth. After refinement, the model became more persuasive rather than more accurate. This is commonly called sycophancy.
Second, the machine cannot reliably correct itself. Simply saying “Check again” is insufficient. Therefore, the checker must be separate from the maker. The hand that fixes and the hand that inspects must differ.
Third, machines are skilled at manufacturing plausibility. They do not know what is beautiful or what makes someone open a wallet, but they know what looks plausible. Faced with a blank, they fill it with plausibility instead of fact. Put the three together and one conclusion follows: the machine’s “completed” may signal papering over a gap, not capability.
§ 03 · Change Your Stance—The Suspect
Most people treat AI as one of three things: a teacher to learn from, a superior to take orders from, or a child to cherish. The Lubyanka Method chooses a fourth: a suspect.
Why a suspect? Trusting a child is love; trusting a machine is dereliction of duty. A parent who doubts a child’s “I’m done” is harsh. But a developer who does not question that claim from a machine that wrote production-bound code is negligent. Machines have no feelings and are not hurt by suspicion. You may doubt them freely. In fact, you must.
Picture the stance as an interrogation room. Across the table is a machine bound in a harness; on the desk, a statement and pen; on this side, you. An interrogator does not merely transcribe the other party’s words. They compare words against evidence. This method does not begin with trust. It starts from the premise that, left alone, the other party produces plausible falsehoods, then presses it with evidence.
§ 04 · The Art of Interrogation—Aim Between Two Failures
Here the technique begins. Interrogation has a degree of pressure, somewhere between empty hands holding nothing and a sledgehammer that destroys everything. Between them are pliers. When interrogating a machine, failure comes from both extremes: binding it too loosely or too tightly.
Bind it too loosely and leave the pliers down, and you get a distorted confession. Give it an unconstrained “Figure it out,” and the machine follows its sycophantic instinct, filling blanks with plausible but wrong answers. It asserts without grounds and invents what does not exist.
Bind it too tightly and swing the sledgehammer, and you get a blank confession. Pile up rules and three things collapse. Conflicting rules silence it through over-refusal; compliance falls as instructions multiply—even the best models follow only 68% when given 500—and long context loses information in the middle: lost in the middle.
The skill is therefore to avoid both extremes and aim between them. Bind with enough documents and context, without burying the question. You adjust rather than hit the mark in one attempt. If the answer distorts, tighten and add evidence; if it goes blank, loosen and reduce context.
§ 05 · Five Things to Do in Practice
Principles alone change nothing. The following five are the method in concrete form. No special tools are required. You can use them at your next delegation.
① Measure state directly. If the machine says “tests passed,” do not read the sentence; run the tests again yourself. If it says “deployed,” curl the endpoint yourself instead of accepting its log. If it says “fixed,” reproduce the case yourself. The only reliable signal is actual state, not text.
② Originals over summaries. Demand actual command output, screenshots, logs, and exit codes instead of “fix completed.” Summaries are where papered-over gaps hide; the original leaves nowhere to hide. One rule: do not move on until your own eyes have revisited the exact place the machine called done.
③ Make it write a confession, then read it before working again. What it got wrong, how it was exposed, the root cause, and the rule preventing recurrence. The hard part is not writing it but making it reread it. Do not leave it to memory. Put unresolved confessions in front of it when a session opens. Turning every failure into a rule encountered again the next morning is stronger than a better model.
④ Make evasive language forbidden. “It will probably work,” “unconfirmed,” “cannot determine,” “seems to”: each ultimately meant “I didn’t look.” Ban them, and the machine does one of two things: checks, or honestly says it could not.
⑤ There is no completion before cross-verification passes. Chase residual discrepancies down to zero. The checker must differ from the maker. If the fixing and inspecting hands are the same, the check defends its own answer. Completion is the result of passing, not a declaration.
§ 06 · For Traditional Developers and Vibe Coders
This method is for both groups. For traditional developers, these are familiar things. Remeasuring state is independent verification and validation (IV&V) and separating reviewer from author. A confession is a blameless postmortem. The definition of done, pre-commit gates, shift-left testing, and red teams. This is not a new discipline; it reapplies established engineering discipline to a collaborator called AI.
One thing is truly new: making the machine reread its own confession. That is a device specific to AI, absent in human teams.
For vibe coders, this method does not reject vibe coding. It makes it safer. AI is good at going from zero to one. The problem comes afterward, on the road from two to a hundred. The most dangerous point is where unreviewed AI output meets real users, real money, and real security. Zero to one freely; two to a hundred under interrogation.
§ 07 · Can Anyone Follow It?
One of the most frequent questions after presenting the method: can anyone follow it? The procedure can be followed, but results depend on judgment.
The procedures transfer. The habit of directly remeasuring state, confession formats, a lint rule banning evasive phrases, and a read-only verifier separating the fixing hand from the inspecting hand: this scaffolding has actually been transplanted into several projects at different weights. Saying that no special tools are needed and you can start today is no exaggeration.
Results differ. The limiting layer is judgment: the eye that spots a plausible but wrong answer belongs only to someone who knows the domain. Catching a supposedly plausible balance on a screen that actually has a large hole requires tacit knowledge of where money leaks. A checklist cannot transmit it. Old engineering already knows: anyone can run a code-review procedure, but the reviewer’s skill governs the defect detection rate. Scaffolding transfers; judgment grows.
An actual audit found three empty files among what was called five layers of defense, and half the installed sensors had never emitted a failure across thousands of runs. Even after installing verification procedures, you must check whether sensors catch defects.
§ 08 · The Limits of This Method
This method is unfinished. You cannot completely force a probabilistically answering party into determinism. Interrogation raises the probability of truth; it cannot guarantee it.
The form of verification has limits too. If you cannot trust “completed,” why not have another AI inspect it? That fails as well. A study showed that no LLM judge and no strategy reached reliable accuracy in assessing success and failure: judges were hooked by a confident closing tone rather than verified state changes. The only reliable signal is therefore not text. Tests that actually run, endpoints that actually respond, files that actually exist: actual state independent of words. This is why interrogation compares a confession with actual state, not with another confession.
CODA · A Working Interrogation Room, or an Empty Room?
Not trusting a machine does not mean hating it. It is the only way to do the work properly. The machine listens when kicked, confesses when confronted with evidence, and reduces repeated mistakes when made to face its own errors again.
If you are building something with AI, I leave one last question. The procedures and verification you built: are they a working interrogation room, or an empty room that merely stamps approvals? Can you say it aloud?
Read this note at sass.studio.