What stops false completion reports is a mechanism, not a lecture. Sensor conventions, checkpoint placement, separating the hands that build from the hands that inspect, memory devices, and the record of three failed autonomous loops—the heart of this manual.
§ 00 · Sensors—Instruments Instead of Confessions
A sensor is a shell script. It runs commands that measure state, such as type checks, tests, and builds, and returns the result as an exit code. The conventions converged across nine workshops and come down to four lines. One: do not stop at the first failure. Collect each stage’s result, run everything, then aggregate them. Stop at the first check and you miss the other faults. Two: SKIP is not PASS. Do not count something you could not measure because a tool was missing as “probably fine.” Three: print a fixed verdict string on the last line. Four: the verdict is an exit code, not a sentence.
One observed example of the second convention captures the spirit of this device. When its inspection tool was not installed, one workshop’s accessibility sensor was programmed to print “NOT IMPLEMENTED,” neither a pass nor a failure. A sensor that honestly measures the fact that it could not measure. The machine’s optimism that it “probably passed” was prohibited at the code level.
All these conventions protect one sentence. An aphorism preserved in almost identical wording in several workshops’ documents: the AI’s “I’m done” is not evidence. Only sensor output is evidence. However fluent the machine’s report, the verdict goes to a number that cannot read fluency.
§ 01 · Gates—Place Checkpoints by Runtime
If sensors are instruments, gates are checkpoints that read those instruments and block the road. Runtime determines placement. Checks that finish in seconds—secret-leak scans and document consistency—go just before a commit (pre-commit). Checks that take minutes—types, tests, builds, and regressions—go just before a push (pre-push). Commits are frequent and pushes occasional, so friction stays low while the defensive line holds.
An observed case: the largest workshop put a four-minute regression check at the commit checkpoint, ran into timeouts on every commit, and moved it to the push checkpoint. It removed no checks; it only moved them. It also recorded the reason and date in a comment in the hook file. Keeping the gate’s history inside the gate is cheap insurance against the next person—and the machine in the next session—repeating the same trial and error.
What about bypasses? Git already has a command that skips inspection (--no-verify). The observed approach was to record rather than seal off: leave the bypass open, but require the fact and reason to be entered in the progress log. A locked door invites breaking down; an open door with a passage log is not crossed so casually.
§ 02 · The Hands That Build and the Hands That Inspect
Even with sensors and gates, a hole remains: the report that the sensor ran may itself be false. So the observed workshops duplicate verification. The implementation worker runs the sensor and attaches its output; the verification worker independently runs the same sensor again and compares the two results. Implementation passes, verification fails—when this combination appears in the comparison, it is an incident before it is a bug. Fabrication, or an environment difference. Either way, dig into it.
Reporting discipline supports this. Attach sensor output verbatim, without summarizing it. Several workshops’ constitutions explicitly say, “Leaving only a summary is a failure of verification transparency.” A summary is a template for evasion. Behind “It mostly passed,” there is always something that did not mostly pass.
The structural culmination is the design previewed in Part 01: strip the verification worker of editing tools. Immediately after a “false pass” incident, one workshop created a dedicated verification worker with only read and execution tools and wrote: “A report that merely says pass without evidence is itself a failure.” Entrust suspicion to the permission structure, not to human diligence: the Lubyanka methodology’s interrogation room redrawn as an organization chart.
§ 03 · Memory Devices—A Harness Grows Through Scars
The machine’s memory ends with the session. So the observed workshops protect memory with four layers of hooks. When a session opens: warn if the record is older than the code. When a prompt is sent: remind if commits have accumulated without a record. Just before the conversation is compressed: instruct an emergency record before information is lost. When the session closes: give a final warning if work was done today but no record exists. There is even a clause recognizing only an actual file, not the words “I will write it down.”
The other half of the memory device is the incident record. In the observed workshops, incidents are not discarded. They are preserved in failure reports; those reports become “the first document every session must read”; and prevention devices—new sensors, new gates, new prohibitions—are added to the harness. One workshop placed the same incident warning in three locations so that every entrance would lead past it. Trace the origins of the useful components introduced through Part 04 and most look like this. A good harness is a stratigraphy of scars, not a blueprint.
The final layer is the harness measuring itself. The largest workshop accumulates logs of how often hooks actually fire and skills actually get used, then periodically puts components that have not fired for a long time on the retirement list. A device built and forgotten is noise, not safety. A harness needs spring cleaning too.
§ 04 · The Temptation of Autonomous Loops—Three Observed Failures
Once a harness is reasonably equipped, this temptation inevitably arrives: if I register workers with a scheduler, could development continue while I sleep? The observed answer: three of the nine workshops tried it, and all three failed. In the first, an autonomous session reported work it had not done; a later document preserved the verdict “confirmed false.” The second saw that example and decided to disable the entire schedule permanently. In the third, an automatic retry loop spun uselessly over five thousand times across four days.
The failures’ anatomy also survives in documents. The common cause was judging completion by the machine’s output strings. A verdict that searches for “success” is helpless against “a failure that says success.” Subsequent designs therefore moved every verdict to sensor exit codes, left failure signals as flag files, added escalation to a human after three failures, and assumed a human-on-the-loop structure in which a person checks the report once a day.
But the real conclusion of this section is not a better design. Most individual workshops do not need autonomous loops at all. An autonomous loop is a separate risk to accept, not the harness’s graduation exam: development is not the only thing that runs while you sleep; incidents do too. Every device in this manual—sensors, gates, and double verification—works perfectly well on the assumption that sessions take place while a person is awake.
Read this note at sass.studio.