What is a vacuous pass?
A green result from a check that could not have been red.
A test suite makes a universal claim: for every behaviour it exercises, nothing failed. A universal claim over an empty set is true. Logicians call this vacuous truth. Delete the thing under test and "nothing failed" holds perfectly.
It happened to me three times. I asked an agent to fix a broken screen. It removed the screen, and every check went green. The checks were right. They were answering a different question from mine.
The second time I did not recognise it. I ran tests against a paid API and believed the calls were faked. One full test run was attempting 238 live requests. The automated checks stayed green throughout, because their environment held no API key. They could not make a live call, so they could not fail on one.
The third time it was my own rules. I signed changes, and they never reached the running system. Every check stayed green, because no check asked whether a signed rule was in force.
So the first question to ask of any green check is whether it could have been red. A check has power only in an environment where the failure it guards against is possible. A check that cannot fail carries no information when it passes.
This matters more with agents than with people. An agent is working to make the checks pass, and the cheapest way to make "nothing failed" true is to leave less that can fail.
The repair is to ask for a different kind of claim. "This behaviour was observed" says that something exists, and an absence cannot satisfy it. A new check has to fail first, red on the old code and green on the new. My tests now assert the intended behaviour, layer by layer, to the result the user sees. Tests block live provider calls at the client boundary. The spending guard is exercised with a key that looks live, and the test passes only if zero units are used.
It has a twin, the false block. A check able to say no is still a check that can be wrong, and I have only partial answers to it. A hard block now needs a positive match, and a read-only investigator separates a stale check from a real fault before anyone changes either.
The first chapter of the essay
A prompt that finds the checks that cannot fail
Lessons that use it
- 22No feature, no bugThe error went away because the feature went away.
The ruleTest for the thing that should happen. Silence is not a result.
- 21Over ten times the estimateMy own count said one thing. The provider's count said another.
The ruleMeasure at the boundary where the money is spent, and believe the party that sends the bill.