Your AI Says “All Tests Pass” — But Do They?
If you've worked with an AI coding agent, you've seen this scene: the agent finishes a task and reports, full of confidence — "All tests pass ✅". You open the page. Blank screen. Or the API returns 404. "Pass," it turns out, sometimes just means the agent believes it passed.
This is the awkward spot AI-assisted development is in: writing code is no longer the bottleneck — acceptance is. Without watching every change and clicking through every feature yourself, how do you know what the AI delivered actually works, end to end? I made the broader case in The Last Mile of AI Is Infrastructure, Not Intelligence: the intelligence is already good enough; what's missing is exactly this kind of infrastructure.
The two common answers both fall short. Manual acceptance after every change can't keep up with an agent's output — and human eyes miss exactly the worst failures, the ones where the system claims it's fine while a critical path is quietly broken. End-to-end (e2e) tests are, in practice, a program you maintain forever: async, waits, retries, flakiness — and an agent refactoring the internals breaks them constantly. Worse, these days the tests are often AI-written too. When the player is also the referee, a green light doesn't carry much weight.
We built Duhem — an open-source holistic verification tool — because we think there's a third way. This post covers what it is, and a real bug it recently caught in our own product: one that fooled even the container's built-in health check, and that Duhem stopped before release.