I Built a Test My System Could Fail — and It Did

On honest evaluation, and why a failure you keep is worth more than a pass you chase.

Spend enough time around AI and tech demos and you notice something: everything passes. 

Every benchmark is crushed. Every model is state-of-the-art. Every chart climbs. After a while, it stops meaning anything — because when nothing is ever allowed to fail, a pass tells you very little.

So I have been doing the opposite.  I build tests that can fail, and I decide what failure looks like before I ever run them.  

The Setup
I am building Asha... an orchestration system that studies a small but useful question:
Given a task, which route should handle it — and when should the system abstain?

Right now, the test domain is compression. That may sound unglamorous, but it gives clean measurements: route choices, time limits, costs, skipped paths, failures, and receipts.  The point is not to replace compression tools.

The point is to study a bigger question:
Can a system make honest, bounded decisions over tools that do not decide for themselves — and show its work instead of simply producing an answer and asking you to trust it?

To test that, I run what I call a locked probe.  I choose a public dataset the system has not seen, write down in advance what the test is and what will count as passing, freeze that document so I cannot quietly edit it later, and only then download the data and run.

The rule that matters most is simple:
Once the test is frozen, the experiment answers to the document.  Not to my hopes. Not to a prettier number I notice afterward.
Not to the version of the story I wish had happened.

The Failure
The system met the new data and, on most of it, did what it was expected to do.  Then, on one file, it missed.  By 3,026 bytes.
The better route was a compression method called brotli. My system chose a different one, and it was worse by about three kilobytes on a multi-megabyte file.  Under the rule I had frozen in advance, that was a failure.  A small failure, yes. A marginal one. But still a failure.
So I left it failed.

I did not re-run the test until it passed.
I did not widen the definition of “pass” after the fact to swallow the miss.
I did not bury the result in a footnote.
I wrote it down as a failure, with the exact byte count, and moved on.
The logs still say failed. They always will.

Why That Matters
There are two reasons that failure matters.  The first is credibility.    
A pass massaged into existence is worth very little.
A failure kept honest is useful because it proves the test had teeth.
If you want to be believed when you say something worked, the price of admission is being believed when you say something did not.

The second reason is more interesting.
When I looked at why the system missed, it did not look like a broken rule. It exposed the boundary of the rule.  The system is designed not to chase every tiny possible gain. In this case, it chose one bounded route and did not spend extra time testing every alternative. The later check showed that another method would have been slightly better — by 3,026 bytes.

Under the frozen rule, that remains a failure.  I am not changing the label.  But it tells me something precise: this is the line where “best possible answer” and “best bounded decision” can diverge.  That is not a failure to hide. It is a boundary worth mapping.

The Harder Question
Finding that edge raised a sharper question: was the system overlooking better routes it should have been considering at all?
So I ran a follow-up probe aimed straight at that boundary.  It came back clean. No case showed the simpler route winning by a margin that would have mattered. And this is where the discipline gets hard — harder than admitting a failure.

The easy move is to write “settled” and bank the win. But the rival route never came near the line that would have counted; the closest it got was only a fraction of that margin.  Which means the test confirmed the easy half: the system holds steady when the alternative is only slightly better. It did not test the hard half: what happens when the alternative is decisively better. No case ever put that on the table.

So that is what I wrote down.  Not “settled.”  
Instead: this confirms one half, leaves the other untested, and points to the experiment that would test it properly.

The Lesson
I have come to count a day as a win if it leaves me with more solid truth than I started with — whether or not the truth flatters me.
A failure I understand beats a pass I do not.  A clean result I have interrogated beats one I simply accepted.

That sounds too simple to be a method. But try holding to it when the number is right there and one quiet edit would make everything look finished.  The discipline is not believing truth matters. Most people believe that in the abstract.

The discipline is not touching the edit.  It is not letting a failure disappear.  It is not letting a clean result claim one inch more than it earned.

So build the test so it can fail.
Decide what failure means before you look.
When it fails, leave it failed and go find out why.
And when it does not fail, ask whether it ever really could have.
The rest is decoration.