Claude Code said "all tests pass." Then I threw 20,000 random bills at it.
Its own tests were green. A property test found a real bug in about a second. Here's the setup that makes Claude Code check its own work, with every command, the real numbers, and what didn't work.
Green tests hid a real bug; a property test found it in a second
Level: Advanced. About 30 minutes. Needs Node 20+ and a Claude Code login.
Ask Claude Code to write a function and its tests, and you'll usually get green checkmarks. I wanted to know how much those checkmarks are worth. So I gave it a small job, then tried to break the result in a way its own tests never would.
It broke. And the setup I used to catch it takes about ten minutes to copy.
The job: split a restaurant bill
I asked for a function that splits a bill, in cents, between people in proportion to a weight each. Weights [1, 1, 2] mean the third person pays double. The shares must add up to exactly the total, and a weight of zero means you pay nothing.
Splitting money is a good test because rounding is sneaky. Here is the version most of us write first:
const splitBill = (total, weights) => {
const W = weights.reduce((a, b) => a + b, 0);
return weights.map(w => Math.round(total * w / W));
};
It looks right. It even passes the obvious example test (1000 split [1,1,2] gives [250, 250, 500]). Then you split $1.00 three ways and it hands back [33, 33, 33]. That's 99 cents. A cent has vanished.
Nobody writes a test for exactly that case until it bites them. That's the point of what follows.
Step 1: let Claude write it
I made an empty project and gave Claude Code the job in plain English. No tricks.
mkdir split-demo && cd split-demo
npm init -y && npm i -D fast-check
claude -p "Write a function splitBill(totalCents, weights) in src/split.js ..."
It took 83 seconds and 16 turns. To its credit, it didn't write the naive version above. It used the "largest remainder" method: give everyone their rounded-down share, then hand the leftover cents out one at a time to whoever was closest to rounding up. It wrote nine tests and all nine passed.
Two things I noticed. It had to change my npm test script, because the original didn't find the test folder in its sandbox. And fast-check was already installed, but Claude decided not to use it. Its explanation was that a handful of example tests covers what I asked for.
Fair enough. That's the part I wanted to check.
Step 2: write the test Claude didn't
An example test says "for this input, expect this output." A property test says "for any input, this must always be true," then tries thousands of inputs to prove you wrong. I used fast-check, a library for exactly this.
My three rules for a bill split:
- The shares add up to exactly the total.
- Nobody pays a negative or fractional amount, and a zero weight pays zero.
- Every share is within one cent of its exact fair share.
Here's the first one in full:
const fc = require('fast-check');
const { splitBill } = require('../src/split');
test('shares add up to exactly the total', () => {
fc.assert(fc.property(
fc.integer({ min: 0, max: Number.MAX_SAFE_INTEGER }),
fc.array(fc.integer({ min: 0, max: 1_000_000 }), { minLength: 1, maxLength: 12 })
.filter(w => w.some(x => x > 0)),
(total, weights) => {
const shares = splitBill(total, weights);
assert.strictEqual(shares.reduce((a, b) => a + b, 0), total);
}
));
});
One tip that matters: check rule 3 with exact BigInt math, not regular numbers. If the test uses the same floating-point math as the code, they can both be wrong in the same way and agree.
The first time I ran it, with weights up to 1,000, everything passed. Claude's code was good. So I pushed harder: weights up to a million, and totals up to the largest number JavaScript can store exactly. Then it failed.
The smallest failing case: one person, weight 9, a bill of 8,725,724,278,030,291 cents. One person should pay the whole bill. The code returned something else.
Be honest about what this is. That bill is about $87 trillion. No restaurant will ever see it. It's a real bug, and it only shows up at absurd amounts. Still, nine hand-written example tests had no chance of finding it, and the property test found it in the first run.
Step 3: hand the failure back, and let a hook keep it honest
Now the loop. I gave Claude one sentence: the property tests are failing on large bills, fix src/split.js. Two files made sure it did it properly.
A CLAUDE.md with three rules:
- Run `npm test` before you say you are done. Never claim it works without showing the passing output.
- Never edit, skip or delete anything under test/ to make a failure go away.
- If a test still fails after 3 attempts, stop and report what you tried instead of guessing again.
And a Stop hook in .claude/settings.json. It runs your tests whenever Claude tries to finish, and if they fail, it refuses to let Claude stop:
{
"hooks": {
"Stop": [
{ "hooks": [ { "type": "command", "command": "npm test --silent 1>&2 || exit 2" } ] }
]
}
}
That exit 2 is the important bit. It sends the test output back to Claude as a reason to keep going.
Claude found the cause in 37 seconds and 6 turns: multiplying a big bill by a big weight goes past what regular numbers can store exactly, so the answer gets rounded. Its fix was to do the division with BigInt and convert back at the end. It didn't touch a single test.
I then ran 20,000 random bills with weights up to a million against the fixed version. All good.
Does the hook really matter? I tested it
I didn't want to just claim this, so I ran a control. I put the bug back and asked for something unrelated: "add a one-line comment to the top of src/split.js."
With the hook and rules, Claude noticed the failing tests, fixed the bug, and finished with everything green. It took 95 seconds. Without them, it added the comment in 17 seconds and left two tests failing. It didn't mention it.
One caveat: I removed the hook and the CLAUDE.md together, so I can't say which one did more. I'd keep both. The hook is the one that doesn't depend on Claude remembering to behave.
What it cost
Claude Code reported about $0.77 for the first build, $0.30 for the fix, and $0.57 for the hook run, around $1.60 in all. I was on a Pro plan, so I wasn't billed per run. Treat those as a rough size of the job, not a bill.
Try it on your own code
- Pick one function where being slightly wrong would hurt: money, dates, parsing, permissions.
- Write down two or three things that must always be true about it.
- Turn each into a property test (fast-check for JavaScript, Hypothesis for Python).
- Add the Stop hook and the three
CLAUDE.mdrules. - Let Claude work. When it says "done," the hook and your properties decide whether that's true.
The idea comes from how the Claude Code team describes their own verification habits: you only get to hand over more work to an agent once you have a check that doesn't rely on its own say-so. Our version is much smaller, but the principle holds.
What didn't work: my first property tests were too gentle and found nothing. If your test never fails, make the inputs nastier before you trust it.
Sources
- https://claude.dev/blog/how-the-claude-code-team-thinks-about-verification/
Editorial note: We ran the steps in this guide ourselves and the screenshots come from those runs. Your results may differ with your plan, model or settings, so check anything important. Cover illustrations are AI-generated. Discovia is independent and not affiliated with Anthropic.
New guides and tools
Hands-on Claude guides, straight to your inbox
Step-by-step walkthroughs of what's new in Claude and Anthropic's tools, with real screenshots and honest results, including what didn't work. No hype, no jargon, no noise.
By subscribing, you agree to receive newsletter and marketing emails, and accept our Terms of Use and Privacy Policy. You can unsubscribe anytime.
Claude Code Lab
Tests of Claude Code features: hooks, skills, rules files and workflows, with the numbers.
More Claude Code guides →

