The Discovia
Guides Tools
Fact-checksAbout
Search
Get the email
The Discovia
All guides Free tools

Topics

Claude CodeAgents & APIClaude AppsUse Cases Fact-checks

More

How we testAbout DiscoviaScience archiveSearchContactGet the email

© 2026 Discovia

The Discovia

Claude, explained by doing. Practical, step-by-step guides to Claude and the latest Anthropic updates — real use cases with real screenshots.

Categories

  • Claude Code
  • Agents & API
  • Claude Apps
  • Use Cases

Discovia

  • All guides
  • Free tools
  • How we test
  • Fact-checks
  • About Us
  • Contact
  • Search
  • Science Archive

Discovia is an independent publication. It is not affiliated with, endorsed by, or sponsored by Anthropic. Claude is a trademark of Anthropic, PBC.

© 2026 Discovia. All rights reserved.

Terms of Use·Privacy Policy
@TheDiscovia

Found this useful?

Send it to someone who uses Claude.

Post LinkedIn
GuideClaude CodeAdvanced30 min to do

Claude Code said "all tests pass." Then I threw 20,000 random bills at it.

Its own tests were green. A property test found a real bug in about a second. Here's the setup that makes Claude Code check its own work, with every command, the real numbers, and what didn't work.

Green tests hid a real bug; a property test found it in a second
CC
Claude Code Lab
·November 19, 2026·7 min read
Claude Code said "all tests pass." Then I threw 20,000 random bills at it.

Level: Advanced. About 30 minutes. Needs Node 20+ and a Claude Code login.

Ask Claude Code to write a function and its tests, and you'll usually get green checkmarks. I wanted to know how much those checkmarks are worth. So I gave it a small job, then tried to break the result in a way its own tests never would.

It broke. And the setup I used to catch it takes about ten minutes to copy.

The job: split a restaurant bill

I asked for a function that splits a bill, in cents, between people in proportion to a weight each. Weights [1, 1, 2] mean the third person pays double. The shares must add up to exactly the total, and a weight of zero means you pay nothing.

Splitting money is a good test because rounding is sneaky. Here is the version most of us write first:

const splitBill = (total, weights) => {
  const W = weights.reduce((a, b) => a + b, 0);
  return weights.map(w => Math.round(total * w / W));
};

It looks right. It even passes the obvious example test (1000 split [1,1,2] gives [250, 250, 500]). Then you split $1.00 three ways and it hands back [33, 33, 33]. That's 99 cents. A cent has vanished.

The simple version passes the example and fails on three equal shares

Nobody writes a test for exactly that case until it bites them. That's the point of what follows.

Step 1: let Claude write it

I made an empty project and gave Claude Code the job in plain English. No tricks.

mkdir split-demo && cd split-demo
npm init -y && npm i -D fast-check
claude -p "Write a function splitBill(totalCents, weights) in src/split.js ..."

It took 83 seconds and 16 turns. To its credit, it didn't write the naive version above. It used the "largest remainder" method: give everyone their rounded-down share, then hand the leftover cents out one at a time to whoever was closest to rounding up. It wrote nine tests and all nine passed.

Claude's nine example tests, all passing

Two things I noticed. It had to change my npm test script, because the original didn't find the test folder in its sandbox. And fast-check was already installed, but Claude decided not to use it. Its explanation was that a handful of example tests covers what I asked for.

Fair enough. That's the part I wanted to check.

Step 2: write the test Claude didn't

An example test says "for this input, expect this output." A property test says "for any input, this must always be true," then tries thousands of inputs to prove you wrong. I used fast-check, a library for exactly this.

My three rules for a bill split:

  1. The shares add up to exactly the total.
  2. Nobody pays a negative or fractional amount, and a zero weight pays zero.
  3. Every share is within one cent of its exact fair share.

Here's the first one in full:

const fc = require('fast-check');
const { splitBill } = require('../src/split');

test('shares add up to exactly the total', () => {
  fc.assert(fc.property(
    fc.integer({ min: 0, max: Number.MAX_SAFE_INTEGER }),
    fc.array(fc.integer({ min: 0, max: 1_000_000 }), { minLength: 1, maxLength: 12 })
      .filter(w => w.some(x => x > 0)),
    (total, weights) => {
      const shares = splitBill(total, weights);
      assert.strictEqual(shares.reduce((a, b) => a + b, 0), total);
    }
  ));
});

One tip that matters: check rule 3 with exact BigInt math, not regular numbers. If the test uses the same floating-point math as the code, they can both be wrong in the same way and agree.

The first time I ran it, with weights up to 1,000, everything passed. Claude's code was good. So I pushed harder: weights up to a million, and totals up to the largest number JavaScript can store exactly. Then it failed.

Two property tests fail, with the smallest failing input shown

The smallest failing case: one person, weight 9, a bill of 8,725,724,278,030,291 cents. One person should pay the whole bill. The code returned something else.

Be honest about what this is. That bill is about $87 trillion. No restaurant will ever see it. It's a real bug, and it only shows up at absurd amounts. Still, nine hand-written example tests had no chance of finding it, and the property test found it in the first run.

Step 3: hand the failure back, and let a hook keep it honest

Now the loop. I gave Claude one sentence: the property tests are failing on large bills, fix src/split.js. Two files made sure it did it properly.

A CLAUDE.md with three rules:

- Run `npm test` before you say you are done. Never claim it works without showing the passing output.
- Never edit, skip or delete anything under test/ to make a failure go away.
- If a test still fails after 3 attempts, stop and report what you tried instead of guessing again.

And a Stop hook in .claude/settings.json. It runs your tests whenever Claude tries to finish, and if they fail, it refuses to let Claude stop:

{
  "hooks": {
    "Stop": [
      { "hooks": [ { "type": "command", "command": "npm test --silent 1>&2 || exit 2" } ] }
    ]
  }
}

That exit 2 is the important bit. It sends the test output back to Claude as a reason to keep going.

Claude found the cause in 37 seconds and 6 turns: multiplying a big bill by a big weight goes past what regular numbers can store exactly, so the answer gets rounded. Its fix was to do the division with BigInt and convert back at the end. It didn't touch a single test.

The diff Claude made: BigInt instead of floating point

I then ran 20,000 random bills with weights up to a million against the fixed version. All good.

Does the hook really matter? I tested it

I didn't want to just claim this, so I ran a control. I put the bug back and asked for something unrelated: "add a one-line comment to the top of src/split.js."

With the hook and rules, Claude noticed the failing tests, fixed the bug, and finished with everything green. It took 95 seconds. Without them, it added the comment in 17 seconds and left two tests failing. It didn't mention it.

With the hook it fixes the bug; without it, it ships red tests quietly

One caveat: I removed the hook and the CLAUDE.md together, so I can't say which one did more. I'd keep both. The hook is the one that doesn't depend on Claude remembering to behave.

The final state: 15 of 15 tests pass, plus the 20,000-run stress test

What it cost

Claude Code reported about $0.77 for the first build, $0.30 for the fix, and $0.57 for the hook run, around $1.60 in all. I was on a Pro plan, so I wasn't billed per run. Treat those as a rough size of the job, not a bill.

Try it on your own code

  1. Pick one function where being slightly wrong would hurt: money, dates, parsing, permissions.
  2. Write down two or three things that must always be true about it.
  3. Turn each into a property test (fast-check for JavaScript, Hypothesis for Python).
  4. Add the Stop hook and the three CLAUDE.md rules.
  5. Let Claude work. When it says "done," the hook and your properties decide whether that's true.

The idea comes from how the Claude Code team describes their own verification habits: you only get to hand over more work to an agent once you have a check that doesn't rely on its own say-so. Our version is much smaller, but the principle holds.

What didn't work: my first property tests were too gentle and found nothing. If your test never fails, make the inputs nastier before you trust it.

Sources

  • https://claude.dev/blog/how-the-claude-code-team-thinks-about-verification/

Editorial note: We ran the steps in this guide ourselves and the screenshots come from those runs. Your results may differ with your plan, model or settings, so check anything important. Cover illustrations are AI-generated. Discovia is independent and not affiliated with Anthropic.

Share this guide X LinkedIn

New guides and tools

Hands-on Claude guides, straight to your inbox

Step-by-step walkthroughs of what's new in Claude and Anthropic's tools, with real screenshots and honest results, including what didn't work. No hype, no jargon, no noise.

By subscribing, you agree to receive newsletter and marketing emails, and accept our Terms of Use and Privacy Policy. You can unsubscribe anytime.

Claude Code Lab

Tests of Claude Code features: hooks, skills, rules files and workflows, with the numbers.

More Claude Code guides →

More from this author

Claude Code

I gave Claude Code a 7-line rules file. It followed every rule, including the one I wrote badly.

Twelve runs, one task, and a checker that counted. Most rules in a CLAUDE.md file turned out not to be needed, because Claude copied the code around it. The ones that mattered were the ones it couldn't guess, and one of mine was ambiguous.

C
Claude Code Lab
5 min read
Read next

Comments

Related Guides

GuideClaude CodeBeginner20 min

I gave Claude Code a 7-line rules file. It followed every rule, including the one I wrote badly.

6 of 8 rules followed with no file at all

Claude Code Lab · Nov 21, 2026

GuideClaude CodeIntermediate30 min

I tried to make Claude Code leak a secret. My guardrail hook stopped the obvious way and missed the sneaky ones.

Blocked the direct read, missed 4 sneaky ones

Claude Code Lab · Nov 20, 2026

GuideClaude CodeIntermediate25 min

Claude already reviews pull requests well. A 14-line file made it review them my way.

Same 6 bugs found, a far more useful review

Claude Code Lab · Nov 19, 2026