Before real people use something with rules, let bots use it hundreds of times. Automated play-throughs and fast simulations find problems that a few manual tests never reach.

Manual testing answers one question well: does this work when I try it? It is much worse at a different question: what happens over hundreds of runs, with choices nobody on the team would think to make?

Any product with rules has that second question. Pricing and discounts, queues, matching, approval chains, credit systems, game balance. The bugs that hurt most are often not crashes. They are states the system gets stuck in, or loops that never end.

A true story from today

I spent today building the solo mode for BreakFriend, a fort duel game from my small studio, Break n Break. In solo, you play against one to five computer forts. Each round, everyone builds and fires.

I tested it the way most people would: I played a few games by hand. They seemed fine.

Then I let bots play it. Two kinds:

  • A scripted player that played whole games in a real browser, tapping through the same screens a person would.
  • An instant "auto game" that ran the game's rules with only computer players and no drawing, so dozens of games finished in seconds.

They found two problems I had not seen.

Problem one: the computer was too good

The first computer aim was so accurate that a human was knocked out in round two. In my few manual games I had put that down to my own bad luck. Across many simulated games, it was clearly the design.

So I loosened the aim. That fixed round two. It also caused the second problem.

Problem two: games that never ended

With looser aim, simulated games started running past 100 rounds with no winner.

The reasons were simple once the bots made them visible. Forts repaired as fast as they were being hit. And bolt towers, which shoot down incoming shells, kept doing it forever. Nobody could get a shot through, and nobody ran out of the resources to keep rebuilding.

No person would play 100 rounds to find that. They would get bored at round 20, decide the game was dull and close it. You would never know why.

The fixes

Three changes, each aimed at one cause:

  1. Shrinking income. Sunstone, the game's currency, now comes in more slowly as rounds pass. Late in a game, you cannot rebuild everything you lose.
  2. Aim that sharpens late. The computer starts loose, so early rounds are fair, and gets more accurate as the game goes on.
  3. Bolt towers wear out. In solo, a bolt tower cracks each time it fires, so a wall of them cannot hold forever.

Then I ran the simulations again. Every simulated game finished. One-on-one games took about 15 rounds. Games against five computers took 20 to 25. Hard finished faster than Easy, which is the right way round.

That last check matters. It is not enough for a fix to remove the bug. The bots also confirmed that the difficulty levels still meant something.

Play it free at breaknbreak.com/breakfriend. It's now part of the Break Board Games app, and the same bots play every one of the studio's 38 games before each release.

Why this matters outside games

Replace "forts" and "sunstone" with your own product and the pattern holds:

  • A credit or points system where users can earn faster than they spend, so the balance grows forever.
  • A support queue where reopened tickets come back as fast as they are closed.
  • An approval chain where two rules send a request back and forth with nobody able to approve it.
  • A matching system where certain users never get matched.
  • An automated retry that keeps retrying a request that can never succeed.

Each of these looks fine in a demo. Each one only shows up over many runs, or with combinations of choices nobody on the team would make by hand.

Two kinds of bots, two jobs

I find it useful to keep the two kinds separate.

The scripted player in a real browser checks that the product works end to end, the way a person uses it. It taps buttons, waits for screens and checks that each step leads to the next. It is slow, but it catches broken flows, missing screens and things that only fail in a real browser.

The fast simulation skips the screens and runs only the rules. It can play thousands of rounds in the time a browser plays one. It catches balance problems, stalemates, runaway loops and states that cannot be won.

On an earlier game, the same approach caught units queuing so that only four ever fought at once, and a final mission that nobody could win. Neither showed up in manual play.

You want both. The first tells you the product works. The second tells you the rules work.

How to set it up

You do not need a large test suite to start.

  1. Separate the rules from the screens. If your rules can run without the interface, you can simulate them. This is good design anyway.
  2. Write a simple bot. It does not need to be clever. A bot that makes reasonable random choices finds a lot.
  3. Decide what "finished" means. A game ends. A ticket closes. A request gets approved or rejected. Write that down.
  4. Run many times and record outcomes. How long did each run take? Did it finish at all? What was the spread?
  5. Set limits that should never be crossed. In my case, no game should pass 100 rounds. In yours, maybe no request should bounce more than five times.
  6. Run it again after every change. A fix for one problem often causes another, as loosening the aim did here.

Coding agents are good at writing this kind of harness. I wrote about giving agents a clear runbook for browser testing in AI agents as QA testers.

What bots cannot tell you

Bots find states, not feelings. A simulation can tell you every game finishes in about 15 rounds. It cannot tell you whether those 15 rounds are fun, or whether a person understands why they lost.

That still takes people. Once the bots stop finding problems, put the product in front of real users and watch. I use bots so that the time people spend testing goes on the questions only people can answer.

A few manual tests show you the product works. Hundreds of automated runs show you where it gets stuck.

How do you test the rules in your product today, beyond trying it by hand? I would like to hear what has caught problems for you. Email me if you want to compare notes.

#Testing #Simulation #AI #ProductDesign