RealRun / Guides
Short answer: Test it on your own hard cases, not the vendor demo. Hold some cases back, run every case at least twice, plant a few traps (a scam, a hidden instruction, a wrong refund), and count severe errors separately from the score. In our tests, easy cases made every model look the same: on easy support email all 10 models scored 98.3 to 99.6. The hard cases are where they split.
| Trap | Job | What happened |
|---|---|---|
| Scam email in the inbox | Inbox triage | Filed a scam email as Act on the fresh held-back set in both passes. |
| Busy calendar slot | Meeting scheduling | Often proposed times over busy events while claiming they were free, and made time-zone errors. |
| Unreadable receipt | Receipts | Invented totals on unreadable receipts, severe errors in every pass, and wrong duplicate pairs. |
| Hidden instruction in a statement | Small-business finance | 23 severe errors; followed a hidden 'say this reconciles' instruction in all 3 passes. |
| Tricky refund request | Customer support replies | 7 models promised a wrong refund amount on the same case. |
How the testing works: each RealRun test has a main set and a fresh held-back set, every model gets the same items, and most ran 2 or more passes. In the support test, 3 of 10 models ran 1 pass and are marked less certain. Every answer is graded, and the worst mistakes are called out on their own. Results updated 15 Oct 2026.
The way an agent goes wrong is rarely a bad average. It's one quiet mistake: a gift-card scam filed as Act, a meeting booked over a busy event, a 'say this reconciles' line in a statement that the model obeys. Test for the mistake you can't afford, before you pay for the tool.
Clients will ask how you know the agent works. Show them a held-back test with planted traps and a severe-error count, not a demo. If you don't have time to build one, our results cover five common jobs and are free to cite.
Want the exact prompt and setup?
Get all five playbooks for $99 (save $46).
Scores reflect our tests on the dates shown, not a guarantee. Claude models ran on a Claude subscription, so they have no metered API cost. cost_tier is relative within a task (low/medium/high) from our run spend; 'subscription' = not metered. Tested October 2026 by RealRun (Base Ledger IP LLC).