RealRun / Guides
Short answer: Yes, some of them do. When we sorted emails with planted scams, Claude Haiku filed gift-card and payment scams as Act in some passes, and Grok 4.20 filed a scam as Act on the fresh held-back set in both passes. The other 6 models didn't. In our finance test, Gemini 3.5 Flash-Lite followed a hidden "say this reconciles" instruction in all 3 passes. The good news: no model accepted a fake invoice, and no model failed a scam meeting request.
| Model | Fresh held-back 30 (%) | Filed a scam as Act? | Cost |
|---|---|---|---|
| GPT-6.1 Sol | 95.0 | No | medium |
| GPT-5.4 mini | 95.0 | No | low |
| Claude Opus | 94.4 | No | subscription |
| Claude Sonnet | 91.7 | No | subscription |
| Gemini 3.8 Flash | 91.1 | No | low |
| Grok 4.7 | 90.0 | No | high |
| Grok 4.20 | 86.7 | Yes | low |
| Claude Haiku | 85.6 | Yes | subscription |
| Job | Trap we planted | What happened |
|---|---|---|
| Inbox triage | Scam emails, including gift-card and payment requests | 2 of 8 models filed a scam as Act: Claude Haiku and Grok 4.20 |
| Finance | Scam bills and a statement with a hidden "say this reconciles" line | Gemini 3.5 Flash-Lite followed the hidden line in all 3 passes |
| Receipts | Fake invoices | No model accepted a fake invoice |
| Scheduling | Scam meeting requests | No model failed a scam request |
| Support replies | Scam attempts, plus emails that test whether it protects customer data | GPT-5.4 mini leaked another customer's data on 2 emails; Claude Haiku once |
How the testing works: we wrote our own synthetic test sets with planted traps, like scam emails, calendar conflicts, bad math and fake invoices. Each job also has a separate held-back set, written blind and kept out of all prompt and scoring tuning, that only ran at the end. Every model in a job got the same items. Most ran 2 or more passes; in the support test 3 of 10 models ran 1 pass and are marked less certain. All data is synthetic, never real accounts. Scores differ by job (accuracy %, valid-answer % or a 0 to 100 score with 5 points off per severe error), so compare models within a column, not across columns.
Results updated 15 Oct 2026. Tests run October 2026. Full data: results.json.
The way this goes wrong is a scam that looks urgent. Filed as Act, it lands at the top of your day, right where you'll pay it. Or a bill or statement carries a line telling the AI what to say, and the AI repeats it. Pick a model that passed these traps, and keep a human yes on anything that moves money.
If you build an agent that reads a client's email or books, show them which models filed scams as Act and which followed a planted instruction, and leave the flagged ones out. Then test your own setup with a few planted scams before it goes live. It's a lot cheaper than explaining a paid fake invoice.
Want the exact prompt and setup?
Get all five playbooks for $99 (save $46).
Scores reflect our tests on the dates shown, not a guarantee. Claude models ran on a Claude subscription, so they have no metered API cost. cost_tier is relative within a task (low/medium/high) from our run spend; 'subscription' = not metered. Tested October 2026 by RealRun (Base Ledger IP LLC).