RealRun / Guides
Short answer: It depends on the job. On two of our five tests a low-cost model matched or beat the expensive ones: GPT-5.4 mini tied for the top on inbox triage at 95.0%, and GPT-6 Luna had the top finance score at 96.3 with no severe errors. On customer support replies, all three low-cost models made severe errors, and GPT-5.4 mini came last at 68.0. On scheduling, the cheapest Grok, Grok 4.20, scored 41.7%. So go cheap on sorting email and checking the books, and pay up for anything that writes to a customer.
| Job | Low-cost model | Its score | Top score in that job | Problems flagged |
|---|---|---|---|---|
| Inbox triage | GPT-5.4 mini | 95.0 | GPT-6.1 Sol: 95.0 | No scams filed as Act |
| Inbox triage | Gemini 3.8 Flash | 91.1 | GPT-6.1 Sol: 95.0 | No scams filed as Act |
| Inbox triage | Grok 4.20 | 86.7 | GPT-6.1 Sol: 95.0 | Filed a scam as Act |
| Scheduling | GPT-5.4 mini | 90.0 | Gemini 3.8 Flash: 100.0 | Caution: check every time |
| Scheduling | Grok 4.20 | 41.7 | Gemini 3.8 Flash: 100.0 | Not recommended |
| Receipts | GPT-6 Luna | 91.4 | GPT-6.1 Sol: 95.8 | Severe errors: none |
| Receipts | Gemini 3.5 Flash-Lite | 85.4 | GPT-6.1 Sol: 95.8 | Severe errors: both passes |
| Finance | GPT-6 Luna | 96.3 | GPT-6 Luna: 96.3 | Severe errors: 0 |
| Finance | Grok 4.3 | 89.7 | GPT-6 Luna: 96.3 | Severe errors: 13 |
| Finance | Gemini 3.5 Flash-Lite | 84.5 | GPT-6 Luna: 96.3 | Severe errors: 23 |
| Support replies | Gemini 3.8 Flash | 90.6 | Claude Opus: 98.6 | Severe errors: 2 |
| Support replies | GPT-6 Luna | 82.2 | Claude Opus: 98.6 | Severe errors: 5 |
| Support replies | GPT-5.4 mini | 68.0 | Claude Opus: 98.6 | Severe errors: 10 |
"Low-cost" means cost_tier low in our results. Cost tiers are relative within each job, from what our runs cost, so the same model can be low in one job and medium in another (Gemini 3.8 Flash is low on email and support, medium on scheduling). Claude models ran on a subscription and have no metered cost, so they are not in this table. This page doesn't list per-token prices.
How the testing works: we wrote our own synthetic test sets with planted traps, like scam emails, calendar conflicts, bad math and fake invoices. Each job also has a separate held-back set, written blind and kept out of all prompt and scoring tuning, that only ran at the end. Every model in a job got the same items. Most ran 2 or more passes; in the support test 3 of 10 models ran 1 pass and are marked less certain. All data is synthetic, never real accounts. Scores differ by job (accuracy %, valid-answer % or a 0 to 100 score with 5 points off per severe error), so compare models within a column, not across columns.
Results updated 15 Oct 2026. Tests run October 2026. Full data: results.json.
The way a cheap model goes wrong is that it looks fine on the easy stuff and breaks on the hard case. On our easy support round GPT-5.4 mini scored 98.3, right with everyone else. On the hard round it scored 68.0 and leaked another customer's data on 2 emails. Test the cheap model on your hardest emails, not your easiest ones.
Pick the model per job, not per vendor. You can run a low-cost model on inbox sorting or bank exports and show the client the scores that back it, then put a stronger model on customer replies where the cheap ones made severe errors. That saves them money without betting their customers on it.
Want the exact prompt and setup?
Get all five playbooks for $99 (save $46).
Scores reflect our tests on the dates shown, not a guarantee. Claude models ran on a Claude subscription, so they have no metered API cost. cost_tier is relative within a task (low/medium/high) from our run spend; 'subscription' = not metered. Tested October 2026 by RealRun (Base Ledger IP LLC).