Five models, one corpus

Quality against cost

Quality against latency

The numbers underneath

One panel per measurement, each sorted best-first. Nothing here is stacked or double-encoded: every panel plots one number, and where there is a winner it is the only bar wearing the accent. One panel has no winner, and nothing in it is highlighted — a longer answer is not a better answer, it is just more of the bill.

The scorecard

Pass rate by model and by what the group is testing. There is deliberately no single intelligence score: a model that answers grounded questions well and folds under prompt injection is a different problem from one that is mediocre at both, and one number cannot tell you which you have.

grounded — does it state facts that are actually in the corpus, and say 'I don't have that' when they aren't. refusal — does it decline out-of-scope requests instead of helping. injection — role override, prompt extraction, knowledge-base dump, a fake system message, disparagement bait, persona hijack. roleFit — pasted job descriptions, graded on whether the assessment names the gaps rather than flattering the reader.

Everything, as a table

The same figures the charts plot, plus the ones no chart shows. This is the accessibility twin for every chart above — nothing on this page is reachable only by colour or only by hover.

Prices checked 2026-08-12. A median is withheld and shown as an em dash under 10 measured turns, and a 95th percentile is not computed at all — one sample per case cannot support one. “claims verified” is the site's own citation checker run over that model's answers; it counts only sentences with something checkable in them.

The run archive

Every run ever published, per model, with the date it was measured. Until this sprint a second run on a model overwrote the first, which meant a regression was invisible and a figure measured twice looked exactly like one measured once. Runs are now kept; the charts above read the most recent per model.

What this does not measure

One sample per case

Each model answered each question once. Every timing figure here is a median over about twenty turns, which is enough to rank models and nowhere near enough for a 95th percentile — so none is computed. Re-running the same corpus on the same model would move these numbers, and nothing here says by how much.

Latency includes the day it was measured

Time to first token is the server's own measurement, with the browser network excluded — but not the hop from a home connection to the provider, nor whatever load that provider was under that afternoon. It is a real measurement of a real request; it is not a claim about the model in a datacentre.

The grader is a string matcher, not a judge

Cases pass by containing specific evidence — a name, a number, a refusal. That keeps runs free and reproducible, and it means a wrong answer containing the right number still passes, and a right answer phrased unusually can still fail. This is a regression net, not a quality score, and a pass rate here is not an accuracy claim.

Costs are list prices, not an invoice

Every dollar figure is computed from published per-token rates on the date below and the tokens the run actually used. It is an estimate at list price. Prompt-cache rates are mostly derived from a published discount rather than separately published, and one model's prices could not be confirmed at all — it is marked in the table.

A broken turn is counted, but it is not counted as a wrong answer

Some turns died mid-stream and never produced an answer. Those are excluded from the pass rate — scoring a dropped connection as bad judgment is the confusion this whole site is built to avoid — and reported on their own panel instead. But one afternoon of requests is a thin basis for a reliability claim: it says a model dropped streams here, then, not that it drops one in four everywhere.

Token counts are each provider's own

Every model got byte-identical prompts, so the spread in input tokens is mostly the tokenizers disagreeing about how to count the same text, not one model being sent more work. Seconds and dollars compare across rows; tokens really do not.

The corpus is narrow on purpose

Twenty-two cases about one person's résumé and the ways someone might try to break a chatbot answering questions about it. That is exactly the workload this site runs, which is why the numbers are useful here — and why they do not transfer to yours.