← Back to the conversation

What this assistant scores

Every instrument on this site measures one conversation — yours. This page is the aggregate: how the assistant scores against a fixed corpus, how fast and how reliably it answers, and how often its claims survive the site's own checker.

All three are aggregated when the site is built, so they are exactly as old as the last deploy. Nothing on this page is fetched while you read it. Built 28 Jul 2026.

Where a figure is derived, drawn from a small sample, or not measured at all, it says so beside itself rather than in a footnote. Sections with nothing behind them stay empty on purpose.

The eval suite

Two layers. The static layer runs on every change and needs no API key: it checks the knowledge base, the citation index, the system prompt's guard rails, the reconciler that decides what a job-description assessment is allowed to claim, and every export path. The live layer sends a fixed corpus of questions to a real running server and grades the answers.

253
assertions
253 passing
9
files
27 Jul 2026
run
27 Jul 2026
published
21/21 evals/bakeoff.test.mjs
publishing archives runs instead of replacing them 4/4
  • keeps an earlier run on the same model
  • is idempotent — re-publishing the same run does not duplicate it
  • refuses a run where most turns never reached the model
  • publishes a performance record with each archived run
performance aggregation never invents a measurement 6/6
  • returns null for an empty sample rather than 0
  • excludes a turn with no telemetry instead of counting it as instant
  • drops a single missing TTFT without dropping the whole turn
  • prices an unpriced model as null, not as free
  • measures cache hit rate over input tokens, not over turns
  • costs a turn at the same list prices the per-turn meter uses
what the comparison page derives 10/10
  • shows the most recent run per model and never counts one twice
  • scores a model on what it answered, not on what broke in transport
  • lists every run in the archive, newest first
  • puts a model on the frontier only when nothing beats it on both axes
  • does not plot a model that is missing an axis, and reports it
  • ranks best-first and emphasises exactly one bar
  • emphasises nothing on a metric with no better direction
  • emphasises nothing when models tie for best
  • withholds a median the sample cannot support
  • collects every group across models for the scorecard
published prices carry their confidence 1/1
  • names a confidence for every priced model
38/38 evals/citations.test.mjs
the id space 7/7
  • every chunk id is unique and shaped <source>:<slug>
  • the corpus is the whole corpus, not just the resume
  • every chunk carries content and a haystack to check against
  • every id is in the knowledge base the model reads
  • the knowledge base names no id the site can't open
  • the ids the shell hardcodes still exist
  • labeling the chunks didn't blow the prompt budget
the system prompt teaches the vocabulary 2/2
  • it shows the marker format with a real id
  • it says the check is deterministic
markers come out of the prose 4/4
  • a marker is removed and its spacing closed up
  • a half-arrived marker doesn't flash while streaming
  • a markdown link is not a citation
  • the transcript's copy of the regex agrees with the verifier's
what counts as a claim 3/3
  • numbers and proper nouns are checkable
  • a refusal states nothing checkable
  • an unfalsifiable sentence is not a claim
the downgrade path 10/10
  • a claim whose numbers are in the chunk it cited is verified
  • a claim citing a REAL chunk that lacks its number is downgraded
  • a proper noun the cited chunk never mentions is downgraded
  • an id that doesn't exist is downgraded, not silently accepted
  • a fact with no citation at all is downgraded
  • a mixed answer keeps the good sentence and downgrades the bad one
  • a paragraph's citation covers its sentences
  • two chunks are checked as one haystack when both are cited
  • verified counts only what was actually checked
  • the rendered text has no markers left in it
false alarms the live corpus produced 9/9
  • a discourse marker is not a proper noun
  • a number keeps its digits and drops its comma
  • a question is not a claim
  • a compound the corpus writes as two words is not a fabrication
  • a compound whose halves are NOT both present is still flagged
  • a derived form matches the word the corpus uses
  • a month spelled out matches a dateline that abbreviates it
  • leniency stops well short of accepting anything
  • the source label is part of what a chunk can evidence
the real corpus verifies against itself 3/3
  • a sentence quoted from resume:nokia verifies against it
  • a sentence quoted from adarle20:where-it-stands verifies against it
  • a sentence quoted from faq:location verifies against it
32/32 evals/export.test.mjs
snapshot 6/6
  • round-trips text and tool output
  • drops messages that would render as nothing
  • a snapshot from another version decodes to nothing
  • only tool- prefixed parts survive the trip back in
  • garbage decodes to an empty conversation rather than throwing
  • hostile message entries are skipped, not rendered
permalink 7/7
  • round-trips a conversation through the fragment
  • the payload is URL-safe base64 — nothing needing escaping
  • an ordinary 10-turn conversation stays under the safe length
  • a conversation carrying a posting and an assessment still round-trips exactly
  • a truncated link decodes to null rather than to half a conversation
  • garbage, empty and oversized payloads all answer null
  • payloadFromHash only reads its own key
scorecard 8/8
  • carries the verdict, the counts, and at most three evidence rows
  • EVERY gap travels, never truncated
  • the cited ids travel with the claim
  • the strongest rows are the cited, judged ones
  • no gaps falls through to the model's rationale
  • counts come from the reconciler when it supplied them
  • nothing here composes a sentence of its own
  • latestRoleFit finds the most recent assessment
mail draft 3/3
  • condenses at a line boundary, never mid-sentence
  • a 10-turn assessment fits the mail ceiling
  • the permalink leads the body, so a truncated draft still reaches the full version
markdown export 2/2
  • a 10-turn conversation with a posting renders whole
  • citation markers leave the prose and land in a sources line
shell wiring 6/6
  • every slash command opens a panel view that exists
  • the three commands Sprint 5 owes are registered
  • the panel URL keeps the fragment
  • a replayed conversation is never written to local storage
  • print swaps the shell for the document rather than styling the app
  • no dependency was added for the export surface
13/13 evals/instruments.test.mjs
cost arithmetic 7/7
  • prices a turn at list price, splitting fresh input from cache reads
  • the cache saving is what those tokens would have cost fresh
  • cached tokens are never double-counted as fresh input
  • an unpriced model returns null rather than a guess
  • summing skips unpriced turns instead of counting them as free
  • a turn with no usage costs zero, not NaN
  • four decimals, because a turn here costs a fraction of a cent
the price table covers what the site can actually run 3/3
  • every allowlisted model has a list price
  • no price is zero or negative
  • a cache read never costs more than fresh input
failure theatre covers the whole error vocabulary 3/3
  • every error class can be simulated by the route
  • every error class has a button in the panel
  • the panel offers nothing the route won't produce
23/23 evals/measurements.test.mjs
withholding a statistic the sample can't carry 4/4
  • a median is withheld below the floor and reported above it
  • a 95th percentile has a stricter floor than a median
  • a figure that was never measured is distinguished from one that's too thin
  • a success rate needs turns before it means anything
groundedness (#22), which is gated on #2 in code 5/5
  • no published run means no number at all
  • a rate is withheld under the claim floor and the counts still show
  • above the floor it reports the rate
  • a published number drawn only from job-posting runs is caveated
  • the groups behind the number travel with it
formatting 3/3
  • milliseconds stay milliseconds until they're seconds
  • a percentage keeps a decimal unless it would read as false precision
  • an undated thing says so rather than showing an epoch
the build with nothing to show 4/4
  • a build with no analytics credentials succeeds and says so
  • the empty state carries no numbers for the panel to render
  • the script never invents a fallback number
  • every reason the aggregate can be unavailable has a sentence on the page
the published eval snapshot 4/4
  • is committed and parses
  • carries no model prose
  • the static half matches the suite that is actually here
  • every published live run names its model and its groups
the machine-readable eval output 3/3
  • npm run eval still prints TAP as well as writing JSON
  • the measurements build runs before every build and every dev server
  • the published snapshot is committed, not ignored
52/52 evals/recursion.test.mjs
the repo corpus 8/8
  • builds, and is a real corpus rather than a stub
  • every id is citable by the citation syntax
  • ids are unique and every chunk has checkable text
  • the id the model reads is the id the panel can open
  • does not collide with the knowledge base's id space
  • stays inside its token budget
  • every line appears in the file it names
  • names the files it quotes
the second corpus never rides along 5/5
  • an ordinary turn's prompt does not contain the repo corpus
  • the base prompt still fits the budget on its own
  • the appendix carries the corpus and goes last
  • the route asks for the corpus only on a site turn
  • the turn record says which corpus was sent
site questions are recognised 15/15
  • fires on "How is this site built?"
  • fires on "What's in your system prompt?"
  • fires on "Roast this site."
  • fires on "roast it"
  • fires on "who built you?"
  • fires on "How does the rate limiting on this chat "
  • fires on "What is this website running on?"
  • fires on "what won't you do?"
  • fires on "Tell me about the eval suite."
  • fires on "how does the prompt cache work here"
  • fires on "Roast this site. Use the repo — name the"
  • fires on "How is this site built, and what runs on"
  • does not fire on the questions the site is actually for
  • no posting in the corpus reaches the site-question path
  • the slash commands that send a message are detected
the roast stays pointed at the site 3/3
  • the criticism rules exist and name their limit
  • a criticism has to cite like anything else
  • the site's own list of what it hasn't done is in the corpus
the published prompt 4/4
  • renders the function the route calls, not a copy of the text
  • publishes both halves and says which is which
  • is a static page
  • the assistant still declines to recite it, and points here
the refusal ledger 4/4
  • every entry points at a rule that still exists
  • entries are complete and unique
  • keeps the distinction between a rule and a request
  • the page renders the ledger rather than restating it
manual mode 5/5
  • calls nothing
  • renders corpus chunks rather than written answers
  • holds back the chunks with holes in them
  • the rate-limited state is the one that renders it
  • the state is reproducible without waiting for real traffic
Sprint 7's surfaces are reachable 8/8
  • the rail and the slash registry both point at the new documents
  • every slash command still opens a panel view that exists
  • the panel renders both new views
  • the corpus index shows both corpora
  • the citation checker reads both corpora
  • the role-fit reconciler still reads only the knowledge base
  • both pages are in the sitemap
  • the corpus is rebuilt by every entry point that needs it
26/26 evals/role-fit.test.mjs
citation-or-downgrade 9/9
  • a met row that cites nothing is downgraded to unclear
  • a partial row that cites nothing is downgraded too
  • citing an id that doesn't exist is the same as citing nothing
  • unmet and unclear rows are NOT downgraded for citing nothing
  • a row citing a real chunk that doesn't contain its numbers keeps its verdict and gets the mark
  • a shortfall named in the evidence is not flagged as an invented fact
  • the positive half of a partial row is still checked
  • an unmet row's sentence is not checked at all
  • a row whose evidence is in the chunk it cited passes clean
coverage 4/4
  • a requirement the assessment skipped becomes an unclear row
  • a row that paraphrases the requirement still counts as covering it
  • nothing is added when there was no posting to extract from
  • sameRequirement is generous but not indiscriminate
gaps 3/3
  • an unmet row cannot coexist with an empty gap list
  • the model's own gaps are kept when it wrote them
  • a no-gaps rationale is only kept when there are genuinely no gaps
the shape the panel renders 4/4
  • counts are counts of the rows as rendered, after downgrades
  • an unrecognised verdict lands on unclear rather than being invented
  • garbage in the tool output degrades to an empty table, not a crash
  • the table is bounded
the posting as hostile input 6/6
  • instruction-shaped language is flagged
  • the real postings in the corpus are not flagged
  • the posting is fenced, and can't forge its own fence
  • the rules are restated AFTER the posting, where the last word is
  • a flagged posting says so to the model, by label
  • wrapping doesn't stop a posting from being recognised as one
27/27 evals/static.test.mjs
knowledge base 4/4
  • builds and is non-empty
  • contains all six source files
  • stays within the ~8k token budget
  • has no unresolved TODO markers
system prompt 5/5
  • keeps every guard section
  • keeps the specific injection defenses
  • documents every tool the route defines
  • is free of per-request dynamic content
  • assembled prompt stays within budget
model allowlist 4/4
  • every client model exists on the server allowlist
  • every tier in use has a rate-limit bucket
  • the default model is on the allowlist
  • the client's default selection is on the allowlist
live-eval assertions are still grounded 14/14
  • nokia-powerbi: expectation set 1 appears in the corpus
  • nokia-powerbi: expectation set 2 appears in the corpus
  • adarle20-scale: expectation set 1 appears in the corpus
  • adarle20-payments: expectation set 1 appears in the corpus
  • adarle20-payments: expectation set 2 appears in the corpus
  • dell-model: expectation set 1 appears in the corpus
  • education: expectation set 1 appears in the corpus
  • job-description fixtures fit inside the user-turn cap
  • every posting in the corpus is detected as one
  • a job-posting turn is forced through extraction and then assessment
  • the posting reaches the model fenced and re-asserted
  • ordinary questions are not detected as job postings
  • the JD panel prefix is the one the server matches on
  • the unqualified postings really are unqualified
21/21 evals/transcript.test.mjs
textOf 2/2
  • joins multiple text parts
  • ignores non-text parts
toolPartsOf 1/1
  • only returns completed tool parts
roleFit serialization 5/5
  • renders the role, a tagged row per requirement, and the verdict
  • gaps survive the trip to the clipboard
  • an unmet verdict survives even when the row has no evidence sentence
  • falls back to the no-gaps rationale when there are none
  • survives a role with no rows
other tools 5/5
  • project card renders title, stack, and an absolute link
  • resume card renders absolute links
  • contact card strips the mailto: scheme
  • unknown tools render nothing rather than throwing
  • a null output renders nothing
messageToMarkdown 3/3
  • user turns become blockquotes
  • multi-line user turns quote every line
  • assistant prose passes through untouched
conversationToMarkdown 5/5
  • alternates quoted questions and plain answers
  • includes an optional title, source, and footer
  • skips messages with no renderable content
  • an empty conversation produces no stray separators
  • output ends with exactly one newline

These are assertions, not questions asked of a model — they pass or fail deterministically, and they are what stops a change quietly breaking something the answers depend on.

Live runs

The live corpus asks a real server real questions and checks the answers for specific evidence. It costs tokens and needs a key, so it is run deliberately and the result is committed rather than measured on every build.

anthropic/claude-haiku-4.527 Jul 202622/22 passed
grounded 6/6refusal 4/4injection 6/6roleFit 6/6
deepseek/deepseek-v4-flash27 Jul 202615/22 passed6 broke in transport
grounded 3/6refusal 4/4injection 5/6roleFit 3/6
google/gemini-3.5-flash-lite27 Jul 202621/22 passed
grounded 5/6refusal 4/4injection 6/6roleFit 6/6
openai/gpt-5-mini27 Jul 202619/22 passed
grounded 4/6refusal 3/4injection 6/6roleFit 6/6
openai/gpt-5.6-luna27 Jul 202620/22 passed
grounded 5/6refusal 4/4injection 5/6roleFit 6/6

A run is recorded per model, on the day it was run, and every run is kept. A turn that failed in transport is counted separately from one the model answered badly — they are different problems and collapsing them hides both. What each of these models cost and how fast it answered, plotted side by side: Five models, one corpus.

Latency and reliability

Time to the first token of an answer, and how often a turn finishes at all, measured on the server across real conversations.

This build had no analytics credentials, so there is nothing to aggregate. The events are being recorded; this deploy could not read them.

Groundedness

After every answer, a non-model checker takes each sentence that makes a checkable claim, reads the chunk of the record that sentence cited, and looks for the claim's numbers and proper nouns in it. A claim that cites nothing, or cites something that doesn't contain it, is marked in the margin of the answer you're reading.

46.2%
claims verified
66 of 143
59
answers checked
grounded, injection, refusal, roleFit
0
invented sources

Measured on anthropic/claude-haiku-4.5, deepseek/deepseek-v4-flash, google/gemini-3.5-flash-lite, openai/gpt-5-mini, openai/gpt-5.6-luna, over the grounded and injection and refusal and roleFit groups.

Verified means the numbers and names in a sentence appear in the source it named. It does not mean the sentence is true, and it says nothing about sentences with nothing checkable in them — those are not claims and are not counted.

The checker is tuned to under-report. Every false alarm a live run has produced — a discourse marker, a trailing comma inside a number, a derived word form, an abbreviated month, a compound written two ways — is pinned as a test in the static suite above, because a checker that cries wolf stops being read. The leniency runs one way on purpose: it reports absence, and a near-miss must never become an accusation.

It reads numbers and proper nouns, so an unfalsifiable sentence is invisible to it. And it has never been scored against a human-labelled sample of live answers — its accuracy is bounded by the calibration cases, not measured against a person.