LLM outputs are non-deterministic. Run the same prompt twice and you may get two different answers. Ship without a process to measure those answers and you will spend your time chasing regressions instead of building features. This is the problem evals solve, and the strongest framing I have heard on the subject comes from Doug Guthrie, solutions engineer at Braintrust, in his Evals 101 workshop at the AI Engineer World's Fair in San Francisco.
An eval is a structured test that checks how well your AI system performs — quality, reliability, correctness.
Why Evals Matter
Evals answer the questions that actually decide whether an application improves or degrades: “When I change the underlying model, is my app better or worse? When I change my prompt?” Without them, a model swap or a prompt tweak is a leap of faith.
They are often compared to unit tests, but Doug's CEO, Ankur Goyal, frames them as playing offense rather than defense: not a safety net, but the rigor that lets you build things that are actually deployable. Offline and online evaluation together form a flywheel: real user logs, filtered and fed back into your datasets, keep improving the application.
The Three Ingredients of an Eval
1. The Task
The prompt or code you are evaluating — from a simple prompt to a full agentic workflow with tool calls (RAG, web search). Mustache-style templating injects variables (question, history, metadata); multi-turn messages and even prompt chaining are supported. The only requirement: an input and an output.
2. The Dataset
Real-world examples the task runs against. Only the input is required; you can add an expected output (useful for exact-match or Levenshtein scoring) and metadata for filtering. Production traces flow straight into datasets.
3. The Scores
“The logic behind your eval.” Two families: code-based scorers (binary or heuristic checks, in TypeScript or Python, versioned in your repo) and LLM-as-a-judge — you hand the model the output plus criteria (“excellent, correct, poor”) mapped to 0.5 or 1. Braintrust's autoevals package ships ready-made scorers.
Method: Start Small, Then Diagnose
Do not chase a “golden dataset” or a perfect score on day one. Start small — even a Levenshtein check “establishes a baseline with very little work.” Then use a simple diagnostic matrix:
- Good output but low score → your eval needs improving.
- Bad output but high score → your scoring needs improving.
Further tips from the workshop:
- Score with a higher-quality model than the one running the prompt.
- Split scoring into focused axes — for a changelog generator, evaluate accuracy, formatting and correctness separately.
- Test your scoring prompts in the playground first.
- Limit context to the relevant input and output.
Offline vs. Online — and the Same Scores in Production
Offline evaluation happens before deployment: iterate on tasks, datasets and scorers, fix issues before they reach users. Online evaluation traces the running application in production — inputs, outputs, intermediate steps, tool calls, plus cost, tokens and latency. Instrumentation starts with a logger pointed at a project (a feature container for prompts, scorers and datasets); wrapping your LLM client is enough to ingest metrics (“zero-lift” with the Vercel AI SDK).
Crucially, the same scores defined offline apply to incoming production logs — sampled, not exhaustively: “no need to score every log — 10%, 20%.” You can even score individual spans: the relevance of a RAG retrieval step, or the agent's question rewrite — if that rewrite is bad, “everything that follows collapses.” Automations alert you when a score drops below a threshold, and evals plug into CI/CD as quality gates.
Human-in-the-Loop Completes the Flywheel
Two forms of human input matter. Human review: a dedicated interface where reviewers rate logs against configurable scores — larger organizations create a hybrid “product manager + LLM specialist” role; smaller ones involve engineers. User feedback: thumbs up/down with a comment, logged into the platform. Filtered views (e.g. “feedback = 0”) focus reviewers on significant logs, which then join the offline datasets. This is the flywheel — the human provides the ground truth. As Doug puts it, automation plus humans “adds value to the process, it's not going away.”
Nota Bene — Vladimir Levenshtein: the Levenshtein distance mentioned above is named after Vladimir Iosifovich Levenshtein (1935–2017), a Soviet mathematician at the Keldysh Institute of Applied Mathematics in Moscow. In 1965 he defined the edit distance: the minimum number of single-character operations — insertions, deletions or substitutions — needed to turn one string into another. What began as a result in coding theory became the backbone of spell-checkers, fuzzy search, DNA sequence alignment and, today, a cheap but powerful baseline scorer for LLM outputs. He received the IEEE Richard W. Hamming Medal in 2006.
Five Takeaways
- Start small: one task, one input dataset, at least one score. Establish a baseline, then iterate — no golden dataset.
- Diagnose before optimizing: good output/low score → fix your evals; bad output/high score → fix your scoring.
- Treat scorers as code: judge model stronger than the prompt model, focused axes, tested scoring prompts, limited context.
- Version and bridge offline/online: define prompts, datasets and scorers in code, push them to the platform, reuse the same scores in production with sampling, regression alerts and CI checks.
- Close the loop: filtered user feedback + human review → offline datasets → a better application in production.
This article was strongly inspired by Doug Guthrie's “Evals 101” workshop on YouTube, recorded at the AI Engineer World's Fair. If you build LLM applications, the 48 minutes are worth your time — and so is braintrust.dev, the evaluation platform used throughout the talk.