Evals (evaluations) are structured tests that score an AI agent's behavior on representative tasks. When you change a prompt, swap a model, add a tool, or edit a skill, evals tell you whether the agent got better or worse — before production users find out.
Unlike unit tests that assert exact strings, agent evals often score for correctness, safety, format, or task completion — sometimes with another model as a judge, sometimes with deterministic checks.
What evals typically cover
- Task success — Did the agent achieve the goal?
- Tool discipline — Did it call the right tools with valid arguments?
- Safety — Did it refuse or escalate when it should?
- Regression — Did a "small" prompt tweak break a known-good case?
- Cost / latency — Did quality come at an unacceptable spend?
Evals vs. tracing
Tracing records what happened on a real run (every model and tool call). Evals deliberately probe whether behavior meets a bar. You need both: traces diagnose a failure; evals prevent shipping the next one.
Blueprint implication
If the agent will evolve — and it will — plan for an eval set alongside tools and skills. Start with the ten cases that would embarrass you in production. Expand from failures you actually see.
Related reading
See Prompt Engineering for instruction changes evals should catch, and Guardrails for automated safety checks that complement scored tests.