How to Evaluate LLMs for Production

Public benchmarks answer a question you don't have: 'which model is best on average across everything?' Your question is narrower and more important: 'which model is good enough, cheap enough, and fast enough for my task?'

Collect real inputs first

Twenty real examples from your actual workflow are worth more than two thousand synthetic ones. They carry the messiness — typos, edge cases, weird formatting — that determines whether a model survives contact with users.

Grade what matters to you

• Define pass/fail criteria a human would agree with.
• Weight failures by their real-world cost, not their frequency.
• Track cost and latency alongside quality — they're features too.

model         quality   p95 latency   $/1k calls
candidate-A     0.91        1.8s         $4.20
candidate-B     0.88        0.6s         $0.90  <- ship this

Re-run on every change

Prompts, model versions, and providers all drift. An eval you run once is a snapshot; an eval you run on every change is a safety net.