Evaluation: how you know the system improved

What a good set is made of

Real cases from the field, classified into categories, including edge cases and ones with no good answer. For each case: input, and a desired answer or acceptance criterion.

Every incident found in production adds a row. That way the set grows in the right places.

Three scoring methods

Automated rules — correct structure, right number, source cited. Cheap and precise where possible.

Model-based evaluation against written criteria — fast and cheap, and needs calibration against human judgment.

Human review of a sample — expensive, and the anchor of truth.

How to read results

Compare to the previous version, not to an absolute score. And check by category breakdown — a stable average often hides a drop in one category that is precisely the important case.