Statistical consulting specialties
AI evaluation audits
Is model B better than model A on the work your users actually do?
A small eval can put a new model, prompt, or vendor ahead by a handful of examples. Before you migrate, I check how much the estimated gain could move and whether the test captures costly production failures.
Request an eval auditWhat the work covers
- Compare the same cases
- Score both versions on the same tasks. Estimate uncertainty in the paired difference, accounting for repeated examples from the same conversation or customer.
- Check the scoring
- Review disagreements and spot-check automated judges against human ratings. Look for rubric choices, answer order, or test-set reuse that could change the winner.
- Test the link to production
- Build a held-out set from representative tasks, including rare, costly mistakes. Where production outcomes are available, check whether offline scores predict them. Otherwise, specify what a shadow run or pilot should measure.