Statistical consulting specialties

AI evaluation audits

Is model B better than model A on the work your users actually do?

A small eval can put a new model, prompt, or vendor ahead by a handful of examples. Before you migrate, I check how much the estimated gain could move and whether the test captures costly production failures.

Request an eval audit

What the work covers

Compare the same cases
Score both versions on the same tasks. Estimate uncertainty in the paired difference, accounting for repeated examples from the same conversation or customer.
Check the scoring
Review disagreements and spot-check automated judges against human ratings. Look for rubric choices, answer order, or test-set reuse that could change the winner.
Test the link to production
Build a held-out set from representative tasks, including rare, costly mistakes. Where production outcomes are available, check whether offline scores predict them. Otherwise, specify what a shadow run or pilot should measure.