linkdrift
Home-made Evals Taste Better

Home-made Evals Taste Better

kig
@kig·2.6K followers
·2 min read

If you are still writing software, using agents, and have never used Evals, this post is for you. A lead typed: "Got married in June, haven't figured out how we're filing yet." A model read that and

If you are still writing software, using agents, and have never used Evals, this post is for you. A lead typed: "Got married in June, haven't figured out how we're filing yet." A model read that and confidently set filing_status to married_jointly. Helpful. Wrong. And invisible if you only eyeball a few outputs. So I built llm-evals, a Ruby gem for running real test cases against many models and ranking them by one weighted score that balances accuracy, hallucinations, errors, latency, and cost.

7
0
12
0
Read full article on X