Evaluating AI Systems
Metrics, Test Sets and Judgement for Models That Matter
- Pages
- 81
- Chapters
- 18
- Level
- Intermediate–Advanced
- Format
- E-book
About this book
'It looked good in the demo' is not an evaluation. Evaluating AI Systems shows how to measure whether a model or AI feature truly works for its users, and how to keep measuring once it is live.
The book covers classic metrics (accuracy, precision, recall, F1, calibration, error analysis) with worked examples; how to build test sets that reflect real use; methods for evaluating generated text, including rubrics, pairwise comparison and human review; when and how to use a language model as a judge; online evaluation with A/B tests and monitoring; and red teaming for safety and robustness.
It ends with an evaluation plan template that you can adapt for any AI project, from a classifier to a chat assistant.
What you will learn
- Choosing metrics that match real costs and benefits
- Precision, recall, F1, calibration and error analysis with examples
- Building representative, versioned test sets
- Evaluating generative output with rubrics and pairwise comparison
- Using LLM judges reliably, and checking the judges
- A/B testing, monitoring and red teaming in production
- Confidence intervals, paired comparisons, the bootstrap and sample sizes, with numbers
- How to evaluate RAG systems, agents, fairness, speech and vision
- How to run human evaluation, build an evaluation pipeline and decide by expected cost
Table of contents
- IntroductionIntroduction
- Chapter 1Why Evaluation Is the Real Work
- Chapter 2Classic Metrics, Worked Through
- Chapter 3Statistics for Evaluators
- Chapter 4Building a Test Set
- Chapter 5Evaluating Generated Output
- Chapter 6Language Models as Judges
- Chapter 7Evaluating Retrieval and RAG Systems
- Chapter 8Evaluating Agents and Multi-Step Systems
- Chapter 9Online Evaluation and Monitoring
- Chapter 10Fairness Evaluation
- Chapter 11Red Teaming and Robustness
- Chapter 12Speech, Vision and Multimodal Systems
- Chapter 13Reading Benchmarks and Leaderboards Critically
- Chapter 14Case Study: Evaluating a Support Assistant
- Chapter 15Running Human Evaluation Well
- Chapter 16Building an Evaluation Pipeline
- Chapter 17From Metrics to Decisions: Expected Cost
- Chapter 18An Evaluation Plan Template
- AppendixExercises
- GlossaryGlossary
Who this book is for
Data scientists, ML and AI engineers, product managers and QA professionals responsible for AI quality. Basic familiarity with machine learning is assumed.
Details
| Format | PDF + web edition |
|---|---|
| Length | 81 pages (approx.) |
| Chapters | 18 |
| Level | Intermediate to Advanced |
| Language | English |
| Topic | AI Engineering |
| Author | Latent Library Editorial |
Keep reading