Lab 14
Evaluation, Metrics and Data Analysis
Overview of the lab
This lab is built around case scenarios and plots to reason about. Every plot comes with questions: answer them yourself first, then open the solution. The plots use invented, illustrative data unless stated otherwise.
- The central question: task, metric, evaluator
- Classification metrics: confusion matrix, accuracy, precision, recall, F1
- Precision vs recall depends on the application
- Plot: accuracy can hide failure
- Evaluating generated text: BLEU and ROUGE-L
- Learned evaluators: COMET and LLM-as-a-judge
- When evaluators disagree
- Plot: training curves
- Averages hide subgroups; distribution shift
- Leakage, duplicates and contamination
- Uncertainty and causation
- Human evaluation
- Final case study: build an evaluation suite and what can you actually conclude?
1. The central question
A model produces a number: accuracy = 92%, BLEU = 34.7, ROUGE-L = 0.51, COMET = 0.78, LLM-judge win rate = 64%. The dangerous question is "which model has the highest score?". The useful one is "what does this score actually measure, and what conclusions does the experiment justify?"
- ❗ A metric is not the goal. A benchmark is not reality. An evaluator is not automatically correct.
Task, metric, evaluator
| What it is | Examples | |
|---|---|---|
| Task | the behaviour we want to measure | detect spam · translate English to Bulgarian · summarise an article · write useful code · follow instructions |
| Evaluator | the mechanism deciding whether an output is good | exact string match · a reference answer · unit tests · a human annotator · a learned metric · an LLM judge |
| Metric | a number that summarises the evaluator's decisions | accuracy · precision · recall · F1 · BLEU · ROUGE-L · COMET · win rate |
A single task can have many possible evaluators, and each evaluator can be summarised by several metrics.
1Identify the task, the evaluator and the metric. A: a spam classifier is evaluated on manually labelled emails; we report F1. B: a translation system is compared against reference translations using BLEU. C: two chatbot answers are shown to an LLM, which selects the better one; we report win rate. D: a coding model writes Python solutions, hidden unit tests execute the code; we report the percentage of problems solved.
| Task | Evaluator | Metric | |
|---|---|---|---|
| A | spam detection | human labels (the prediction is compared with them) | F1 |
| B | translation | n-gram comparison against reference translations | BLEU |
| C | helpful chat answers | an LLM judge | win rate |
| D | code generation | hidden unit tests | % of problems solved (pass rate) |
In A, B and D the evaluator is only as good as the labels, the references, or the tests. In C the evaluator is itself a model that can be wrong.
2. Classification metrics
For binary classification, every prediction falls into one of four groups: the confusion matrix.
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True Positive (TP) | False Negative (FN) |
| Actually negative | False Positive (FP) | True Negative (TN) |
- Precision: of the examples predicted positive, how many were actually positive? High precision: when the model says "positive", it is usually right.
- Recall: of all actual positives, how many did the model find? High recall: the model misses few positive cases.
- F1: the harmonic mean, so it is low if either precision or recall is low.
Try it: the confusion-matrix calculator
Starts with a disease-screening model: TP = 80, FP = 20, FN = 40, TN = 860. Accuracy is 94%, which sounds excellent, yet the model missed 40 of the 120 sick patients. Edit any cell, or load another case.
| Pred. + | Pred. − | |
|---|---|---|
| Actual + | ||
| Actual − |
2A fraud detector produces TP = 45, FP = 15, FN = 55, TN = 885. Compute accuracy, precision, recall and F1 by hand (then check with the calculator). Which metric would you care about most if missing a fraudulent transaction is very expensive? Don't answer only with a metric name: explain the trade-off.
Accuracy = 930 / 1000 = 0.93 · Precision = 45 / 60 = 0.75 · Recall = 45 / 100 = 0.45 · F1 = 2·0.75·0.45 / 1.2 = 0.5625.
The model misses 55 of 100 frauds, hidden behind 93% accuracy. If a missed fraud is expensive, recall matters most: we would lower the decision threshold to catch more fraud. The price is more false alarms (lower precision): more legitimate transactions blocked or sent for review, annoyed customers and analyst time. The right operating point depends on how much each error costs, not on the metric alone.
3Case: rare disease. 10,000 patients, 100 positive, 9,900 negative. The model predicts negative for every patient. Compute the accuracy and the recall for the positive class. Is accuracy useful here? Propose a better evaluation report.
Accuracy = 9,900 / 10,000 = 99%. Recall = 0 / 100 = 0. Precision is undefined (the model never says "positive"). The model is useless, yet its accuracy is 99%: accuracy measures mostly how rare the disease is.
A better report: the full confusion matrix; recall (sensitivity) and precision for the positive class, plus specificity; the baseline of always predicting the majority class; a precision–recall curve (or recall at a fixed false-alarm rate) rather than one threshold; confidence intervals (with only 100 positives, recall is uncertain); and subgroup results (age, sex, hospital).
3. Precision vs recall depends on the application
There is no universal rule saying precision > recall, or recall > precision. The cost of the different errors decides. The metric must follow the task, not the other way around.
- Spam filter: a false positive may hide an important email, so you may care strongly about precision.
- Disease screening: a false negative may miss a sick patient, so you may care strongly about recall.
- Search engine: depends on the product. Retrieving everything possible favours recall; showing only highly relevant results favours precision.
Try it: move the decision threshold
A classifier gives each case a score; everything to the right of the threshold is predicted positive. 100 positives (red) and 900 negatives (blue). Switch the view to "each class normalised" and notice how much better separated the classes look once you hide the imbalance.
negatives positives shaded: predicted positive
4Case scenarios. Using the widget, where would you put the threshold for (a) a spam filter for a company's inbox, (b) a first-round cancer screening test followed by a cheap second test, (c) a legal e-discovery search where lawyers must find every relevant document? What happens to accuracy as you move to each choice, and why is accuracy the wrong guide here?
(a) Spam: high threshold (≈ 0.7), precision ≈ 0.95. A few spam emails slip through; almost no real email is lost. (b) Screening followed by a second test: low threshold (≈ 0.4), recall ≈ 0.98; the false alarms are filtered by the second test. (c) E-discovery: also recall-oriented, since a missed document can lose a case, and humans review what is retrieved.
Accuracy barely changes near 0.6–0.7 (0.95 → 0.94) but drops to 0.67 at 0.4, even though 0.4 is the right choice for screening. With 90% negatives, accuracy is dominated by the negatives and rewards thresholds that simply predict "negative" more often.
4. Plot: accuracy can hide failure
Four models, evaluated on 1,000 cases. 100 of them belong to a minority class that represents a safety-critical event.
overall accuracy minority-class recall minority-class precision
5(1) Which model has the highest overall accuracy? (2) Which has the highest minority-class recall? (3) If the minority class is a safety-critical event, is accuracy alone a sufficient evaluation? (4) What additional information would you want before selecting a model? (5) Is it valid to conclude that Model C is "best"? Be careful with the last question.
(1) Model A (0.943), with B almost equal (0.940). (2) Model C (0.85), which has the lowest accuracy. (3) No: A has the best accuracy and misses 55 of the 100 critical events.
(4) Precision (how many alarms are false: turn it on, C's precision is 0.49, so half of its alarms are false), the cost of each error type, the confusion matrices, the test-set size and confidence intervals, how the threshold was chosen (all four could move along their PR curves), and subgroup results.
(5) No. The plot supports specific comparisons ("C finds the most critical events", "A makes the fewest errors overall"). It does not support a universal ranking. Which model is best depends on what a missed event costs relative to a false alarm, which the plot does not show. Try it below.
Try it: put a price on the errors
Total cost = FN × (cost of a missed event) + FP × 1. Slide the cost of a miss: each of the four models is the "best" one for some cost ratio.
5. Evaluating generated text: BLEU and ROUGE-L
Classification usually has one correct label. Language generation does not. For the reference The cat is sleeping on the sofa., the outputs The cat sleeps on the couch. and A cat is asleep on the sofa. may both be good translations; The sofa sleeps on the cat. is not. Exact-match accuracy is inappropriate: we need evaluators that tolerate variation.
BLEU
A classic reference-based metric for machine translation (Papineni et al., 2002): compare the n-grams of the generated text with the n-grams of one or more references. An n-gram is a sequence of \(n\) tokens: the cat sat down has the bigrams the cat · cat sat · sat down.
\(p_n\) = clipped n-gram precision: the fraction of the candidate's n-grams that also appear in the reference (each reference n-gram can be matched at most as many times as it occurs). \(c\), \(r\) = candidate and reference length. The brevity penalty stops a model from scoring well by outputting only a few very safe words.
💡 BLEU was designed to be computed over a whole test corpus. On a single sentence, one n-gram order with zero matches makes the score 0, so sentence-level scores are usually smoothed; the widget shows the common "BLEU+1" variant, which adds 1 to the matched and total counts for \(n \ge 2\).
ROUGE-L
ROUGE metrics (Lin, 2004) are widely used for summarisation. ROUGE-L is based on the longest common subsequence (LCS): tokens in the same order, but not necessarily adjacent. For the reference the small cat sat on the mat and the candidate the cat sat near the mat, the LCS is the cat sat the mat (5 tokens).
Common implementations report this F-measure (the original paper allows weighting recall more).
Try it: the overlap lab
Load the exercise sentences, or type your own reference and candidates (one per line). Click a row to see which words match: gold words form the LCS.
6BLEU intuition. Reference: the cat is sitting on the mat. A: the cat is sitting on the mat · B: the cat sits on the mat · C: mat the on sitting is cat the. Without the full formula: which candidate should get the highest BLEU? Why might B score lower although it is a perfectly reasonable sentence? Why might n-gram overlap fail to represent meaning?
A (identical, BLEU = 1). B changes one verb form and loses a word, which breaks every 3-gram and 4-gram around it: p₄ = 0/3, so unsmoothed BLEU is 0, BLEU+1 ≈ 0.41, and it is also shorter than the reference. C uses exactly the right words (p₁ = 1) in a meaningless order: only the higher-order n-grams catch that.
n-grams only check surface form: synonyms (sofa / couch), inflections (sits / is sitting) and valid reorderings are penalised, while a single changed word that flips the meaning barely is. Lexical similarity ≠ semantic equivalence.
7Longest common subsequence. Reference: machine learning models can overfit training data. A: machine learning models may overfit the training data · B: training data can overfit machine learning models. Which candidate has the stronger ordered overlap? Why does word order matter for ROUGE-L? Could a good summary get a mediocre ROUGE-L, and a poor summary a surprisingly good one? Give an example of the latter.
A: LCS = 6 (machine learning models overfit training data), F = 0.80. B uses almost the same words, but in an order that reverses the meaning; its LCS is only 3 (machine learning models), F = 0.43. Interesting: BLEU+1 prefers B (0.47 vs 0.37), because every one of B's words and several bigrams appear in the reference, and A is longer. Two overlap metrics, two different winners.
Yes to both. A good paraphrased summary (overfitting: the model memorises its training set) shares few words. A poor one can copy long ordered stretches of the reference while changing one crucial word: machine learning models cannot overfit training data scores F = 0.86 with the opposite meaning.
8Surface overlap vs meaning. Reference: The committee rejected the proposal. A: The proposal was rejected by the committee. B: The committee accepted the proposal. Which candidate should humans prefer? Which one gets the higher overlap scores in the lab? Why could a learned semantic metric help, and what new risks appear when the evaluator itself is a learned model?
Humans prefer A (same meaning, passive voice). Both overlap metrics prefer B: BLEU+1 0.45 vs 0.32 and ROUGE-L 0.80 vs 0.50, although B says the opposite. The paraphrase preset shows the same thing: the correct paraphrase scores ROUGE-L 0.20, the contradiction …contained no noise scores 0.90.
A learned metric compares meaning in a representation space, so it can recognise the passive paraphrase and (ideally) notice that accepted ≠ rejected. But it inherits the weaknesses of a model: biases from its training data, domain shift, adversarial inputs, poor calibration, and weaker performance on rare languages and domains (next section).
6. Learned evaluators: COMET and LLM-as-a-judge
COMET
COMET (Rei et al., 2020) is a learned metric for machine translation. Instead of counting n-grams, a neural model reads the source sentence, the candidate translation and the reference (depending on the variant) and predicts a quality score. It is trained to agree with human quality judgements.
❗ Learned metrics are not magic. Replacing a simple metric by a neural evaluator does not remove the evaluation problem; it moves part of it into the evaluator: training-data bias, domain shift, adversarial examples, poor calibration, spurious correlations, systematic mistakes in rare languages or domains.
LLM-as-a-judge
A language model receives the task prompt, one or two candidate answers and evaluation criteria, and returns A wins / B wins / tie, or scores such as correctness 4/5, relevance 5/5, clarity 3/5. It is cheap at scale compared with expert annotators, can evaluate open-ended text, can follow an explicit rubric, and speeds up iteration. But judges have known failure modes (several documented by Zheng et al., 2023):
Position bias
prefers the answer shown first (or second) because of presentation order
Verbosity bias
longer answers look more complete even when they are not more correct
Style bias
polished language beats plain but technically correct language
Self-preference
a model family may prefer outputs resembling its own style
Prompt sensitivity
small changes in the judge instructions change the result
Knowledge limits
it cannot verify specialised or recent claims
Shared failures
if generator and judge make the same mistake, the judge rewards it
Read the plot: possible judge bias
Illustrative, simulated data. Pairs of answers that human raters consider about equal in quality, grouped by how much longer Answer A is than Answer B. y-axis: how often the LLM judge (and, for comparison, humans) prefer A. Then split the judge's votes by which answer was shown first.
LLM judge LLM judge, A shown second human raters
9(1) What pattern do you observe? (2) What bias might it suggest? (3) Does the plot prove that length causes the judge's preference? (4) What controlled experiment would you run? (5) How could you change the protocol to reduce the bias? (6) After splitting by order: what second bias appears, and why did the pooled view hide it?
(1) The judge's preference for A rises steadily with A's extra length, from ≈ 0.15 to ≈ 0.8; the human curve stays flat near 0.5. (2) Verbosity bias.
(3) No. It is an association. Longer answers might contain real extra content that these human raters did not value (or noticed less), or length might correlate with style, formatting or the topic of the question. "Humans rate them equal" is itself a measurement with noise.
(4) Take the same answer and create a padded version that adds only neutral, content-free words (or a shortened one that removes only redundancy); ask the judge to compare them, in both orders, many times. If the padded version wins, length itself is moving the judge.
(5) A rubric that says length is not a merit and penalises padding; separate scores for correctness and completeness; length-controlled comparisons or reporting win rate by length bin; randomised order and scoring both orders; allowing ties; validating on a human-labelled sample.
(6) Position bias: at equal length the judge picks A ≈ 65% of the time when A is first and ≈ 38% when it is second. Because each pair was shown in both orders equally often, the two effects cancel in the average. If your evaluation always shows your model first, they would not cancel.
Building a better judge
- a clear rubric with separate scores per dimension, e.g. correctness 1–5, relevance 1–5, completeness 1–5, clarity 1–5, unsupported claims yes/no instead of "which answer is better?";
- randomised answer order, multiple judge prompts, multiple independent judges, explicit permission to choose a tie;
- hidden reference answers when appropriate; human validation on a representative sample and calibration against expert judgements.
10Design an LLM judge for answers to "Explain why validation data should not be used for final model reporting." It receives the question and one candidate answer, scores at least three dimensions with explicit criteria, separates factual errors from style, and returns a structured result. Then: how would you test whether your judge is reliable?
You are grading a student-facing answer to an ML question.
Question: {question}
Answer: {answer}
A correct answer must say: validation data was used to make choices
(hyperparameters, early stopping, model selection), so the model is
indirectly fitted to it and its validation score is optimistically
biased; the final report needs an untouched test set.
Score each dimension from 1 to 5 using ONLY these criteria:
- correctness: are all technical claims true? (ignore style)
- completeness: does it mention the selection bias AND the need
for a separate held-out test set?
- clarity: could a 2nd-year student follow it? (ignore length;
do not reward padding)
List every factual error separately. Do not let style affect correctness.
Return JSON only:
{"correctness": int, "completeness": int, "clarity": int,
"factual_errors": [str], "justification": str}
Reliability tests: have instructors grade 50–100 answers and measure agreement with the judge (correlation, κ); include planted answers with known errors and check they are caught; re-run with a different wording of the prompt, a different temperature or seed, and check stability; pad a good answer and check its score does not rise; test a fluent-but-wrong and a plain-but-right answer.
11Case: the 68% win rate. You compare two chatbots with the judge prompt "Which answer is more helpful? Choose A or B." Across 1,000 comparisons, A wins 68% and B 32%. Before concluding that A is better, list at least eight questions you would ask about the evaluation setup.
- Was chatbot A always shown in position A? Was the order randomised or both orders scored?
- Is there a rubric, or only "more helpful"? Helpful to whom?
- Were ties allowed? (Forced choice turns near-equal pairs into noise.)
- Which model is the judge? Is it from the same family as chatbot A (self-preference)?
- What temperature, and how stable is the result across seeds or re-runs?
- Are A's answers systematically longer or more formatted?
- Were factual claims verified, or could a confident wrong answer win?
- Where did the 1,000 prompts come from? Do they represent real use, or favour A's strengths? Any overlap with A's training data?
- Was the judge calibrated against humans on a sample? What is the agreement?
- What is the confidence interval, and how many distinct prompts are there (1,000 comparisons of 50 prompts is not 1,000 independent samples)?
7. When evaluators disagree
Five translation systems, scored by four evaluators (illustrative numbers). BLEU, ROUGE-L, COMET and human scores do not share a scale, so the plot shows ranks: the point is the ordering. The raw score is written in each circle. Pick a system to follow it.
12(1) Does every evaluator rank the systems in the same order? (2) Which systems change position? (3) Why could this happen? (4) If your paper reports only the metric that makes your system look best, what is the methodological problem? (5) What would a stronger evaluation section report?
(1) No. The two lexical metrics (BLEU, ROUGE-L) roughly agree with each other; COMET and humans agree with each other; the two pairs disagree. (2) System 2 goes from 1st (BLEU, ROUGE-L) to last (COMET, human); System 3 goes from 4th to 1st; System 4 and 5 swap around the middle. System 1 is stable (2nd–3rd).
(3) The evaluators measure different things. System 2 may copy phrasing that matches the references (high overlap) while making meaning errors humans notice; System 3 may paraphrase correctly with different words. In real data, learned metrics also have their own blind spots.
(4) It is a search over metrics until the conclusion looks good. With four evaluators, almost any system can be "first" somewhere, so the reported win carries little information (the same problem as testing many hypotheses and reporting the significant one).
(5) Metrics chosen from the task before seeing test results; all of them reported, including those where the system loses; human evaluation on a sample; confidence intervals or significance tests; a discussion of why metrics disagree, with examples.
13Case: summarisation. System A writes concise summaries with frequent paraphrasing. System B copies more phrases directly from the source and reference. ROUGE-L: A 0.41, B 0.48. Human factuality: A 4.7/5, B 4.1/5. Human usefulness: A 4.5/5, B 4.0/5. Which system does ROUGE-L favour? Which do humans favour? Is ROUGE-L "wrong"? What might it be measuring that differs from the human criteria? Which metrics should the final report include?
ROUGE-L favours B; humans favour A on both criteria. ROUGE-L is not "wrong": it correctly reports that B's wording overlaps more with the reference. It measures lexical, ordered overlap, which rewards copying and penalises paraphrase; it does not measure factuality or usefulness directly.
Report ROUGE-L (comparability with prior work) and human factuality and usefulness (or a validated factuality check), with the number of annotated summaries, inter-annotator agreement and examples of each system's typical errors. Human judgement should decide the headline claim.
14Case: translation. Three translations of the same sentence: A BLEU 32, COMET 0.79 · B BLEU 35, COMET 0.76 · C BLEU 30, COMET 0.84. Humans prefer C > A > B. Which metric tracks the human ordering more closely? Is one example enough to conclude it is universally better? What dataset would you need to compare the metrics properly? Why might evaluator quality differ by language pair or domain?
COMET orders them C > A > B exactly; BLEU gives the reverse (B > A > C). But one sentence is an anecdote.
To compare metrics you need many (thousands of) segments across several systems, language pairs and domains, each with human quality judgements; then measure how well each metric's scores correlate with the human scores (segment level and system level), with confidence intervals. This is how the WMT metrics shared tasks evaluate metrics.
A learned metric is only as good as its training data: for language pairs or domains (medical, legal, slang) that were rare in its training judgements, it may be poorly calibrated. BLEU depends on tokenisation and morphology: in a morphologically rich language like Bulgarian, one inflection change breaks many n-gram matches.
8. Plot: training curves
Training and validation loss over 40 epochs (illustrative). Move the stopping epoch, then compare the curves after two interventions. Compare with the bias–variance picture in Lab 06.
training loss validation loss
15In the baseline: (1) What happens to the training loss over time? (2) To the validation loss? (3) Around which epoch does the model begin to overfit? (4) Would training for more epochs necessarily improve deployment performance? (5) Which interventions could you try (early stopping, regularisation, more data, data augmentation, a simpler model, better validation design)? For each, explain why it could help, and check the two you can switch on.
(1) It falls all the way (to ≈ 0.08). (2) It falls until ≈ epoch 12, then rises steadily. (3) Around epoch 12, where the gap starts to grow. (4) No: after epoch 12 more training makes the validation (and likely deployment) loss worse. Training loss is not the goal.
(5) Early stopping: keep the weights from the best validation epoch. Regularisation (weight decay, dropout): limits how precisely the model can fit noise; in the plot the training loss stays higher, the gap is smaller and the minimum moves to ≈ epoch 21. More data and augmentation: harder to memorise, so the gap shrinks and the best validation loss is the lowest of the three. Simpler model: less capacity to memorise. Better validation design: make sure the validation set is not leaking or too small, so the curve you stop on is trustworthy.
❗ Early stopping uses the validation set to make a choice, so the validation loss at the chosen epoch is optimistic. The final number must come from an untouched test set.
9. Averages hide subgroups; distribution shift
Two models, F1 per subgroup and overall (illustrative). Group 3 is small. Move its share of the data.
Model X Model Y
16At 5%: (1) Which model has the higher overall F1? (2) What happens for Group 3? (3) If Group 3 is only 5% of the data, how can the overall metric hide the problem? (4) What additional statistics would you request? (5) Should subgroup metrics always be reported? When is subgroup analysis especially important?
(1) Model Y, 0.881 vs 0.838. (2) Y's F1 on Group 3 is 0.43, X's is 0.81: Y misses most of Group 3's positives (recall 0.35). (3) The overall score is pooled over all examples, so each group counts in proportion to its size; a failure on 5% of the data costs only a few points. Only when Group 3 reaches ≈ 17% of the data does X overtake Y.
(4) Per-group size, precision and recall, confusion matrices, confidence intervals (small groups have wide intervals), and worst-group performance.
(5) Report them whenever the groups are known and meaningful. They are especially important when errors fall on people (medical, hiring, credit, speech recognition across accents), when a group is safety-critical, when a group is small in the test set but large in deployment, and when a regulator or user will ask "does it work for us?".
Distribution shift
A model trained and tested on 70% news, 20% Wikipedia, 10% social media is deployed where traffic is 10% news, 10% Wikipedia, 80% social media. It can do well on an IID test set and still degrade in deployment. The central question: does the test set represent the environment where the model will actually be used?
Try it: the same model, a different mix
Per-domain accuracy is fixed (news 0.94, Wikipedia 0.92, social media 0.71). Set the deployment mix.
17Case: sentiment in the wild. A sentiment classifier achieves 92% F1 on a benchmark of movie reviews. A company deploys it on short social-media comments about mobile apps, and performance drops dramatically. Give at least four possible reasons. Then propose a better evaluation dataset.
- Vocabulary: crashes, laggy, battery drain, update never appear in movie reviews; plot, acting never appear in app comments.
- Length: long reviews carry many sentiment cues; a 6-word comment carries one or two.
- Sarcasm, emojis, slang, misspellings: great, another update that deletes my data 🙃.
- Class balance: benchmarks are often balanced; app comments may be mostly complaints, or mostly neutral.
- Label definitions: star ratings of a film vs "is the user angry at the app?"; neutral / mixed comments may not exist in the benchmark.
- Language variety and temporal drift: code-switching, non-native English, new features and new slang after training.
Better evaluation: sample real comments from the deployment platform (across apps, time periods and languages), label them with guidelines written for this use case, measure inter-annotator agreement, keep the natural class balance, report per-slice results (length, emoji / no emoji, app category), and refresh the set regularly to track drift.
10. Leakage, duplicates and contamination
Data leakage occurs when information that is not available at real prediction time enters training or evaluation. Example: predicting whether a patient will be readmitted to hospital, using the feature number of follow-up appointments scheduled after discharge. Some appointments are scheduled only because clinicians already suspect readmission risk, so the feature partially reveals the future. The test score can look excellent while deployment performance collapses.
18Spot the leakage. For each case, explain the possible leak, why the score could be inflated, and a safer split or procedure. A: randomly splitting multiple medical records from the same patient across train and test. B: using the article's publication date to predict whether it will receive many views after one year. C: using a feature computed from the entire dataset before cross-validation. D: training a language model on web data that may include benchmark questions later used for evaluation.
A: leak. The model can recognise the patient (age, rare conditions, lab values) rather than learn the disease; test patients are not new patients. Split by patient (group k-fold).
B: probably no leak. The publication date is known at prediction time. Watch for a subtler problem: with a random split, the model learns from articles published later than the test articles (e.g. which topics trended). Use a time-based split: train on older articles, test on newer ones. Also make sure no feature is measured after publication (e.g. views after one week).
C: leak. Scaling statistics, feature selection, TF-IDF vocabularies or target encodings computed on all data have "seen" the test folds. Fit every preprocessing step inside each fold, on training data only (a pipeline).
D: leak (contamination). The model may have memorised the questions or answers; the benchmark then measures recall of training data, not ability. Deduplicate the training data against benchmarks (n-gram overlap), report results on clean vs overlapping items, use held-out or freshly written test sets.
Read the plot: scores before and after near-duplicate removal
Illustrative. A benchmark is checked against the models' training data; 15% of its items have near-duplicates there and are removed.
full benchmark after deduplication on the removed items only
19(1) What happens to performance after deduplication? (2) Give at least two possible explanations. (3) Does the plot prove intentional benchmark contamination? (4) What additional evidence would be needed? (5) Why is deduplication particularly important for large language models trained on web-scale corpora?
(1) Every model drops, but by very different amounts: Model 1 by 3.9 points, Model 3 by 2.0, Models 2 and 4 by less than 0.5. Model 2 now ranks above Model 1.
(2) (a) Memorisation: models that saw the items during training reproduce their answers. (b) The removed items are easier: popular questions get copied around the web because they are standard, simple ones, so removing them lowers everyone's score a bit, even for models that never saw them. (c) Noise: on a small benchmark, differences of 0.5 points may be random.
(3) No. It shows the scores depend on the overlapping items; it says nothing about intent (web data contains benchmarks by accident), and it does not even prove memorisation. (4) Compare each model's score on the removed vs the kept items (show them: Model 1 scores 96% vs 70%, while Model 4 scores 59% vs 58%, so the items are only slightly easier); check whether the model can complete benchmark questions verbatim; check training-data documentation; test on freshly written, never-published items.
(5) Web-scale corpora are too big to inspect by hand and contain copies of popular benchmarks, their solutions and discussions. A model with billions of parameters can memorise rare text. Without deduplication, benchmark scores can silently measure memory rather than generalisation.
20Case: data cleaning. You receive 50,000 text examples. Initial inspection finds 4,000 exact duplicates, 2,000 near-duplicates, 800 empty or corrupted examples, an unknown language distribution, inconsistent labels, and multiple examples from the same users. Design a cleaning and splitting pipeline: in what order would you inspect, remove unusable examples, normalise fields, handle duplicates, investigate labels, create train/validation/test splits, prevent user-level leakage, and document the changes? Why does the order matter?
- Inspect: read random samples, look at length distributions, detect languages, count labels per user. Keep a copy of the raw data.
- Remove unusable examples (empty, corrupted, wrong encoding): 800.
- Normalise fields (Unicode, whitespace, casing where appropriate, label spellings) so that duplicate detection works.
- Deduplicate: exact duplicates, then near-duplicates (e.g. MinHash); where duplicates disagree on labels, that is a label-noise signal for step 5.
- Investigate labels: annotation guidelines, disagreement between duplicates, a relabelled sample with agreement statistics; fix or drop inconsistent ones.
- Split by user (all of a user's examples go into one split), stratified by label and language; decide whether the test set should mirror deployment.
- Document every step with counts (a data card), and freeze the test set.
Why order matters: normalising before deduplication finds more duplicates; deduplicating and grouping by user before splitting prevents copies of the same text or user from landing in both train and test (leakage); label investigation is easier after duplicates reveal conflicts; any statistic used for cleaning decisions should not be tuned on the test split.
11. Uncertainty and causation
A benchmark score is an estimate. Is Model B with 84.5% meaningfully better than Model A with 84.2%? You need the test-set size, the uncertainty, repeated runs (different seeds), a significance test, and a judgement of practical significance. A 0.3-point difference may be meaningful on a huge stable benchmark, or pure noise.
Try it: the same accuracies, more test examples
Model A is 82% accurate, Model B 84%. Bars show 95% confidence intervals (Wilson). Increase the test-set size.
21Case: small differences. Two models are evaluated on only 100 examples: A 82 correct, B 84 correct. (1) Is the 2-point difference convincing? (2) What additional experiment would you run? (3) Would your conclusion change with 1,000,000 independent examples? (4) What if the models were evaluated on different test examples?
(1) No. The 95% intervals are about ±7 points and overlap almost completely; p ≈ 0.7.
(2) A larger test set; several training seeds per model; and a paired test on the same examples, e.g. McNemar's test or a paired bootstrap, which uses only the examples where the two models disagree and is much more sensitive than comparing two independent proportions.
(3) Yes: intervals shrink to ±0.08 points, so a 2-point gap is statistically unambiguous. It may still not be practically important, and it still doesn't tell you it holds on a different distribution.
(4) Then the comparison is confounded by test difficulty: B may simply have received easier examples. Scores on different test sets are not comparable; evaluate both on the same set.
Correlation is not causation
Models with more parameters also tend to have higher benchmark scores. You cannot conclude that parameter count caused the improvement: those models usually also differ in training compute, training data, data quality, architecture, post-training and evaluation setup. A plot can reveal association; causal claims need a controlled experiment.
22Case: confounded experiment. A team compares Model A (7B parameters, trained on 1T tokens) with Model B (13B parameters, trained on 3T tokens). Model B performs better. The report concludes: "Larger models are better." What is wrong with this conclusion? Design a fairer experiment.
Two things changed at once: size (×1.9) and training data (×3), so compute also changed (≈ 5.6×). The improvement could come from either, or both. With only two models there is also no estimate of variance, and "better" on which benchmarks?
Fairer: vary one factor at a time. Train 7B and 13B on the same data, same number of tokens, same recipe; separately, train one size on 1T vs 3T tokens. Better still, a grid of sizes × data amounts (as in scaling-law studies), including a compute-matched comparison, several seeds, and the same evaluation suite for all.
12. Human evaluation
For many tasks humans remain an important evaluator of correctness, fluency, usefulness, clarity, safety, faithfulness and style. But humans are not perfect instruments: disagreement, fatigue, inconsistent standards, insufficient expertise, order effects, anchoring, cultural differences and unclear rubrics. Human evaluation also requires experimental design.
- Inter-annotator agreement: if annotators disagree constantly, the target may be subjective, poorly defined, difficult, or badly explained. Agreement statistics quantify reliability.
- ❗ High agreement does not guarantee correctness: a team can agree consistently on the wrong criterion.
\(p_o\) = observed agreement; \(p_e\) = agreement expected by chance from each annotator's label frequencies. \(\kappa = 1\): perfect agreement; \(\kappa = 0\): no better than chance.
Try it: 90% agreement, but how good is it?
Two annotators label 100 model outputs as acceptable or not. Compare the first two presets: same raw agreement, very different κ.
| B: acceptable | B: not | |
|---|---|---|
| A: acceptable | ||
| A: not |
13. Build an evaluation suite
Design an evaluation suite for an AI assistant that answers questions about university course material. It should be factually correct, relevant, clear, appropriately concise, faithful to the course material, and honest when information is missing. You have reference answers, course notes, students, instructors, an LLM judge and automatic text metrics.
23Your design must cover: (1) dataset design, (2) automatic evaluation, (3) the LLM-judge rubric, (4) human evaluation (who annotates what?), (5) subgroup analysis, (6) failure analysis, (7) data-leakage controls. One possible solution is below; compare it with yours.
1. Dataset. Real (anonymised) student questions from past forums and office hours, plus instructor-written questions covering every lab; include easy definitions, multi-step reasoning, questions whose answer is not in the notes (to test honesty), ambiguous questions, and questions phrased in Bulgarian and English. A few hundred, with reference answers and links to the supporting passage.
2. Automatic. Checks that are cheap and objective: is the cited passage really in the notes (faithfulness), numerical answers compared exactly, code answers run against tests, answer length, refusal detection on unanswerable questions. ROUGE-L/BLEU against references only as a rough signal, never the headline.
3. LLM judge. Per-dimension 1–5 scores (correctness vs the notes, relevance, clarity, conciseness), yes/no for "unsupported claim" and "admits missing information when appropriate", with the reference answer and the relevant course passage given to the judge; randomised order in pairwise mode; JSON output.
4. Humans. Instructors / TAs judge correctness on a random sample and on all disagreements between judge and automatic checks; students rate clarity and usefulness (they are the users); measure agreement (κ) and calibrate the judge against the instructors.
5. Subgroups. Per lab or topic, question type (definition / computation / code / conceptual), language, answerable vs unanswerable, difficulty.
6. Failure analysis. Read the worst-scored answers per subgroup, tag error types (wrong fact, hallucinated citation, too long, off-topic, over-refusal), count them, and track them across versions.
7. Leakage. Keep the test questions out of prompt engineering and any fine-tuning data; hold out a secret set that is only used for final reports; write new questions each semester; check for overlap with anything used during development.
14. Final challenge: what can you actually conclude?
For each claim: is the evidence sufficient? If not, what additional evidence is needed? Then rewrite the claim into the strongest statement the evidence does support.
A"Model X is better because it has higher accuracy." Evidence: X accuracy 94%, Y accuracy 91%.
Insufficient: test-set size and uncertainty, class balance and per-class errors (Section 2), the cost of errors, same test set? Supported: "On this test set, X's accuracy (94%) is 3 points higher than Y's (91%)."
B"Model X summarises better because it has higher ROUGE-L." Evidence: X 0.47, Y 0.43.
Insufficient: ROUGE-L measures overlap with references, not factuality or usefulness (Exercise 13). Needs human or validated factuality evaluation, and uncertainty. Supported: "X's summaries overlap more with the reference summaries (ROUGE-L 0.47 vs 0.43)."
C"Users prefer X." Evidence: an LLM judge prefers X in 63% of comparisons.
Insufficient: the judge is not the users (Exercise 11: order, length, self-preference, calibration). Needs a user study or human preferences on representative prompts. Supported: "Under this judge prompt and setup, the LLM judge preferred X in 63% of comparisons."
D"Increasing training time improves generalisation." Evidence: training loss decreases for 30 epochs.
Insufficient: training loss says nothing about generalisation; in Section 8 it keeps falling while validation loss rises. Needs validation / test loss per epoch. Supported: "The model continues to fit the training data better for 30 epochs."
E"The model is robust." Evidence: it scores 90% on one IID benchmark.
Insufficient: robustness means performance under shift, noise, adversarial or rare inputs, and across subgroups (Section 9); an IID test says nothing about those. Supported: "The model scores 90% on test data drawn from the same distribution as its training data."
Metric selection cheat sheet
| Situation | Useful starting point | Main warning |
|---|---|---|
| Balanced classification | accuracy, F1 | inspect class-level errors |
| Imbalanced classification | precision, recall, F1, precision–recall curve | accuracy can mislead |
| Machine translation | BLEU, COMET, human evaluation | lexical overlap is not meaning |
| Summarisation | ROUGE-L + factuality / usefulness evaluation | overlap does not guarantee faithfulness |
| Open-ended assistant responses | human or LLM judge with a rubric | judge bias and evaluator reliability |
| Code generation | unit tests / verifiable execution | incomplete tests can be exploited (Lab 13) |
| Safety-critical tasks | task-specific error costs, subgroup analysis | aggregate scores may hide dangerous failures |
A starting point, not a universal recipe.
Summary: key takeaways
- Metric vs objective: the metric is a measurement proxy; the objective is what you actually care about.
- Accuracy vs usefulness: with imbalanced classes, look at precision and recall, and at what each error costs.
- Reference overlap vs meaning: BLEU and ROUGE-L measure forms of overlap; meaning needs learned or human evaluators.
- Learned evaluator vs truth: COMET and LLM judges capture richer behaviour but bring their own errors and biases.
- Aggregate vs subgroup: averages can hide systematic failures on small groups.
- Benchmark vs deployment: a model can do well on a test set and fail under distribution shift, leakage or contamination.
- Estimate vs fact: small differences on small test sets may be noise.
- Correlation vs causation: observed association does not prove what caused the result.
💡 If one model has a higher score, what must you know before concluding that it is better for the task you care about? What the task is, what the evaluator is, what the metric is, what the test distribution is, what errors are hidden by the average, and what conclusion the evidence actually supports.
One sentence: a trustworthy evaluation does not ask only whether a model scores higher; it asks whether the evaluator measures the right behaviour, whether the data represents the real task, whether important failures are hidden by averages, and whether the evidence actually supports the conclusion being made.
Further reading
- BLEU: Papineni et al. (2002), BLEU: a Method for Automatic Evaluation of Machine Translation
- ROUGE: Lin (2004), ROUGE: A Package for Automatic Evaluation of Summaries
- COMET: Rei et al. (2020), COMET: A Neural Framework for MT Evaluation
- LLM-as-a-judge: Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Significance testing: Koehn (2004), Statistical Significance Tests for Machine Translation Evaluation (paired bootstrap)
25 auto-graded, data-heavy tasks, 100 points: confusion matrices, thresholds and costs, imbalance, ROUGE-L and disagreeing evaluators, judge biases, training curves, subgroups and shift, leakage, uncertainty, confounds and Cohen’s κ.