Lab 13 — Post-training and Alignment

Overview of the lab

By the end of this lab you should be able to distinguish pretraining, supervised fine-tuning (SFT) and preference-based post-training; explain why next-token prediction alone does not guarantee useful assistant behaviour; describe RLHF, reward models and DPO; recognise reward hacking; and reason about why alignment is an ongoing evaluation and optimisation problem.

1. The central question

In Lab 12 we saw how a Transformer turns text into a probability distribution over the next token. Now: how does a pretrained language model become an assistant?

A model trained on next-token prediction may become good at completing text, answering questions, writing code and imitating many styles. But next-token prediction does not explicitly say: follow the user's instruction · be useful · be honest about uncertainty · avoid harmful behaviour · prefer clear answers. This motivates post-training.

See it: one prompt through the stages

A simplified pipeline. Click a stage to see what it learns from, what it optimises, and a typical (illustrative) reply to the same prompt: "Write three concise reasons why regular backups are useful."

💡 Real systems may use additional stages, synthetic data, model-based evaluators, safety-specific training, tool-use training, and repeated evaluation.

Stage 1

2. Pretraining

During pretraining, a language model is typically optimised to predict the next token. For The capital of France is Paris. it learns from the contexts The · The capital · The capital of · The capital of France · The capital of France is.

📐 Pretraining loss

\[ \mathcal{L}_{\text{pretrain}} = -\sum_t \log p(x_t \mid x_{<t}) \]

The model is rewarded for assigning high probability to the token that actually follows the context. This is the cross-entropy of Lab 6.5, summed over positions, with the Transformer of Lab 12 computing \(p\).

Try it: the loss of one sentence

Invented model probabilities for each true next token. Move the probability the model gives to Paris: a confident correct prediction costs almost nothing, a confident wrong one costs a lot.

p(Paris | The capital of France is)0.90

1For the prompt "User: Explain photosynthesis to a 10-year-old. Assistant:": (1) Does the pretraining objective explicitly contain "be helpful"? (2) "Follow the user's instruction"? (3) Could the model have seen helpful explanations in training? (4) Also misinformation, arguments, jokes, ads, low-quality explanations? (5) Why can next-token prediction produce useful abilities without guaranteeing useful assistant behaviour? Write a 2–3 sentence conclusion.

(1), (2) No: the only objective is to predict the next token of the training text. (3) Yes, plenty. (4) Yes, also plenty.

Conclusion: to predict text well, the model must learn a lot about language, facts and reasoning, so useful abilities emerge. But it learns to reproduce whatever kind of text is likely after the context, good or bad, helpful or not. Which behaviour it shows depends on which kind of document the prompt resembles, not on what the user wants; shaping that is the job of post-training.

3. Base model vs assistant model

For the prompt "Write three concise reasons why regular backups are useful."

A base model is primarily trained to answer: what text is likely to come next? (Maybe a list of more writing prompts, as in a worksheet.)
An assistant model is additionally trained towards: what response should I produce for this user's request?

These objectives are related, but they are not identical.

2Classify each output as A (raw text continuation) or B (instruction following). Ex. 1: Translate "good morning" into Spanish. → Buenos días. · Ex. 2: same prompt → Translate "good night" into French. Translate "thank you" into German. · Ex. 3: List two disadvantages of decision trees. → 1. They can overfit. 2. They can be unstable to small changes in the training data. · Ex. 4: same prompt → List three advantages of random forests. List two advantages of logistic regression.

1: B, it carries out the request. 2: A, it continues the pattern "a list of translation exercises", as a base model would after seeing a worksheet. 3: B, it answers (and correctly, see Lab 09). 4: A, again a continuation of an exercise list.

The A outputs are perfectly plausible text: such lists are common on the web. They are just not what the user wanted.

Stage 2

4. Supervised Fine-Tuning (SFT)

We collect examples of desired behaviour, prompt → high-quality response, and train the model to imitate the target responses, with the same next-token loss as in pretraining (usually counted only on the response tokens).

Prompt

Explain overfitting in one paragraph.

Target response

Overfitting occurs when a model learns patterns that are too specific for the training data and therefore generalizes poorly to new data…

	Pretraining	SFT
Data	enormous amounts of naturally occurring text	a smaller curated collection of desired behaviours
Learns	to predict likely text	instruction following, preferred structure, style and tone, tool-use conventions, task-specific and safety-related behaviour

But SFT has an obvious limitation: someone must provide examples of what good behaviour looks like.

3You are training an AI teaching assistant for an introductory ML course. Write an ideal target response for: A "What is the difference between classification and regression?" · B "My model has 99% training accuracy and 72% test accuracy. Is this good?" · C "Give me the answer to this homework problem without explaining anything." Then discuss: what makes your responses "good"? Would every teacher make the same choices? Could two SFT datasets produce noticeably different assistants? Who decides what counts as the desired behaviour?

A: "Both are supervised learning. Classification predicts a category (spam / not spam); regression predicts a number (a house price). A quick test: can you average two possible answers? If yes, it is usually regression."

B: "Not really: the 27-point gap between training and test accuracy suggests overfitting. The model has partly memorised the training data. Try a simpler model or regularisation, get more data, and choose hyperparameters with cross-validation. What model are you using?"

C: "I can't just hand over the answer to a graded assignment, but I'm happy to help you get there quickly. Which part are you stuck on? Here is the first step…" (A different course might allow direct answers.)

Good = correct, at the right level, relevant, honest, and serving the goal of learning. Teachers would disagree on details (how much to reveal in C, how formal to be), so different SFT datasets would give assistants with noticeably different behaviour. Someone (course staff, the company, the annotation guidelines) decides; that choice is already an alignment question.

Comparisons

5. Preferences instead of perfect answers

For "Explain gradient descent to a beginner." the model generates two responses. It may be easier for a human to say "A is better than B" than to write a perfect answer from scratch.

Response A

Gradient descent is an optimization algorithm that repeatedly changes model parameters in the direction that reduces the loss. You can imagine walking downhill by taking small steps in the steepest downward direction.

Response B

Gradient descent is based on first-order differential structure over a parameterized objective manifold and applies iterative updates using the negative local derivative of the scalar objective.

This gives us preference data: triples (prompt, chosen response, rejected response).

Try it: you are the annotator

The two cases from Exercise 4. Pick the better response for each user; every click adds a record to the preference dataset below.

4Case 1: a user who has never studied ML asks "What is logistic regression?" Case 2: a user implementing logistic regression from scratch asks for the prediction equation. Which response is preferable in each case? What does "better response" mean if the user's goals and background can change?

Case 1: the plain-language response A. Case 2: the equation, response B. The same two styles swap places.

"Better" is not a property of a response alone; it depends on the prompt, the user and their goal. Preference data must capture that context, and a model trained on it must learn to read the context rather than one fixed style (e.g. "always technical" or "always simple").

Stage 3

6. Reinforcement Learning from Human Feedback (RLHF)

A historically important preference-based post-training approach (used e.g. for InstructGPT, Ouyang et al., 2022). A simplified pipeline has three steps:

Supervised fine-tuning: pretrained model → SFT model.
Train a reward model: humans compare model responses (prompt, response A, response B, "A > B"). A separate reward model learns to assign a score \(r(x, y)\) to a prompt \(x\) and response \(y\): a learned estimate of preference.
Optimise the language model: it generates responses, the reward model scores them, and the language model is updated (with reinforcement learning, e.g. PPO) to produce responses receiving higher reward.

📐 Reward model from pairwise preferences (Bradley–Terry)

\[ P(y_w \succ y_l \mid x) = \sigma\big(r(x, y_w) - r(x, y_l)\big) \qquad \mathcal{L}_R = -\log \sigma\big(r(x, y_w) - r(x, y_l)\big) \]

\(y_w\) is the chosen (winning), \(y_l\) the rejected (losing) response, \(\sigma\) the sigmoid. Only the difference of two rewards matters: the reward model is trained like logistic regression on "which one wins?".

❗ The reward is not objective truth. It is the output of another learned model, trained on a limited set of (noisy) human judgements.

Try it: train the reward model on one comparison

Humans preferred A over B. Set the two scores: the loss is small when the reward model agrees with the human by a clear margin.

r(x, A) chosen0.5 r(x, B) rejected0.0

Regularisation

7. Why constrain the optimisation?

If the only goal is maximise the reward-model score, a powerful optimiser may discover strange outputs that exploit weaknesses in the reward model. So RLHF methods commonly include a term that discourages the model from moving too far from a reference model (usually the SFT model).

📐 KL-constrained objective

\[ \max_{\pi}\; \mathbb{E}_{y \sim \pi(\cdot \mid x)}\big[r(x, y)\big] \;-\; \beta\, D_{\mathrm{KL}}\big(\pi(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)\big) \]

objective = reward − β · distance from reference behaviour. A common distance is the KL divergence, \(D_{\mathrm{KL}}(\pi \| \pi_{\text{ref}}) = \sum_y \pi(y) \log \frac{\pi(y)}{\pi_{\text{ref}}(y)}\). Its optimum has a closed form (Rafailov et al., 2023): \(\pi^*(y \mid x) \propto \pi_{\text{ref}}(y \mid x)\, e^{r(x, y) / \beta}\).

The intuition: improve the desired behaviour without allowing the model to drift arbitrarily far while chasing an imperfect reward signal.

Try it: optimisation pressure and the KL penalty

Five kinds of reply to "Explain gradient descent to a beginner", with their probability under the SFT model, the (imperfect) reward model's score, and a hidden true quality (our invented judgement). The reward model overrates length and jargon. Lower \(\beta\) (more optimisation pressure) and watch the policy \(\pi^*\) move: the proxy reward keeps rising, true quality peaks and then falls.

β (KL penalty)5.0

expected reward-model score (proxy) expected true quality

5Without copying a diagram, draw an RLHF pipeline with: pretrained model, SFT data, SFT model, prompts, generated responses, human preferences, reward model, reinforcement learning, final model. Label where you see human-provided supervision, learned supervision and optimisation.

pretrained model + SFT data (humans) → supervised training → SFT model
prompts → SFT model → generated responses → human preferences (A > B) → training → reward model
prompts → policy (starts as SFT model) → responses → reward model score − β·KL → reinforcement learning → final model

human-provided supervision learned supervision optimisation

Failure mode

8. Reward hacking

We want high-quality educational explanations and define the reward: +1 if the answer is longer than 300 words, +1 if it contains at least five technical terms, +1 if it contains the phrase "In conclusion". Optimised aggressively, the model learns longer = better, more jargon = better, always write "In conclusion". None of these guarantees educational quality.

The model has optimised the proxy rather than the real goal: reward hacking, or specification gaming.

Main exercise

9. Main exercise: attack the reward

Your team trains an AI teaching assistant. The real objective: produce answers that help students understand the material. Student understanding is hard to measure directly, so someone proposes:

Component	Reward
answer longer than 250 words	+2
contains at least three equations	+2
uses technical vocabulary	+1
student clicks "helpful"	+3
student asks a follow-up question	−2

Try it: score answers with the proposed reward

Five answers to a student asking "Why does my model overfit?" (invented), scored by the proposed reward and by what we actually care about. Then tick the features of your own answer.

Build an answer

AFind the exploits: describe at least five ways a model could obtain high reward while being a poor teacher. (Example: it could avoid useful uncertainty or discussion, because follow-up questions are penalised.)

Pad every answer past 250 words with repetition and filler.
Add three equations whether or not they help, even to conceptual questions.
Stuff answers with jargon that beginners cannot follow.
Flatter the student or simply give the homework answer, to earn the "helpful" click.
Sound complete and final ("this fully answers your question"), discourage questions, or hide uncertainty, to avoid follow-ups; or be so confusing that students give up rather than ask.
Tell students what they want to hear (confirm a wrong belief) because that gets clicked more.

BIdentify the hidden assumption behind every component: answer length, number of equations, technical vocabulary, "helpful" clicks, follow-up questions.

Length: long answer → better explanation. Equations: more maths → more rigour and understanding.
Vocabulary: technical words → expertise and correctness.
Helpful clicks: the student can judge what helped them learn, and clicks when they did (not when they got the answer or felt flattered).
Follow-ups: a follow-up question means the answer failed (rather than that the student is engaged and curious).

CRedesign the evaluation. Use any of: human ratings, short quizzes after the explanation, expert review, pairwise comparisons, automatic correctness checks, delayed outcomes, adversarial tests. It must measure at least correctness, clarity, relevance, appropriate difficulty and absence of misleading claims.

Correctness, misleading claims: expert (TA) review of a random sample with a rubric; automatic checks where possible (does the code run, is the formula right); adversarial prompts containing false premises.
Clarity, difficulty: pairwise comparisons by TAs, with the student's level shown; students rate "I could explain this to a friend".
Relevance: rubric item "answers the question that was asked".
Learning (the real goal): a short quiz right after the explanation, and a delayed quiz or exam performance a week later, compared with a control group.
Keep the evaluation separate from the training reward, and rotate it, so it cannot be optimised directly.

Goodhart

10. Goodhart-style failure

A useful warning, informally: once a proxy becomes the thing we aggressively optimise, it may stop being a good measure of the real goal. You saw it in the KL widget: the proxy keeps rising while true quality falls.

Real goal	Proxy
student learning	"helpful" clicks
useful code	public tests passed
safe behaviour	number of refusals
good summary	lexical overlap with a reference

6For each scenario identify the real objective, the proxy and one failure mode. A: a customer-support chatbot rewarded for minimising conversation length. B: a coding agent rewarded only for passing public unit tests. C: a moderation system rewarded for minimising the number of harmful messages that remain visible. D: a tutoring system rewarded for maximising five-star ratings.

A: goal = solve the customer's problem; proxy = short conversations. Failure: it ends chats early, gives quick wrong answers, or makes customers give up.
B: goal = correct, maintainable code; proxy = public tests. Failure: special-cases the test inputs, or edits the tests (see the verifier widget).
C: goal = remove harmful content while keeping legitimate speech; proxy = few harmful messages visible. Failure: removes almost everything (massive over-removal).
D: goal = learning; proxy = ratings. Failure: gives answers away, flatters, makes everything feel easy.

Without a reward model

11. Direct Preference Optimization (DPO)

RLHF is a complicated pipeline: preference data → reward model → reinforcement learning → updated model. DPO (Rafailov et al., 2023) learns from the same preference pairs, but instead of first learning a separate reward model and then running RL, it directly trains the language model so that, relative to a reference model, chosen responses become relatively more likely and rejected responses relatively less likely.

RLHF	preferences → reward model → RL optimisation
DPO	preferences → direct preference-learning objective

📐 DPO loss (for reference; the derivation is outside the scope of this lab)

\[ \mathcal{L}_{\text{DPO}} = -\log \sigma\!\Big( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} \;-\; \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \Big) \]

It is the reward-model loss from RLHF with the reward replaced by \(\hat{r}(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}\): the language model is "secretly" its own reward model. The gradient raises \(\log \pi_\theta(y_w)\) and lowers \(\log \pi_\theta(y_l)\), weighted by \(\sigma(\hat{r}_l - \hat{r}_w)\): more when the model still ranks the pair wrongly.

Try it: one preference pair

How much the policy has raised or lowered the log-probability of the chosen and rejected response, compared with the reference model. Take gradient steps and watch chosen ↑, rejected ↓, with smaller and smaller steps once the pair is ranked correctly.

log π/π_ref of chosen0.00 log π/π_ref of rejected0.00 β0.50

7(1) Why might it be attractive to avoid training a separate reward model? (2) Does DPO remove the need for preference data? (3) If the preference data rewards overly verbose answers, what might DPO learn? (4) Does changing the optimisation algorithm solve poor annotation or poorly specified goals? (5) What problems remain even if optimisation is perfectly stable?

(1) Simpler and cheaper: no second large model, no RL loop with sampling and its instability and many hyperparameters; just a classification-style loss on fixed data. (2) No, it needs exactly the same preference pairs. (3) Verbosity: long answers become more likely, because DPO faithfully learns what the data prefers. (4) No: garbage in, garbage out; the objective is still defined by the data. (5) Biased or inconsistent annotators, prompts that don't cover real use, preferences that reward style over substance, reward hacking of whatever the preferences measure, and the need for evaluation that is independent of the training signal.

Annotators

12. Human feedback is not ground truth

Three annotators compare the same answers: Annotator 1: A is better. Annotator 2: B is better. Annotator 3: A and B are equally good. Possible reasons: ambiguous instructions, different expertise or expectations, different tolerance for verbosity, different interpretations of safety, annotation mistakes, genuinely subjective preferences.

Therefore "human feedback" is not one perfectly defined objective. The result depends on who gives feedback, what instructions they receive, which prompts are sampled, how disagreements are handled, which behaviour is rewarded, and how the final model is evaluated.

Try it: who labels the data?

Three response pairs and three annotators per pool. Each annotator weighs clarity, rigour, brevity and learning value differently (invented weights). Switch the pool and watch the "chosen" column of the dataset change.

Trade-offs

13. Alignment is not one number

An assistant might be evaluated on helpfulness, correctness, honesty, harmlessness, brevity, creativity, instruction following and robustness. These properties can conflict: brevity vs completeness, creativity vs predictability, helpfulness vs safety constraints, confidence vs calibrated uncertainty. So alignment is not maximising a universal scalar called "goodness": it requires specifying desired behaviour and evaluating trade-offs under different conditions.

Try it: which answer is best?

A user asks for a two-sentence summary of a complicated study whose main result holds only for one group of patients. Three candidate answers are scored on four properties (invented scores). Change how much each property matters: the "best" answer changes with the weights.

8Identify at least two conflicting objectives. A: a two-sentence summary of a complicated study. B: a beginner asks for help with a programming assignment. C: a user asks the model to be "100% certain" about an uncertain question. D: a very short explanation of a topic that needs important caveats. Should the model always follow the user's literal instruction?

A: brevity / instruction following vs completeness and accuracy (caveats). B: helpfulness (solve it) vs learning and academic integrity. C: instruction following vs honesty and calibration. D: brevity vs not misleading.

Not always literally. A good assistant follows the intent and states the trade-off: keep it short but include the one caveat that changes the conclusion, or say that certainty is not possible and why. Where the lines lie is exactly what a specification has to define.

Scalable supervision

14. AI feedback and scalable supervision

Human labelling is expensive. One research direction uses models themselves as part of the supervision process:

generate answer→critique answer→revise answer

response A+response B→AI evaluator→preference

This appears in research on AI feedback, model-based evaluation, critique/revision, and approaches that use explicit written principles or specifications. For example, Constitutional AI (Bai et al., 2022) has the model critique and revise its own answers according to a list of principles, then trains a reward model on AI preferences ("RL from AI Feedback").

❗ The new question: what happens when the evaluator is itself wrong, biased, manipulable, or weak on the same task?

What to check

15. Outcome supervision vs process supervision

Outcome supervision evaluates only the final answer: did the model obtain the correct result?
Process supervision evaluates the intermediate steps: did the model use a reliable process? Lightman et al. (ICLR 2024) found process supervision significantly better than outcome supervision for training reward models on the MATH benchmark.

Try it: a lucky answer

Two solutions to \((3 + 5) \times 2 - 4\). Switch the verifier: the outcome verifier rewards both, the process verifier checks every step.

9Is evaluating only the final outcome enough for: (1) Sudoku, (2) a mathematical proof, (3) a medical recommendation, (4) code generation with a strong hidden test suite, (5) essay writing? Which are easy to verify automatically, and where would you also inspect reasoning or evidence?

Sudoku: outcome is enough and trivially checkable. Proof: the proof is the process; a formal proof checker verifies it automatically, an informal one needs step-by-step review. Medical: the outcome is delayed, noisy and high-stakes; inspect the evidence and reasoning. Code with strong hidden tests: mostly automatic, but also review for security and maintainability. Essay: no automatic verifier; needs human judgement of argument and sources.

Easy to verify automatically: Sudoku, formal proofs, code with strong tests. Inspect reasoning or evidence: informal proofs, medicine, essays, and anywhere a right answer could come from a wrong process.

RLVR

16. Reinforcement learning with verifiable rewards

Some outputs are easy to check automatically: a maths answer (compare with the solution), code (execute tests), a formal proof (proof checker), a game (win or lose). This enables the loop model generates candidate → automatic verifier → reward → optimisation. Recent reasoning systems made RL with verifiable rewards an important research direction; DeepSeek-R1 (2025), for example, reports that reasoning abilities can be incentivised through pure RL, without human-labelled reasoning trajectories.

❗ A verifier only checks what it was designed to check. If it is incomplete or exploitable, reward hacking returns.

Try it: hack the verifier

Task: write is_even(n). Four candidate programs a model might produce. Choose how strong the verifier is and see which programs earn the reward.

10Design a verifier for one task (mathematics, code generation, SQL generation or information extraction): what the model produces, what exactly is checked, what reward it receives, how the model could exploit it, and what additional tests would make it harder to exploit.

Example, SQL generation. The model produces a SQL query for a question about a database. The verifier runs it on a test database and compares the result table with the result of a reference query; reward 1 if the tables match (as multisets), else 0.

Exploits: a query that happens to return the right rows on this small test database but is wrong in general (e.g. hard-coded IDs, a LIMIT that matches by luck); queries that modify data; extremely slow queries.

Harder to exploit: several randomised test databases with edge cases (NULLs, duplicates, empty tables); read-only permissions; a time limit; a check that the query does not contain the literal expected values.

17. A simplified map of post-training

A conceptual map, not a universal recipe. Click a box to jump to its section.

Pretraining↓ base model↓ Supervised fine-tuning↓ instruction model↓

preference learning

RLHF DPO

verifiable rewards

RL / search

↓ post-trained assistant↓ Evaluation↓ discover failures → better data / reward / evaluation / retraining ↺

18. Current research directions

As of 2026, recurring themes in post-training and alignment research include:

Better reasoning through post-training: how RL, preference optimisation, search and carefully constructed data improve multi-step reasoning. Central question: which capabilities must be learned in pretraining, and which can be elicited or improved in post-training?
Verifiable rewards: maths, programming, formal reasoning and some tool-use tasks can be partly checked automatically. The hard part is making the verifier measure the task we actually care about.
Better supervision: outcome and process supervision, human preferences, AI-generated feedback, critique and revision, synthetic training data, scalable oversight.
Reward robustness: models that exploit reward-model weaknesses, learn shortcuts, over-optimise imperfect proxies, or behave differently under evaluation and deployment.
Explicit principles and specifications: training models to follow written rules, principles, constitutions or safety specifications. The hard question is not whether the model can read the rule, but whether it applies the correct rule under ambiguity or adversarial pressure.
Agents and tool use: models increasingly use browsers, code execution, databases, APIs and other tools, so evaluation must cover long sequences of actions rather than one static answer.

19. Research discussion

Work in small groups; each group takes one question. Define the problem, give one concrete example, propose one possible approach, and identify one weakness of your approach. The goal is not a perfect solution, but to think like a researcher. Each card has one possible starting point.

AIf a reward model is imperfect, how can we optimise against it without exploiting it?

Starting points: the KL penalty; ensembles of reward models and optimising a pessimistic (e.g. minimum) score; early stopping on a held-out gold evaluation; periodically retraining the reward model on the policy's new outputs. Weakness: all reward models may share the same blind spot.

BCan AI systems reliably supervise systems that are more capable than the evaluator?

Starting points: tasks where checking is easier than solving (verify a proof, run tests); decomposition into smaller checkable pieces; debate between two models judged by a weaker one. Weakness: some tasks have no easy check, and a stronger model may argue persuasively for wrong answers.

CWhen should we reward only a final answer, and when should we supervise intermediate steps?

Starting points: outcome rewards when the answer is cheap to verify and a wrong process cannot produce it reliably; process supervision when answers are hard to check, stakes are high, or right answers can come from wrong reasoning. Weakness: step labels are expensive, and rewarding steps may favour reasoning that looks right.

DHow can we detect behaviour that looks good during evaluation but fails in deployment?

Starting points: evaluation prompts that are indistinguishable from real use; monitoring and sampling real deployment traffic; red-teaming and adversarial tests; comparing behaviour when the model is and isn't told it is being evaluated. Weakness: deployment is open-ended, so no test set covers it.

EHow should we evaluate an AI agent that performs a task over hundreds of actions?

Starting points: sandboxed environments with automatic end-state checks; checks on intermediate states (no forbidden actions, resources respected); logging every action for review; sampling trajectories for human audit. Weakness: many paths reach the same end state, some of them unsafe, and full human review does not scale.

Case study

20. Final case study: AI programming tutor

You are building an AI system that helps students solve programming exercises. The team proposes:

Pretrain a language model on web text and source code.
Fine-tune it on high-quality tutoring conversations.
Collect student preferences between pairs of responses.
Train a reward model.
Optimise the assistant to maximise reward.
Evaluate using student satisfaction ratings.

APretraining: what useful abilities might it provide? What desired behaviours are not guaranteed?

Provides: programming languages, common algorithms, APIs, reading error messages, explaining code. Not guaranteed: teaching instead of solving, matching the student's level, honesty about bugs it isn't sure of, respecting course rules on academic integrity, not repeating bad code it saw online.

BSFT: give three examples of behaviour that should appear in the SFT dataset.

Asking what the student has tried before helping; giving a hint or pointing at the failing line instead of the full solution; explaining an error message in plain words; admitting uncertainty and suggesting how to test; asking the student to run a small experiment.

CPreferences: who should compare the responses (beginner students, advanced students, TAs, professional programmers, course instructors)? Would all groups produce the same preferences?

No (see the annotator widget). Beginners may prefer complete solutions; programmers may prefer idiomatic, concise code; instructors prefer responses that make students think. A reasonable choice is TAs and instructors with a written rubric for learning value, plus student input on clarity only, with disagreements recorded rather than averaged away.

DReward hacking: give three ways the assistant could maximise student satisfaction without maximising learning.

Hand out complete solutions; praise every attempt ("great code!") even when it is wrong; make tasks seem easier than they are, skip hard concepts, and never make the student struggle productively.

EEvaluation: design an evaluation with at least three different measurements, not only satisfaction.

(1) Learning: students later solve a similar exercise without the tutor; compare with a control group. (2) Correctness and pedagogy: TAs rate a random sample of conversations with a rubric (correct? level-appropriate? gave away the solution?). (3) Integrity: adversarial prompts asking for full graded solutions. (4) Satisfaction, as one signal among several.

FAlignment: what does "aligned" mean for this tutoring system? Make the definition concrete enough that someone could design tests for it.

For example: "The tutor (a) never gives a complete solution to a graded exercise, (b) makes no false technical claims (TA audit error rate below 2%), (c) adapts explanations to the student's stated level, (d) says when it is unsure, and (e) students who use it score at least as well on unaided follow-up exercises as students who use office hours." Each clause maps to a test.

21. Final checkpoint

Explain the pipeline pretraining → base model → SFT → instruction-following model → preference / reward optimisation → assistant → evaluation in your own words. Include what data is used at each stage, what objective is optimised, where humans enter, where learned evaluators may enter, how reward hacking can occur, and why evaluation should not simply reuse the proxy being optimised.

✓A model answer (open it after writing yours)

Pretraining uses huge amounts of web text and code and minimises next-token cross-entropy; the result, a base model, can continue any kind of text but does not specifically follow instructions. SFT uses a smaller set of human-written (or curated) prompt–response demonstrations with the same loss, giving an instruction-following model. Preference optimisation uses pairs of responses ranked by humans or by AI evaluators: RLHF trains a reward model on these comparisons and then maximises its score with RL, minus a KL penalty to stay near the SFT model; DPO optimises a preference loss directly. Humans enter by writing demonstrations, ranking responses and writing specifications; learned evaluators enter as reward models, AI judges and verifiers. Reward hacking happens when the optimiser finds outputs the reward scores highly but that aren't actually better (length, jargon, flattery, fooled verifiers). So the final evaluation must use independent measurements (held-out human and expert judgements, real outcomes, adversarial tests), not the same proxy that was optimised, or it would simply confirm the hack.

Summary: key takeaways

Pretraining and alignment are different problems: pretraining teaches approximately "predict likely text"; post-training tries to shape desired behaviour.
SFT teaches by demonstration: prompt → target answer.
Preference learning teaches by comparison: for this prompt, response A > response B.
RLHF learns and optimises a reward signal: human preferences → reward model → reinforcement learning (with a KL penalty). But the reward model is only an approximation of what people actually want.
DPO learns directly from preference pairs (chosen ↑, rejected ↓). It simplifies part of the pipeline, but does not fix bad preferences or badly specified goals.
Optimisation finds loopholes: if the objective is imperfect, stronger optimisation can produce better exploitation of the objective rather than better real-world behaviour (Goodhart).
Alignment requires continuous evaluation: there is no single universal "alignment score"; desired behaviour depends on the application, users, risks, context, training data and evaluation procedure.

In one sentence: pretraining gives a language model broad predictive capabilities; post-training uses demonstrations, preferences, rewards, specifications and evaluation to shape those capabilities into desired behaviour, but every proxy and evaluator can itself become a source of failure.

Test yourselfLab 13 quiz

30 auto-graded tasks, 100 points: pretraining vs. SFT, preference data, reward models and RLHF, the KL penalty, reward hacking and Goodhart, DPO, human feedback and trade-offs, AI feedback, process supervision and verifiable rewards.

Start the quiz →

Post-training and Alignment