Lab 12
Transformers and Generative Models
Overview of the lab
By the end of this lab you should be able to trace the path from raw text to next-token probabilities: explain subword tokenization and BPE, how token IDs become vectors and why position matters, compute a small self-attention example with queries, keys and values, explain causal masking, the roles of the MLP layer and residual connections, and how a model turns logits into generated text.
- The central question
- Why do we need tokenization?
- Byte Pair Encoding (BPE)
- From tokens to vectors
- Why position matters
- Self-attention
- Manual attention calculation
- Causal masking
- The Transformer block
- Residual connections
- From hidden states to the next token
- The complete pipeline and text generation
- Mini coding exercise: scaled dot-product attention
- Conceptual challenge
- Attention is not automatically an explanation
- Final checkpoint
1. The central question
Throughout this lab we answer one question: how does a Transformer turn text into a prediction for the next token?
A language model assigns high probability to continuations such as mat, floor or chair. Our goal is to understand the computations that make such a prediction possible. This is also all a generative language model does: predict one token, append it, and predict again.
2. Why do we need tokenization?
Computers do not directly operate on words. Before text enters a neural network, it must be converted into discrete units called tokens. Consider the words play, playing, played, player, plays, replay, replaying. There are several possible ways to represent them.
| Option | Example | Advantages ✅ | Problems ❌ |
|---|---|---|---|
| A. Word-level | playing → [playing] | short sequences; tokens are often easy to interpret | the vocabulary can become extremely large; rare or unseen words are difficult to handle; related words (play, playing, played) are unrelated vocabulary entries |
| B. Character-level | playing → [p, l, a, y, i, n, g] | very small vocabulary; any word can be represented | sequences become much longer; the model must learn useful combinations of characters by itself |
| C. Subword | replaying → [re, play, ing] | a compromise: common pieces are shared across related words | tokens are less interpretable; the split depends on the training corpus |
Subword tokenization balances vocabulary size ↔ sequence length ↔ ability to represent unseen words.
Try it: three tokenizers
Type any text. The subword tokenizer here uses a tiny hand-made vocabulary of pieces (play, re, ing, ed, er, s, …) and splits each word into the longest pieces it knows, falling back to single characters. Compare the number of tokens, and try an unseen word like replayers.
1For word-level, character-level and subword-level tokenization, write one advantage and one disadvantage. Why is subword tokenization a useful compromise for large language models?
See the table above. Subword tokenization keeps the vocabulary at a manageable size (tens of thousands of pieces) and the sequences reasonably short, while any word, even one never seen in training, can still be written as a sequence of known pieces (in the worst case, single characters or bytes). Related words share pieces (play in playing, replay, player), so what the model learns about one transfers to the others.
3. Byte Pair Encoding (BPE)
One common idea behind subword tokenization is Byte Pair Encoding. The basic idea is simple:
- start with small units (characters or bytes);
- count which adjacent pairs occur frequently;
- merge the most frequent pair into a new unit;
- repeat, until the vocabulary has the desired size.
The list of merges is the tokenizer: to tokenize new text, split it into characters and apply the learned merges in the same order.
Try it: BPE step by step
The toy corpus low, low, lower, lowest, new, newer, split into characters. Each step counts all adjacent pairs and merges the most frequent one (ties are broken alphabetically here; real implementations use their own fixed rule). After a few merges, the vocabulary contains units such as low, er, new, est. Then tokenize a word the corpus never contained.
Corpus
Merges so far (the tokenizer)
Adjacent pair counts
bpe.py Python
from collections import Counter
corpus = ["low", "low", "lower", "lowest", "new", "newer"]
words = [list(w) for w in corpus] # start from characters
def pair_counts(words):
counts = Counter()
for w in words:
for a, b in zip(w, w[1:]):
counts[(a, b)] += 1
return counts
def merge(words, pair):
a, b = pair
out = []
for w in words:
new, i = [], 0
while i < len(w):
if i + 1 < len(w) and w[i] == a and w[i + 1] == b:
new.append(a + b); i += 2 # replace the pair by one unit
else:
new.append(w[i]); i += 1
out.append(new)
return out
merges = []
for step in range(8):
counts = pair_counts(words)
best = sorted(counts, key=lambda p: (-counts[p], p))[0] # most frequent; ties: alphabetical
merges.append(best)
words = merge(words, best)
print(step + 1, best, counts[best], [" ".join(w) for w in words])
def tokenize(word):
w = list(word)
for pair in merges: # apply the merges in the learned order
w = merge([w], pair)[0]
return w
print(tokenize("newest"), tokenize("lowest"), tokenize("renew"))
# 1 ('l', 'o') 4 ['lo w', 'lo w', 'lo w e r', 'lo w e s t', 'n e w', 'n e w e r']
# 2 ('lo', 'w') 4 ['low', 'low', 'low e r', 'low e s t', 'n e w', 'n e w e r']
# 3 ('e', 'r') 2 ['low', 'low', 'low er', 'low e s t', 'n e w', 'n e w er']
# 4 ('e', 'w') 2 ['low', 'low', 'low er', 'low e s t', 'n ew', 'n ew er']
# 5 ('n', 'ew') 2 ['low', 'low', 'low er', 'low e s t', 'new', 'new er']
# 6 ('e', 's') 1 ['low', 'low', 'low er', 'low es t', 'new', 'new er']
# 7 ('es', 't') 1 ['low', 'low', 'low er', 'low est', 'new', 'new er']
# 8 ('low', 'er') 1 ['low', 'low', 'lower', 'low est', 'new', 'new er']
# ['new', 'est'] ['low', 'est'] ['r', 'e', 'new']2Manual BPE on the corpus above: write the character-level representation, count the pairs, and perform three merges. If newest never appeared in the training corpus, could a subword tokenizer still represent it? Write one possible tokenization.
Initial pair counts: (l, o) 4, (o, w) 4, (w, e) 3, (e, r) 2, (n, e) 2, (e, w) 2, (e, s) 1, (s, t) 1.
- Merge (l, o) → lo (4 times; tied with (o, w)).
- Now (lo, w) occurs 4 times → low.
- (e, r), (e, w), (low, e), (n, e) all occur twice; with alphabetical tie-breaking → er.
Yes: newest can always be written with smaller known pieces, e.g. new + est (after the merges ew, new, es, est), or at worst character by character. The widget reproduces this.
4. From tokens to vectors
Suppose tokenization turns "The cat sat" into tokens, and the tokenizer maps them to the IDs [17, 324, 981]. These numbers are only identifiers: the model should not read cat = 324, dog = 517 as meaning that dog is somehow numerically "larger" than cat.
Every token ID is mapped to a learned vector, its embedding: row number ID of an embedding matrix \(E \in \mathbb{R}^{|V| \times d}\), with \(|V|\) the vocabulary size and \(d\) the model width.
\[ \text{cat} \;(324) \;\mapsto\; E_{324} = [0.2, -0.7, 0.4, \dots] \qquad \text{dog} \;(517) \;\mapsto\; E_{517} = [0.3, -0.6, 0.5, \dots] \]The embedding provides a continuous representation that the neural network can transform. The values are learned during training, so tokens used in similar contexts end up with similar vectors.
Try it: IDs versus embeddings
Toy 2-D embeddings (real models use hundreds or thousands of dimensions). Click a token: its nearest neighbours by cosine similarity are listed. Compare with the IDs: cat (324) and car (325) are neighbours by ID but not by meaning.
5. Why position matters
dog bites man and man bites dog contain the same tokens, but the meaning is different. If the model only knew which tokens were present, it could not distinguish the two sequences. Self-attention, the core operation below, treats its input as a set: shuffle the input vectors and it produces the same vectors, shuffled. So Transformers need information about token position.
Different Transformer architectures encode position in different ways: fixed sinusoids (the original Transformer), learned position vectors (GPT-2), or rotations of the queries and keys (RoPE, used in many recent models). The central idea is the same: the model must know both what a token is and where it appears.
Try it: the same bag of tokens
Toy 2-D token and position vectors. Filled dots are the input vectors of dog bites man, rings those of man bites dog. Switch the positional information on and off.
3(1) Why are token IDs not sufficient as neural-network inputs? (2) What information do embeddings provide? (3) What problem would appear if a Transformer had no positional information?
(1) IDs are arbitrary labels: their numeric order and distances mean nothing, yet a network would treat 517 as "bigger" than 324.
(2) A learned, continuous description of each token, in which tokens used in similar ways get similar vectors, and which the layers can transform.
(3) It could not tell sequences with the same tokens in a different order apart (dog bites man vs man bites dog): word order, and with it grammar and meaning, would be lost.
6. Self-attention
Self-attention allows each token to gather information from other tokens. In The cat sleeps, the representation of sleeps can pull in information about who is sleeping. Each token vector \(x_i\) is transformed into three vectors by three learned matrices: \(q_i = x_i W_Q\), \(k_i = x_i W_K\), \(v_i = x_i W_V\).
- Query (Q): what information am I looking for?
- Key (K): what information do I contain?
- Value (V): what information should I contribute?
A token compares its query with every key. The better a key matches, the more of that token's value is copied into the result. For a whole sequence (one row per token):
\(QK^T\) holds the dot product of every query with every key; softmax (row by row) turns each row into weights that sum to 1; multiplying by \(V\) mixes the value vectors with these weights. \(d_k\) is the length of the key vectors: dividing by \(\sqrt{d_k}\) keeps the scores from growing with the dimension, so the softmax does not saturate.
💡 Softmax is the same function as in multi-class logistic regression: \(\text{softmax}(z)_i = e^{z_i} / \sum_j e^{z_j}\). Do not focus on memorising the formula yet: the next section computes a tiny example.
7. Manual attention calculation
The rows correspond to The, cat, sleeps:
Compute attention for the token sleeps, whose query is \(q = [1, 1]\).
- Step 1, dot products: \(q \cdot k_{\text{The}} = 1\), \(q \cdot k_{\text{cat}} = 1\), \(q \cdot k_{\text{sleeps}} = 2\). The scores are \([1, 1, 2]\).
- Softmax (ignoring the scaling for this tiny example): \(\frac{e^1}{e^1 + e^1 + e^2} \approx 0.21\), so the attention weights are approximately \([0.21, 0.21, 0.58]\).
- Step 2, weighted combination of values: the output is \(0.21\, V_{\text{The}} + 0.21\, V_{\text{cat}} + 0.58\, V_{\text{sleeps}}\), with \(V_{\text{The}} = [1, 2]\), \(V_{\text{cat}} = [3, 0]\), \(V_{\text{sleeps}} = [2, 2]\).
Try it: the attention calculator
Starts on the example above. Pick the query token, switch the \(\sqrt{d_k}\) scaling and the causal mask on and off, or edit any number in \(Q\), \(K\), \(V\). The heatmap shows the full attention-weight matrix, one row per query.
4Calculate \(0.21[1,2] + 0.21[3,0] + 0.58[2,2]\). What did attention do conceptually? Complete the sentence: "Attention created a new representation of sleeps by …"
… by taking a weighted average of the value vectors of all tokens in the sequence, where the weights (how much sleeps "listens" to each token) come from how well its query matches each token's key. The new vector mixes information from The and cat into sleeps, while keeping most of its own value.
8. Causal masking
Autoregressive language models predict the next token. In The cat sat on the mat, while predicting the token after The cat, the model must not use information from sat on the mat: that would be cheating. A causal mask prevents each position from attending to future positions.
Future positions get a very large negative score before the softmax, so they receive weight 0 and contribute nothing. Position \(i\) may attend to positions \(1 \dots i\) only: a lower-triangular pattern.
💡 Why it matters in training: the model is trained to predict every position's next token at once, in one pass over the sentence. The mask makes each of these predictions use only the tokens before it, exactly as at generation time, when the future does not exist yet.
Try it: attention with and without the mask
Invented attention scores for The cat sat on the mat. Rows are the tokens that attend (queries), columns the tokens attended to (keys). Click a row to see its weights. Switch the mask on: all weight above the diagonal moves back onto the past.
5In The scientist wrote the paper: (1) while processing The, which later tokens may it attend to in a causal language model? (2) While processing wrote, may it attend to scientist? (3) To paper? (4) Why is causal masking necessary during next-token training?
(1) None: The is the first token, so it can only attend to itself.
(2) Yes, scientist comes earlier. (3) No, paper is in the future.
(4) Training predicts the next token at every position simultaneously from the full sentence. Without the mask, position \(i\) could simply look at token \(i + 1\), the answer it is supposed to predict, and learn nothing useful. It would also behave differently at generation time, when later tokens are not available.
9. The Transformer block
A simplified Transformer language model:
There are two especially important operations inside each block:
- Self-attention lets tokens exchange information: attention = communication between token positions.
- Feed-forward / MLP layer: after attention, each token representation is independently transformed by a small neural network (the MLP of Lab 11, typically two layers with a nonlinearity, applied to every position with the same weights): MLP = computation within each token position.
- A compact mental model is therefore: communicate → compute, repeated in every block.
- Real models run several attention computations in parallel in each block (multi-head attention), each with its own \(W_Q, W_K, W_V\), so that different heads can track different relations. GPT-2 small, for example, has 12 blocks with 12 heads each and 768-dimensional token vectors.
See it: communicate, then compute
Five token positions, each holding a vector (the coloured bars). Switch between the two sublayers: attention moves information between columns (only from the past, because of the mask); the MLP changes each column on its own.
10. Residual connections
A Transformer does not simply replace the current representation with the result of every operation. Instead it computes
- This is a residual connection: new representation = old representation + useful modification. Each sublayer only has to learn a change, and the original information is never overwritten.
- Residual connections help information and gradients travel through very deep networks: the derivative of \(x + f(x)\) is \(1 + f'(x)\), so there is always a direct path of slope 1 for back-propagation (Lab 11) that does not fade layer after layer.
- The vector that flows through all the blocks and that every sublayer reads from and adds to is often called the residual stream.
Try it: 24 layers with and without residuals
A 16-dimensional vector passes through 24 random layers \(f_\ell(x) = \tanh(g\, W_\ell x)\), where \(g\) is the layer strength. The plot shows how similar the representation still is to the input (cosine similarity). Without residuals every layer replaces the vector and the input is soon forgotten; with residuals every layer adds to it.
6(1) What is the main role of self-attention? (2) Of the MLP? (3) What is the intuition behind a residual connection? (4) Why might stacking many Transformer blocks be more useful than using only one?
(1) Moving information between positions: each token gathers what it needs from the (earlier) tokens.
(2) Processing each position's gathered information on its own: nonlinear feature computation, and much of the model's stored knowledge (most of the parameters are in the MLPs).
(3) Keep what you have and add a correction: \(x + f(x)\). Nothing is lost, and gradients flow straight through.
(4) Each block builds on the previous ones: later blocks can combine information that earlier blocks gathered, forming more abstract features (like the hidden layers of Lab 11, where depth gave compact representations). One round of communication can only relate tokens directly; several rounds can chain relations, e.g. "she" → "scientist" → "who wrote the papers".
11. From hidden states to the next token
Return to The cat sat on the. After many Transformer blocks, the model has a final vector for each token position. The final position holds a contextual representation \(h_{\text{last}}\), which has "read" the whole prompt.
A linear layer maps \(h_{\text{last}}\) to one score per vocabulary token: the logits. Softmax converts them into probabilities. \(T\) is the temperature (\(T = 1\) is the plain softmax).
- The model can then choose the next token (greedy decoding: take the most probable) or sample it from the distribution.
- Temperature: \(T < 1\) sharpens the distribution (safer, more repetitive text), \(T > 1\) flattens it (more varied, more errors). As \(T \to 0\), sampling becomes greedy.
- Top-k sampling: sample only among the \(k\) most probable tokens, so that very unlikely tokens (moon) are never picked.
Try it: logits to probabilities to a token
The logits from the exercise (mat 8.2, floor 6.7, chair 5.1, moon −0.3), plus a few more vocabulary entries (invented) so that, as in the exercise, mat gets about 0.61. Change the temperature and sample.
💡 The exercise's table lists chair with probability 0.05. With these logits that cannot happen together with mat 0.61 and floor 0.14: softmax fixes the ratio of two probabilities by the difference of their logits, \(P_{\text{floor}} / P_{\text{chair}} = e^{6.7 - 5.1} \approx 4.95\), so chair gets about 0.03. The other vocabulary tokens only change the shared denominator.
12. The complete pipeline
You should now be able to explain every step: text → tokens → token IDs → embeddings + positional information → Transformer blocks → contextual token representations → vocabulary logits → probabilities → next token.
See it: follow The cat sat on the through the model
Step through the stages. The sizes are those of GPT-2 small (vocabulary 50 257 tokens, 768-dimensional vectors, 12 blocks); the token IDs and numbers are illustrative.
How a generative model writes text
A generative language model repeats the pipeline in a loop, autoregressively: predict a distribution for the next token, pick one (greedy or sampling), append it to the input, and predict again, until an end token or a length limit. Everything a chatbot writes is produced this way, one token at a time.
Try it: the generation loop
To keep it in the browser, the Transformer is replaced by a tiny count-based model (how often each word followed the previous two words in 24 short training sentences). The loop around it is exactly the one a real LLM runs. Step one token at a time, or generate to the end.
13. Mini coding exercise
Implement scaled dot-product attention with NumPy (or PyTorch):
attention_exercise.py Python
import numpy as np
def softmax(x, axis=-1):
x = x - np.max(x, axis=axis, keepdims=True) # subtract the max for numerical stability
exp_x = np.exp(x)
return exp_x / np.sum(exp_x, axis=axis, keepdims=True)
def attention(Q, K, V, mask=None):
# 1. Compare queries with keys.
scores = ______________________________
# 2. Scale the scores.
scores = ______________________________
# 3. Apply the causal mask if provided.
if mask is not None:
scores = ______________________________
# 4. Convert scores to attention weights.
weights = ______________________________
# 5. Combine the value vectors.
output = ______________________________
return output, weightsattention_solution.py Python
import numpy as np
def softmax(x, axis=-1):
x = x - np.max(x, axis=axis, keepdims=True)
exp_x = np.exp(x)
return exp_x / np.sum(exp_x, axis=axis, keepdims=True)
def attention(Q, K, V, mask=None):
scores = Q @ K.T # 1. (n_queries, n_keys) similarity matrix
scores = scores / np.sqrt(K.shape[-1]) # 2. divide by sqrt(d_k)
if mask is not None:
scores = np.where(mask, scores, -np.inf) # 3. forbidden positions -> -infinity
weights = softmax(scores, axis=-1) # 4. every row sums to 1
output = weights @ V # 5. weighted average of the values
return output, weights
Q = np.array([[1, 0], [0, 1], [1, 1]], dtype=float) # The, cat, sleeps
K = np.array([[1, 0], [0, 1], [1, 1]], dtype=float)
V = np.array([[1, 2], [3, 0], [2, 2]], dtype=float)
# the exercise's example: no scaling, query "sleeps"
w = softmax(Q[2] @ K.T)
print(w.round(2), (w @ V).round(2)) # [0.21 0.21 0.58] [2. 1.58]
causal = np.tril(np.ones((3, 3), dtype=bool)) # position i may see 0..i
out, weights = attention(Q, K, V, mask=causal)
print(weights.round(3))
# [[1. 0. 0. ]
# [0.33 0.67 0. ]
# [0.248 0.248 0.503]]
print(out.round(3))
# [[1. 2. ]
# [2.34 0.66 ]
# [2. 1.503]]Questions after implementation
aWhat is the shape of Q @ K.T?
(number of queries) × (number of keys): with \(n\) tokens, \(n \times n\). One score for every (query, key) pair; here 3 × 3.
bWhat does one row of the attention-weight matrix represent?
How much one token (the query) attends to each token of the sequence: the mixing proportions used to build that token's output from the value vectors.
cWhy do the values in one row sum to 1?
The softmax is applied row by row, and softmax outputs are positive and normalised to sum to 1. So the output is a weighted average (a convex combination) of the value vectors.
dWhat changes after a causal mask is applied?
Every weight above the diagonal becomes exactly 0, and the remaining weights in each row are renormalised to sum to 1. The first token can only attend to itself (row 1 = [1, 0, 0]).
eWhat happens if one attention score becomes much larger than all others?
Its weight approaches 1 and all others approach 0: attention becomes a hard lookup that copies a single value vector. The softmax then saturates and its gradients become tiny, which is one reason for the \(1/\sqrt{d_k}\) scaling.
14. Conceptual challenge
Consider: The scientist who wrote the papers was tired because she …
AWhy might a word-level tokenizer struggle more than a subword tokenizer with rare scientific terminology?
Rare terms (photoluminescence, transcriptomics) are probably not in a word vocabulary and become one "unknown" token, losing all information. A subword tokenizer splits them into known pieces (photo, lumin, escence) that carry meaning shared with other words.
BWhich earlier words might the token she attend strongly to?
scientist (the likely referent), and perhaps tired and was (what is being said about her) or because (the clause structure). Different heads may track different relations.
CCould the token scientist attend to the later token tired in a causal language model while processing scientist?
No. tired is a future position, and the causal mask sets its score to −∞, so its weight is 0. Information flows only from earlier to later positions: later, tired can attend to scientist, not the other way round.
DWhat does the MLP do that self-attention does not?
It applies a nonlinear transformation to each position's vector on its own, turning the gathered information into new features, and it stores much of the model's knowledge in its weights. Attention only moves information between positions, as weighted averages of value vectors (mixing, not new per-token computation).
EWhy does the model produce logits for the entire vocabulary rather than directly outputting one word?
The next token is uncertain: several continuations can be plausible. A full probability distribution expresses that uncertainty, is needed to train with cross-entropy (the probability of the correct token must be defined), and lets us choose how to decode (greedy, sampling, temperature, top-k).
FWhat would go wrong if the model had no information about token position?
Attention would see the sentence as an unordered bag of tokens: she could not tell whether scientist came before or after it, dog bites man would equal man bites dog, and word order, grammar and much of the meaning would be lost.
15. Attention is not automatically an explanation
Suppose you inspect one attention head and find "she" → "scientist": 0.70. Can you conclude "the model believes with 70% certainty that she refers to scientist"? No.
- Attention weights are part of the model's internal computation. They are not automatically a faithful explanation of the model's reasoning or beliefs.
- There are many heads in many layers, and they may perform different computations. One head's weights are one piece of a large computation.
- A weight says how much of a value vector is copied, not how important it is: a large weight on a value vector with small or irrelevant content changes little. Later MLPs and layers can substantially transform the representation.
7Explain why "High attention weight proves that one word caused the model's final prediction" is too strong. Write a better, more careful interpretation.
Attention weights are correlational, local to one head and one layer, and apply to value vectors that later layers keep transforming. Other heads, the MLPs and the residual stream also shape the output, so a high weight is neither necessary nor sufficient for causing the prediction.
Better: "In this head and layer, the representation of she copied a large share of its information from the position of scientist. This suggests the model may use this link. To show that it causes the prediction, we would need an intervention, e.g. removing or changing that information and checking whether the prediction changes."
16. Final checkpoint
Without looking back at the notes, explain in your own words how a Transformer turns the text The cat sat on the into a probability distribution over possible next tokens. Include tokenization, embeddings, positional information, self-attention, causal masking, MLP layers, residual connections, logits and softmax.
✓A model answer (open it after writing yours)
The text is split by a subword tokenizer (e.g. BPE) into tokens, which are mapped to IDs. Each ID selects a learned embedding vector, and positional information is added so the model knows where each token is. The vectors then pass through a stack of Transformer blocks. In each block, self-attention lets every position gather information from the earlier positions: queries are compared with keys, softmax turns the scaled scores into weights, and the value vectors are averaged with those weights; the causal mask sets the scores of future positions to −∞ so that nothing is read from the future. Then an MLP transforms every position separately. Both sublayers add their result to their input through residual connections. After the last block, the vector at the final position (the) represents the whole prompt; a linear layer turns it into one logit per vocabulary token, and softmax turns the logits into probabilities, e.g. mat 0.61, floor 0.14. A token is chosen or sampled, appended, and the process repeats to generate text.
Summary: key takeaways
A Transformer tokenizes text into subword units, maps those units to vectors, repeatedly allows token positions to exchange contextual information through self-attention and transform that information through feed-forward layers, and finally converts the resulting representation into scores and probabilities over possible next tokens.
- Tokenization: subwords (e.g. BPE, learned by repeatedly merging the most frequent pair) balance vocabulary size, sequence length and unseen words.
- Embeddings + position: IDs select learned vectors; positional information tells the model where each token is.
- Self-attention = communication across positions: \(\text{softmax}(QK^T/\sqrt{d_k})V\), with a causal mask so that no position sees the future.
- MLP = computation within a position. Residual connections: new = old + modification.
- Logits → softmax → probabilities → next token, and a generative model repeats this loop token by token.
- Attention weights are internal computation, not automatically an explanation.
The overall pipeline: text → tokens → vectors → contextual representations → logits → probabilities.
27 auto-graded tasks, 100 points: tokenization and BPE, IDs and embeddings, position, queries, keys and values, attention by hand, causal masking, blocks and residuals, logits, temperature, top-k and generation.