Lab 09

Sentence Embeddings and Semantic Search

Encoders Pooling Semantic search Bi-encoders

Work in progress These notes are still being written and reviewed: content may change, and some parts may still contain mistakes.

Where Lab 08 left us

text → tokenizer → token IDs → initial token embeddings → Transformer → contextual token vectors

Lab 08 ended with one context-aware vector per token. Search needs one vector for an entire query or document. This lab covers how we get it (encoders and pooling) and how we search with it (semantic search and bi-encoders).

Part I

What is an encoder?

📐 Encoder

\operatorname{Encoder}(text) \rightarrow \mathbb{R}^{d}

A model that consumes an input and returns a representation of that input. For retrieval, vector = encoder("how to repair a car"): the goal is a representation that can later be compared with document representations.

❗ Encoder does not mean generator. An encoder produces a representation (text → encoder → vector); a generative model instead repeatedly predicts output tokens.

Three broad Transformer families

Architecture	Main role	Example family	Useful for
Encoder-only	represent / understand the input	BERT (and MiniLM)	classification, token labeling, retrieval embeddings, reranking components
Decoder-only	generate autoregressively	GPT-style models	text generation; the model behind RAG answers
Encoder–decoder	transform one sequence into another	T5	translation, summarization, sequence-to-sequence tasks

These are broad architectural categories, not rigid task boundaries.

Encoder-only: input tokens → Transformer encoder → contextual representations. Every token can attend to every other token, left and right.
Decoder-only: predicts the next token from the previous ones, \(P(x_t \mid x_1, \ldots, x_{t-1})\). Generation is a loop: prompt → predict next token → append it → predict next token → append → … This is the conceptual model needed for RAG. A token may only attend to itself and earlier tokens (a causal mask), since the later ones do not exist yet when it is generated.
Encoder–decoder: input sequence → encoder → internal representations → decoder → output sequence; the decoder attends to the encoder's representations while it generates.

Try it: who can see whom

The attention pattern allowed in an encoder and in a decoder. Click a row to see which tokens that token may attend to. In decoder mode, Generate appends the next word, as in the generation loop.

itself or an earlier token a later token ✕ not allowed

Part II

From token vectors to one text vector

After a Transformer we normally have \(h_1, h_2, \ldots, h_n\): one contextual representation per token. But retrieval often wants \(E(text) \in \mathbb{R}^d\): one vector for the whole text. So we need a pooling strategy. A simple one is mean pooling, averaging the token representations:

E(text) = \frac{1}{n} \sum_{i=1}^{n} h_i

Other strategies: the representation of a special token such as [CLS], weighted pooling, learned pooling (and max pooling, the largest value per dimension). The exact choice depends on the model: all-MiniLM-L6-v2 was trained with mean pooling followed by normalization to length 1.

See it: pooling the token vectors

The real last-layer vectors of the 8 tokens of how to repair a car engine (one row each, 384 cells, blue positive, red negative) and the pooled vector. Switch the pooling strategy to see its vector and how it ranks the five demo texts.

💡 Mean pooling reproduces model.encode exactly. With the [CLS] vector or max pooling every score lands between about 0.5 and 0.9: banana nutrition gets 0.54 for car repair, where mean pooling gives 0.05. The model was trained to make the mean useful; the other poolings are not what it learned.

The sentence embedding pipeline

text → tokenizer → token embeddings → Transformer → contextual token vectors → pooling → sentence embedding

Why not any Transformer?

A generic Transformer representation is not automatically optimized so that semantically similar sentence → nearby vector. Retrieval models are usually trained with an objective that explicitly encourages useful geometry: similar / relevant pairs move closer, irrelevant pairs move farther apart (all-MiniLM-L6-v2 was fine-tuned this way on over a billion sentence pairs). This is why a model trained for sentence embeddings or retrieval is usually preferable to taking arbitrary hidden states from a generic model.

See it: a big generic model vs a small trained one

roberta-large (355 million parameters, 24 layers, 1,024 dimensions, pre-trained only to fill in masked words), mean-pooled exactly like MiniLM, against all-MiniLM-L6-v2 (22.7 million parameters), on the five demo texts.

💡 Every roberta-large score lies between 0.988 and 0.996: all texts point in almost the same direction, and the ranking is close to noise (what should I eat to stay healthy ranks how to repair a car engine first). Size is not the point; the training objective is.

Part III

Semantic search: what changes?

Lexical search asks

Do the query and document share useful words?

Semantic search asks

Are the query and document close in a learned representation space?

Query car repair, document automobile maintenance: a lexical system sees no term overlap; a semantic-search system can still assign high similarity (0.733 here) because the encoder maps the two texts to nearby vectors:

q \rightarrow E(q) \qquad d \rightarrow E(d) \qquad score(q, d) = sim(E(q), E(d))

This is not a separate magical search paradigm. It is still ranked retrieval; the main difference is the representation used to compute the score.

Question	Lexical search	Semantic search
Main signal	term overlap	embedding similarity
Representation	sparse term vector	dense learned vector
Good at exact identifiers	usually yes	variable
Good at paraphrases	limited	often better
Needs a learned model	no	yes
Typical methods	BM25, TF-IDF	bi-encoder + vector search

A useful course-level view: lexical search retrieves by words, semantic search retrieves by learned representations. Lab 11 combines both.

See it: a map of fifteen phrases

Real embeddings of fifteen phrases, projected from 384 to 2 dimensions with PCA. Click a dot (or a phrase in the list): the dashed lines go to its three nearest neighbors by the true 384-dimensional cosine, and the list ranks all the others.

💡 The two PCA axes keep only 40% of the variance (22% + 18%). In the picture the food and the history phrases lie on top of each other; in 384 dimensions they are far apart (click one). A 2-D picture of an embedding space is always a distorted one.

Part IV

Bi-encoders

A bi-encoder encodes the query and the document independently, then compares the two vectors, for example with the cosine:

score(q, d) = sim(q, d) = \cos(E(q), E(d))

query↓encoder↓q vector

\(sim(q, d)\)the only point where query and document meet

document↓encoder↓d vector

Why bi-encoders are efficient: documents can be encoded once (documents → encoder → stored vectors). At query time only the query is encoded (query → encoder → query vector → compare with the stored vectors). This makes large-scale retrieval possible.
Same encoder or two encoders? Some systems use the same model for both sides (like all-MiniLM-L6-v2); others use a query encoder and a document encoder with separate parameters or roles. The core idea remains: encode independently, compare vectors.
❗ Important limitation: a bi-encoder compresses each text before comparison. The query and the document never interact token by token. This is efficient, but some fine-grained relevance signals may be lost; it motivates cross-encoder reranking in Lab 11, where query and document go through the model together.

Try it: count the model calls

documents

queries per day

Part V

A small embedding demo

A sentence-embedding model, all-MiniLM-L6-v2: the encoder-only Transformer of this lab (6 layers, mean pooling, 384 dimensions). All scores on this page are its real outputs, computed in Python (the model does not run in your browser).

embed.py Python

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

texts = [
    "how to repair a car engine",
    "automobile maintenance and servicing",
    "banana nutrition and healthy diets",
    "treatment for myocardial infarction",
    "therapy after a heart attack",
]
embeddings = model.encode(texts, normalize_embeddings=True)
print(embeddings.shape)       # one row = one text embedding
# (5, 384)
print(embeddings[0][:10])     # not interpretable as words, unlike TF-IDF
# [-0.08241399  0.06654591  0.06800879 -0.03249352 -0.06150604  0.02232708
#  -0.04297901  0.01786646 -0.08514726 -0.04902695]

semantic_search.py Python

query = "car repair"
q = model.encode([query], normalize_embeddings=True)
scores = q @ embeddings.T                 # unit vectors: dot product = cosine

for text, score in sorted(zip(texts, scores[0]), key=lambda x: x[1], reverse=True):
    print(round(float(score), 3), text)
# 0.655 how to repair a car engine
# 0.647 automobile maintenance and servicing
# 0.187 treatment for myocardial infarction
# 0.135 therapy after a heart attack
# 0.047 banana nutrition and healthy diets

Before running it: which sentence should rank highest? automobile maintenance and servicing shares no word with car repair, yet scores 0.647, almost as high as the text that contains repair and car.

Try it: more queries on the five texts

What not to claim

Do not say "the model understands meaning perfectly". Embeddings can fail on negation, numbers, rare entities, specialized terminology, domain shift, long text, ambiguous queries. A learned representation is useful, not magical. Real scores for four candidate texts per query:

💡 Dense: cosine of all-MiniLM-L6-v2. BM25: rank_bm25 on the four texts only, so its IDF is crude: a word in half of the texts gets IDF 0 (why every BM25 score is 0 for the negation query). The point is not a winner: semantic retrieval is not uniformly superior, and neither is lexical.

Summary: key takeaways

An encoder maps text to a representation; a decoder-style LLM generates tokens autoregressively.
Sentence embeddings require a way to turn contextual token vectors into one text vector (pooling).
Retrieval models are trained so that vector geometry is useful for relevance or semantic similarity.
Semantic search retrieves by learned representations, lexical search by words; neither is uniformly better.

Optional exercise

Encode car repair, automobile maintenance, banana nutrition, heart attack treatment, therapy for myocardial infarction and create the pairwise cosine-similarity matrix (E @ E.T). The heatmap below shows these five and ten more phrases; click a cell.

Click a cell.

1Among the five, which pairs are most similar? Are the results what you expected?

car repair – automobile maintenance 0.733 and heart attack treatment – therapy for myocardial infarction 0.641: the two paraphrase pairs, with no shared word. Every other pair is below 0.16.

2Can you find one surprising similarity?

The largest cross-topic value among the five is car repair – therapy for myocardial infarction (0.154), about the same as car repair – heart attack treatment (0.153): "repair" and "treatment/therapy" are both about fixing something. Among the fifteen phrases: car repair – cardiac surgery = 0.321, higher than two history phrases score with each other (castles of the middle ages – the fall of the Roman Empire, 0.262).

3Construct examples with negation, numbers, rare names and acronyms. Where does the model behave unexpectedly?

See What not to claim: documents not about neural networks is closest to the documents about neural networks; error code 0x80070005 is closest to a different error code (0.937); BERT paper 2018 ranks RoBERTa (2019) and Sesame Street's Bert above the BERT paper; and the word-order table of Lab 08 shows flights from Sofia to London ≈ flights from London to Sofia (0.993).