Class Notes

Lab 00

Introduction

Meet the team Course content SemEval-2027 Top AI conferences

Overview of the lab

  1. About us
  2. Course content
  3. Take part in SemEval-2027
  4. The top AI conferences
  5. How the labs work
Meet the team

About us

Your teaching assistants for the labs.

Teaching assistant · Labs

Dr. Dimitar Dimitrov

PhD in Artificial Intelligence from Sofia University. He has more than 8 papers accepted at ACL* conferences, the most of any Sofia University PhD student at top-tier AI conferences. He works with Prof. Ivan Koychev (Sofia University), Prof. Preslav Nakov (MBZUAI) and Prof. Giovanni Da San Martino (University of Padova).

His research is on disinformation and propaganda: detecting propaganda and persuasion techniques in text and images, harmful memes, and framing and narratives in the news. His work has more than 1,700 citations, and he has co-organised SemEval shared tasks on persuasion techniques (2021, 2024), news narratives (2025) and now multimodal framing (2027).

Selected papers

Google Scholar ACL Anthology
Photo of Elitsa Yotkova

Teaching assistant · Labs · Author of these notes

Elitsa Yotkova

Ex-Google engineer · Google Search · Zurich · 2026

Ex-Google engineer from Google Search and Master's student in Information Retrieval and Knowledge Discovery at Sofia University. In summer 2026 she worked at Google on Google Search in Zurich, on autoraters and on evaluating how reliable they are. Before that she interned at Uber (Amsterdam & Sofia).

Her research is on the robustness of LLM judges, evaluation, and the broader questions of frontier AI safety. At Sofia University she works with Dr. Dimitar Dimitrov, Prof. Ivan Koychev and Prof. Preslav Nakov (MBZUAI), and she collaborates with Violeta Kastreva (ETH Zurich).

eyotkova@g.fmi.uni-sofia.bg Website Scholar LinkedIn GitHub

Course content

Thirteen labs in three parts: classic IR, the Web and language models, and neural retrieval and RAG.

LabAt a glanceWhat it is about
Part 1 · Classic information retrieval
01Inverted Index and Boolean Queries catdogfish 137 23 3589 Dictionaries and postings lists: building an index and answering AND / OR / NOT and phrase queries.
02Text Preprocessing RunningCats! runcat From raw text to index terms: tokenizers, normalization, stop words, stemming and lemmatization.
03Tolerant Retrieval and Spelling Correction qeury query Queries with typos and wild-cards: edit distance, n-gram overlap, context-sensitive correction, Soundex.
04Ranked Retrieval: TF-IDF and BM25 d3d1d7d2 Scoring documents instead of matching them: TF, IDF, TF-IDF vectors, cosine similarity and BM25.
05Evaluating IR Systems 12345 P@5 = 3/5 Is the ranking any good? Precision and recall, MRR, MAP, nDCG and per-query analysis.
Part 2 · The Web and language models
06Web Crawling and PageRank How search engines collect the Web: crawlers, the URL frontier, robots.txt and politeness, and ranking pages with PageRank.
07Query Expansion and Language Models car autovehiclesedan Bridging the vocabulary gap: Rocchio and pseudo-relevance feedback, n-gram language models and query likelihood.
Part 3 · Neural retrieval and RAG
08Neural Text Representations bankriver Text as dense vectors: tokens and embeddings, self-attention with queries, keys and values, and the Transformer.
09Sentence Embeddings and Semantic Search tokenstext From token vectors to one text vector: encoders vs decoders, pooling into sentence embeddings, semantic search and bi-encoders.
10Dense Retrieval and Vector Databases Search by meaning: bi-encoders, similarity functions, approximate nearest neighbours (IVF, HNSW), FAISS and vector databases.
11Hybrid Retrieval and Reranking BM25dense fused Combining lexical and dense search: score and rank fusion (RRF) and cross-encoder reranking.
12Retrieval-Augmented Generation LLM [1] Grounding an LLM in retrieved documents: chunking, top-k context, grounded prompts and citations, prompt injection.
13Evaluating RAG Systems RFC Evaluating each stage: evidence recall, faithfulness and citation quality, unanswerable questions, ablations and LLM judges.
SemEval-2027

Take part in SemEval-2027

SemEval is the yearly international workshop on semantic evaluation: organisers publish a task with data and a metric, teams build systems, and all systems are scored on the same hidden test set.

  • A ready-made problem: the data, the evaluation and the leaderboard come from the organisers, so you spend your time on the system and its analysis.
  • Compete with the world: your results are compared with teams from universities and companies everywhere.
  • Publish a paper: teams can write a system description paper; SemEval papers appear in the ACL Anthology. Example: Elitsa's SemEval-2026 Task 13 paper with Violeta Kastreva.
  • Course project: we recommend a SemEval-2027 task as your project, in teams of at most 2.

The 11 tasks

1RETECO: Reasoning-Oriented Retrievalclosest to this course 2Directional-Consistent Fine-Grained NLI (DiCo-NLI) 3Semantic Change Across Multiple Time Periods 4Par-PARSEME: Paraphrasing of Idioms 5ComPartMENT: Compound and Particle Verb Compositionality 6Culturally Grounded Spoken and Visual Question Answering 7CLaS: Cross-Lingual Alignment and Steering of LLMs 8StereoQueerEval: Stereotypes and Hate Speech towards LGBTQIA+ 9Analysis of Multimodal Framing in the Newsorganised by us 10AgentRisk: Risk in Agentic Multi-Turn Interactions 11VAKRA-Advanced: Multi-Hop Multi-Source Tool Calling

Task 9: Analysis of Multimodal Framing in the News

We are among the organisers of Task 9: Dr. Dimitar Dimitrov, Elitsa Yotkova and Prof. Ivan Koychev, together with Giovanni Da San Martino, Preslav Nakov, Jakub Piskorski and colleagues from Padova, MBZUAI, Helsinki, Porto, Beira Interior and ETH Zurich. It builds on the SemEval-2023 task on persuasion techniques, framing and news genre.

Framing is choosing which aspects of an issue to make salient, so that a reader takes away a particular meaning. Images can reinforce, complement or contradict the framing of the text.

Task 9 page → All SemEval-2027 tasks →
Top AI conferences

The top AI conferences

In AI, the main results are published first at conferences, not in journals. Two groups matter most for this course.

Machine learning

ICLRInternational Conference on Learning Representations. Since 2013, focused on deep learning; reviews are public on OpenReview.
ICMLInternational Conference on Machine Learning. Held since 1980, in the summer.
NeurIPSConference on Neural Information Processing Systems. Held since 1987, in December; the largest ML conference.

Natural language processing · ACL*

ACLAnnual Meeting of the Association for Computational Linguistics: the flagship NLP conference.
EMNLPConference on Empirical Methods in Natural Language Processing. Held since 1996.
NAACLConference of the North American Chapter of the ACL.
  • ACL*: short for the conferences of the Association for Computational Linguistics (ACL, EMNLP, NAACL, and also EACL and AACL). All their papers are free in the ACL Anthology.
  • Findings: papers that are accepted but not selected for the main conference appear in Findings of ACL / EMNLP / NAACL, a companion volume.
  • Workshops: SemEval is a workshop co-located with an ACL* conference, which is why SemEval papers are in the ACL Anthology too.
  • For IR specifically: the main venues are SIGIR (ACM) and ECIR (European); much neural retrieval and RAG work also appears at ACL* and the ML conferences above.

How the labs work

  1. Class notes: every lab has a page with the definitions, worked examples and “Try it” widgets you can play with.
  2. Quizzes: labs are followed by an auto-graded quiz with an explanation for every answer. Answers stay in your browser; nothing is collected.
  3. Menu: the left menu (☰ on phones) lists every lab; switch between Class notes and Quizzes at its top.
  4. Notebooks: the lab exercises (Jupyter notebooks, datasets and setup instructions) are in the course repository.