Elitsa Yotkova

Free-form answers from LLMs can be correct in many surface forms, and they can fail in very different ways: they may be incomplete, contradict the reference, add hallucinated content, or go along with a false premise. Standard judge-based and similarity-based metrics tend to blur these cases together. We propose a semantic correctness taxonomy with eight ordered classes that, for example, separates answers that are verbose but correct from answers that also contain hallucinations. We release CAP-Correctness, an 8.8k-example benchmark built from widely used QA datasets, and CAP-Statements, an 11k-example dataset for turning question–answer pairs into declarative statements for NLI. Finally, we introduce CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements with bidirectional NLI. In a monotonicity test that checks whether a metric respects the taxonomy's ordering, CAP outperforms established baselines.

SemEval-2026 Task 13 asks systems to detect machine-generated code across programming languages and application scenarios, including languages and domains not seen during training. For Subtask A (binary classification), we compare pretrained code encoders with lightweight feature-based approaches. We design ratio-based stylometric features that are less sensitive to snippet length, and use parsers and a programming-language classifier to extract signals about how descriptive the code is. We also train a separate line-level classifier that finds natural-language text embedded in code samples. A shallow decision tree combined with rules derived from data analysis produces the final predictions. The approach trains on CPU only and runs inference almost instantly, making it a lightweight alternative to large pretrained models.