How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Yotkova, Elitsa; Kastreva, Violeta; Velkov, Petar; Boyanov, Hristo; Dimitrov, Dimitar; Nakov, Preslav; Koychev, Ivan

Why Is Open QA Evaluation Hard?

Free-form answers often have no explicit correctness label.
A gold answer gives us a semantic anchor, not a string to match.
Binary accept/reject judgments hide important distinctions.
Answers can be partial, overinclusive, unsupported, contradictory, or based on a false premise.

Correctness Is Not Binary

Question

Name 3 cities in Europe

Gold

Paris, Rome, Sofia

Overinclusive invalid

Paris, Berlin, Tokyo

Partial

Oslo, Amsterdam

Overinclusive valid

Paris, Berlin, Sofia, all of which have a population over 1.2 million people

Not every answer is simply correct or wrong. Correctness comes in different semantic forms and degrees.

How Should We Classify Correctness Instead?

1ExactPrediction is identical to the gold answer.

2EquivalentRephrased or semantically equivalent to the gold answer.

3Alternative CorrectDifferent answer, but still correct under the question.

4Overinclusive ValidContains the correct answer plus extra valid information.

5PartialContains only part of the required information.

6Overinclusive InvalidContains the correct answer plus extra invalid information.

7InvalidSemantically incorrect answer.

8ContradictoryExplicitly contradicts the gold answer or question premise.

The gold/reference answer defines the expected content.
The taxonomy captures (1) the relationship between gold and predicted answers and (2) common LLM-response patterns.
A good metric should score more correct answers higher than less correct ones.

Monotonicity Criteria for Correctness

exact ≈ equivalent ≈ alternative-correct ≥ overinclusive_valid > partial >
overinclusive_invalid > invalid ≥ contradictory

More correct classes should receive higher metric scores.

CAP & Our Evaluation Framework

CAP (Context-Aware Precision) is a reference-based metric for evaluating semantic correctness in open QA.

Gold Answer

Q: Name a city in Bulgaria.
A: Sofia

Predicted Answer

Q: Name a city in Bulgaria.
A: Burgas

QA → Statements

\(s_g := s(q, g)\)Sofia is a city in Bulgaria.

\(s_p := s(q, p)\)Burgas is a city in Bulgaria.

Scoring

Bidirectional
weighted NLI

g → psemantic compatibility

p → gsemantic completeness

CAP

CAP(g,p)
∈ [0, 1]

\[ D(s_g \to s_p) := P_{\text{entailment}}(s_g \to s_p) + \lambda\, P_{\text{neutral}}(s_g \to s_p) \]

\[ \begin{aligned} D(s_g \to s_p) := {} &P_{\text{entailment}}(s_g \to s_p) \\ &+ \lambda\, P_{\text{neutral}}(s_g \to s_p) \end{aligned} \]

where \(\lambda \in [0, 1]\) controls the contribution of the neutral class

\[ \mathit{CAP}(s_g, s_p) := \alpha\, D(s_g \to s_p) + (1 - \alpha)\, D(s_p \to s_g) \]

\[ \begin{aligned} \mathit{CAP}(s_g, s_p) := {} &\alpha\, D(s_g \to s_p) \\ &+ (1 - \alpha)\, D(s_p \to s_g) \end{aligned} \]

In our experiments, we use \(\alpha = 0.85\) and \(\lambda = 0.30\).

Our Contributions

Taxonomy + dataset: 8 correctness classes, 8.8k examples
Metric + reformulation data: CAP + 11k QA→statement triples
Diagnostic protocol: checks monotonicity criteria for correctness and reveals metric failure modes

How do we know whether a metric score reflects correctness?

Gold statementPrediction statementRelationshipCAP Water boils at 100°CWater boils at 0 °CcontradictoryCAP 0.01 Water boils at 100°CThe boiling point of water is 100°CequivalentCAP 0.98

We use our taxonomy as a human-interpretable target ordering.

chance (50%)

CAP

77.70

COMET

61.10

BERTScore

56.18

ROUGE-L

55.35

METEOR

53.71

BLEU

51.57

020406080100

Pairwise accuracy (%)

Monotonicity violations

CAP4/25

COMET9/25

BERTScore11/25

Evaluators in our study often fail to preserve the intended semantic correctness ordering.

Does the taxonomy generalize to real LLM outputs?

In 3,000 responses from 3 LLM families, (1) all eight taxonomy classes appeared, and (2) CAP largely preserved the semantic ordering.

Reusable Resources

CAP-Correctness

8,827

question, gold, predicted, label

8 semantic correctness classes

OpenBookQA, ARC, MMLU

1,638 human labeled

κ = .779 human–LLM agreement

CAP-Statements

11,000

question, answer, statement

4 question types

train/dev/test reformulation

1,353 manually checked

90% correct

Use CAP

pip install -e .

from cap_eval import CAPScorer

scorer = CAPScorer()  # cross-encoder/nli-deberta-v3-large, pinned revision
scorer.score(
    ["The powerhouse of the cell is the mitochondria."],   # premises (gold)
    ["The mitochondria is the powerhouse of the cell."],   # hypotheses (candidate)
)

See the repository for scoring raw (question, gold answer, answer) triples and for reproducing the results.

Poster

The full poster from EMNLP 2026 in Budapest.

Open the poster (PDF)

Citation

@inproceedings{yotkova2026correct,
  title     = {How Correct Is Your Answer? A Semantic Correctness Framework for Open {QA} Evaluation},
  author    = {Yotkova, Elitsa and Kastreva, Violeta and Velkov, Petar and Boyanov, Hristo and
               Dimitrov, Dimitar and Nakov, Preslav and Koychev, Ivan},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  url       = {https://arxiv.org/abs/2609.01369}
}