Skip to main content

agrag.eval.answer.CitationAccuracyMetric

CitationAccuracyMetric(judge:DeepEvalBaseLLM, *, threshold:float = 0.5) -> None

Bases: BaseMetric

Score whether each cited sentence follows from the evidence it cites.

The unit is the sentence. A sentence counts as cited when it carries a citation key. A judge decides whether the text of the cited evidence supports the sentence. It scores each sentence with one GEval run. A cited key that the run's ledger did not assign is fabricated: the sentence is unsupported and no judge call happens.

The score is the F1 of two ratios. Precision is supported cited sentences over cited sentences. Recall is supported cited sentences over all sentences of at least four words, plus any shorter cited sentence. score_breakdown holds both and the sentence counts. An answer with no citations scores 0. An abstention, the exact text No relevant evidence found., scores 1. score_breakdown also holds one row per cited sentence with its keys, fabricated flag, support verdict and judge reason.

The test case must come from answer_case, which puts the evidence under metadata["citations"]. A judge failure on any sentence raises.

Attributes:

Functions:

  • a_measure – Judge the cited sentences with up to eight concurrent calls.
  • measure – Judge the cited sentences one after the other.

a_measure​

a_measure(test_case:LLMTestCase, *args:Any, **kwargs:Any) -> float

Judge the cited sentences with up to eight concurrent calls.

Parameters:

  • test_case (LLMTestCase) – The test case built by answer_case.
  • *args (Any) – Additional positional arguments accepted by DeepEval.
  • **kwargs (Any) – Additional keyword arguments accepted by DeepEval.

Returns:

  • float – The citation accuracy score.

judge​

judge = judge

measure​

measure(test_case:LLMTestCase, *args:Any, **kwargs:Any) -> float

Judge the cited sentences one after the other.

Parameters:

  • test_case (LLMTestCase) – The test case built by answer_case.
  • *args (Any) – Additional positional arguments accepted by DeepEval.
  • **kwargs (Any) – Additional keyword arguments accepted by DeepEval.

Returns:

  • float – The citation accuracy score.

score_breakdown​

score_breakdown: CitationScoreBreakdown

threshold​

threshold = threshold