agrag.eval

Public evaluation metrics for agrag, built on DeepEval and AgentEvals.

Needs the eval extra: pip install 'agentic-graphrag[eval]'.

Modules:

• adapter – Adapters that let plain scoring functions report through DeepEval.
• answer – Answer-quality metrics for agrag, scored from questions and reference answers.
• extraction – Extraction quality: entity and relation-triple F1 against gold annotations.
• judge – A DeepEval judge model backed by an agrag chat model.
• resolution – Resolution quality: B-cubed and pairwise scores of mention clusters.
• settings – Env-backed configuration for the eval judge model.
• trajectory – Agent trajectory evaluation: read runs, check structure, judge quality.
• verifier – Verifier calibration: does the verifier give the right verdict?

Classes:

• ChatModelJudge – Wrap a LangChain chat model as a DeepEval judge.
• CitationAccuracyMetric – Score whether each cited sentence follows from the evidence it cites.
• CitationScoreBreakdown – Available precision, recall, and sentence-count fields for one score.
• CitationSentenceRow – One cited sentence and its support verdict, for the report.
• ClusterAssignment – A grouping of mentions, listed by position in the mention list.
• EvalJudgeSettings – LLM client config for the eval judge.
• ExtractionGold – One gold-annotated chunk of text.
• MicroScores – Dataset scores pooled over every item, for entities and relations.
• ScoreMetric – A DeepEval metric backed by a plain scoring function.
• ScoreResult – The outcome of one scoring function call.
• Scores – Precision, recall and F1.
• SpanCapture – Capture one agent run's spans for read_trajectory.
• Trajectory – The ordered steps of one agent run.
• VerdictItem – One fixed verifier input with its gold verdict.
• VerdictReport – Scores of predicted verdicts against gold verdicts.

Functions:

• answer_case – Build the test case that every answer-quality metric scores.
• cluster_quality_metric – Build a metric for cluster quality on one test case.
• context_precision – Build the metric for useful evidence ranked before noise.
• context_recall – Build the metric for reference facts that the evidence covers.
• correctness – Build the answer correctness metric against the reference.
• entity_quality_metric – Build a metric for entity F1 on one test case.
• expected_tools_metric – Build the metric that the run called every expected tool.
• extraction_case – Build a test case that holds a predicted and a gold extraction.
• faithfulness – Build the metric for claims that the evidence the agent saw does not contradict.
• final_answer – Return the text of the last assistant message in an agent run.
• micro_scores – Pool measured entity and relation metrics into dataset scores.
• parse_json_case – Read the (actual, expected) models back from a JSON test case.
• read_trajectory – Read the tool and model steps from finished spans.
• relation_quality_metric – Build a metric for relation triple F1 on one test case.
• resolution_case – Build a test case that holds predicted and gold clusters.
• retry_budget_metric – Build the metric that retries stay within budget.
• run_extractor – Run an extractor over gold items and build one test case per item.
• run_resolver – Resolve mention strings and return the clusters the resolver forms.
• run_resolver_detailed – Resolve mention strings and return the clusters plus raw evidence.
• run_verifier – Run the verifier over items and return one label per item.
• task_completion – Build the judged metric for task completion.
• to_json_case – Build a test case that carries structured data as JSON.
• trajectory_case – Build the test case every trajectory metric scores.
• trajectory_quality – Build the judged metric for trajectory quality.
• verdict_case – Build a test case with the predicted and the gold verdict.
• verdict_match_metric – Build a metric that scores 1.0 when the verdict equals the gold verdict.
• verdict_report – Score predicted verdicts against gold verdicts.
• verifier_before_answer_metric – Build the metric that the verifier ran before the answer.