Skip to main content

agrag.eval.verifier

Verifier calibration: does the verifier give the right verdict?

Each item is a fixed (question, sub-questions, findings) input with a gold verdict. The verdict and the gold label are both one of three values, so scoring is an equality check and needs no judge. verdict_report gives per-class precision, recall and F1, the macro F1 that gates, and a confusion matrix.

Classes:

  • ClassScores – Precision, recall and F1 of one verdict class.
  • VerdictItem – One fixed verifier input with its gold verdict.
  • VerdictReport – Scores of predicted verdicts against gold verdicts.

Functions:

  • run_verifier – Run the verifier over items and return one label per item.
  • verdict_case – Build a test case with the predicted and the gold verdict.
  • verdict_match_metric – Build a metric that scores 1.0 when the verdict equals the gold verdict.
  • verdict_report – Score predicted verdicts against gold verdicts.