Skip to main content

agrag.eval.verifier.VerdictItem

Bases: BaseModel

One fixed verifier input with its gold verdict.

Attributes:

  • id (str) – A stable id for the item.
  • question (str) – The original question.
  • sub_questions (list[str]) – The sub-questions the question was split into.
  • findings (str) – The findings text with citation keys such as E1.
  • gold (Literal['PASS', 'INSUFFICIENT', 'CONTRADICTORY']) – The verdict the verifier should give.
  • human_reviewed (bool) – True when a person confirmed the gold label.

findings​

findings: str

gold​

gold: Literal['PASS', 'INSUFFICIENT', 'CONTRADICTORY']

human_reviewed​

human_reviewed: bool = False

id​

id: str

question​

question: str

sub_questions​

sub_questions: list[str]