Skip to main content

agrag.eval.verifier.run_verifier

run_verifier(model:Any, items:Sequence[VerdictItem], *, concurrency:int = _CONCURRENCY) -> list[str]

Run the verifier over items and return one label per item.

A call that raises gives the label ERROR. It is wrong for every gold class and shows in the report. It is never dropped.

Parameters:

  • model (Any) – The chat model under test.
  • items (Sequence[VerdictItem]) – The fixed inputs.
  • concurrency (int) – The most calls that run at once. Lower it for an endpoint that limits concurrent requests.

Returns:

  • list[str] – One verdict label per item, in the order of items.

Raises:

  • ValueError – concurrency is less than 1.