Skip to main content

agrag.eval.extraction.run_extractor

run_extractor(extractor:Extractor, items:Sequence[ExtractionGold], schema:GraphSchema, *, concurrency:int = _CONCURRENCY) -> list[LLMTestCase]

Run an extractor over gold items and build one test case per item.

Chunk and document ids come from the item id, so runs are repeatable.

Parameters:

  • extractor (Extractor) – The extractor under test.
  • items (Sequence[ExtractionGold]) – The gold-annotated chunks.
  • schema (GraphSchema) – The schema the extractor works to.
  • concurrency (int) – The most extractor calls that run at once. Lower it for an endpoint that limits concurrent requests.

Returns:

  • list[LLMTestCase] – One test case per item, in the order of items.

Raises:

  • ValueError – concurrency is less than 1.