Skip to main content

agrag.ingestion.resolve.comparators.LLMVerify

LLMVerify(*, chunks_by_id:dict[UUID, Chunk], settings:ExtractionLLMSettings | None = None, client:object | None = None, max_pairs_per_batch:int = 50, tracer:Tracer | None = None) -> None

Bases: Comparator

Ask an LLM to verify an ambiguous pair. Last resort. Never UNCERTAIN.

Never raises from an LLM-call failure: it resolves to NO_MATCH instead, by the same fail-safe design as every comparator a Resolver runs. An ambiguous or failed comparison never merges two entities. A missing package extra is a configuration error, not an ambiguous judgment call, and is raised outright instead (see compare's Raises section).

Functions:

  • compare – Return the LLM's verdict for one pair, or NO_MATCH on failure.
  • compare_batch – Verify ambiguous candidate pairs across bounded LLM requests.
  • compare_batch_detailed – Verify pairs and count how many verdicts came back uncertain.
  • compare_with_evidence – Compare two entities and retain any available decision evidence.

Attributes:

Parameters:

  • chunks_by_id (dict[UUID, Chunk]) – Maps a Chunk id to the Chunk, for prompt context.
  • settings (ExtractionLLMSettings | None) – LLM client config. Defaults to ExtractionLLMSettings(). Ignored when client is given: an injected client also disables settings.retry, since a caller building its own client is assumed to own its own retry behavior too.
  • client (object | None) – An already-built BAML client. Tests inject a fake here.
  • max_pairs_per_batch (int) – Maximum pairs sent to the LLM in one request. A large ambiguous population is split into requests of at most this size so one oversized request cannot exceed the model's context limit and silently fail every pair in the batch.
  • tracer (Tracer | None) – Opens the agrag.resolution.llm_verify span and the LLM call spans below it.

chunks_by_id​

chunks_by_id = chunks_by_id

compare​

compare(a:ExtractedEntity, b:ExtractedEntity) -> ComparisonVerdict

Return the LLM's verdict for one pair, or NO_MATCH on failure.

Runs through compare_batch so the single-pair path shares the batch validation and fail-safe behavior.

Raises:

compare_batch​

compare_batch(pairs:list[tuple[int, int, ExtractedEntity, ExtractedEntity]], *, similarities:dict[tuple[int, int], float] | None = None, neighbors_by_index:dict[int, list[str]] | None = None) -> dict[tuple[int, int], ComparisonResult]

Verify ambiguous candidate pairs across bounded LLM requests.

Splits into requests of at most max_pairs_per_batch pairs so one oversized population cannot exceed the model's context limit. Invalid, missing, and uncertain model responses do not merge entities.

Parameters:

  • pairs (list[tuple[int, int, ExtractedEntity, ExtractedEntity]]) – (left_index, right_index, left, right) tuples to verify.
  • similarities (dict[tuple[int, int], float] | None) – Embedding similarity per pair, sent to the model as decision context. Defaults to 0.0 when unknown.
  • neighbors_by_index (dict[int, list[str]] | None) – Entity index to its neighboring-relationship context strings. Looked up globally, so every chunk sees the same map.

compare_batch_detailed​

compare_batch_detailed(pairs:list[tuple[int, int, ExtractedEntity, ExtractedEntity]], *, similarities:dict[tuple[int, int], float] | None = None, neighbors_by_index:dict[int, list[str]] | None = None) -> tuple[dict[tuple[int, int], ComparisonResult], int]

Verify pairs and count how many verdicts came back uncertain.

Parameters:

  • pairs (list[tuple[int, int, ExtractedEntity, ExtractedEntity]]) – The candidate pairs to verify.
  • similarities (dict[tuple[int, int], float] | None) – Embedding similarity per pair, sent to the model as decision context. Defaults to 0.0 when unknown.
  • neighbors_by_index (dict[int, list[str]] | None) – Entity index to its neighboring-relationship context strings. Looked up globally, so every chunk sees the same map.

Returns:

  • tuple[dict[tuple[int, int], ComparisonResult], int] – The per-pair results and the count of raw uncertain verdicts, before the fail-safe maps them to NO_MATCH. A request that errors maps its pairs to NO_MATCH and increments failed_requests.

compare_with_evidence​

compare_with_evidence(a:ExtractedEntity, b:ExtractedEntity) -> ComparisonResult

Compare two entities and retain any available decision evidence.

failed_requests​

failed_requests: int = 0

max_pairs_per_batch​

max_pairs_per_batch = max_pairs_per_batch

settings​

settings = settings