agrag.eval.trajectory

Agent trajectory evaluation: read runs, check structure, judge quality.

A trajectory is the ordered tool and model steps of one agent run, read from its OpenTelemetry spans (see agrag.agents.tracing). Structural rules over it are deterministic; task completion and trajectory quality use an LLM judge.

Classes:

• SpanCapture – Capture one agent run's spans for read_trajectory.
• Step – One tool or model step of an agent run.
• Trajectory – The ordered steps of one agent run.

Functions:

• expected_tools_metric – Build the metric that the run called every expected tool.
• read_trajectory – Read the tool and model steps from finished spans.
• retry_budget_metric – Build the metric that retries stay within budget.
• task_completion – Build the judged metric for task completion.
• trajectory_case – Build the test case every trajectory metric scores.
• trajectory_quality – Build the judged metric for trajectory quality.
• verifier_before_answer_metric – Build the metric that the verifier ran before the answer.