Skip to main content

agrag.eval.settings.EvalJudgeSettings

Bases: BaseSettings

LLM client config for the eval judge.

Attributes:

  • client (LLMClientConfig) – The judge model's client config.
  • temperature (Annotated[float | None, NoDecode]) – The sampling temperature the judge sends. None sends none, for models that reject the parameter. Set EVAL_JUDGE_TEMPERATURE empty to get None. Env: EVAL_JUDGE_TEMPERATURE.

Env prefix: EVAL_JUDGE_.

Functions:

client​

client: LLMClientConfig

from_openai_compatible_env​

from_openai_compatible_env() -> EvalJudgeSettings

Build settings from OpenAI-compatible env vars.

Loads .env first, then resolves EVAL_JUDGE_BASE_URL, EVAL_JUDGE_API_KEY and EVAL_JUDGE_MODEL_ID through pydantic-settings. Each falls back to the shared LLM_* variable when unset or empty, so the judge is the agent's own model unless EVAL_JUDGE_* is set. That model grades its own answers, which biases scores upward. There is no default model.

Returns:

  • EvalJudgeSettings – EvalJudgeSettings with one openai-generic client.

Raises:

  • ValueError – No model id resolves from either set of variables.

model_config​

model_config = SettingsConfigDict(env_prefix='EVAL_JUDGE_', env_file='.env', extra='ignore', hide_input_in_errors=True)

temperature​

temperature: Annotated[float | None, NoDecode] = 0.0