Skip to content

Evaluation and Release Gates

AgenticLens evaluates agent behavior from local outputs and structured traces. The evaluation layer converts workflow evidence into repeatable quality, tool-use, latency, and cost checks. It can run without a hosted service or model provider.

Evaluation Model

An evaluation uses two inputs:

  • a versioned YAML or JSON test suite containing acceptance criteria
  • samples containing the observed output and its AgenticLens trace

Each test case can define:

  • exact output or required output phrases
  • required and forbidden tool calls
  • required tool arguments
  • expected JSON output structure and required fields
  • maximum turn count
  • maximum end-to-end latency
  • maximum estimated cost
  • tags and application-specific metadata

Built-in checks are deterministic. Custom evaluators and LLM-as-a-Judge evaluators use the same normalized score contract. A case passes only when every required check reaches its configured threshold. The report includes per-check evidence, aggregate pass rate, average score, latency, and total cost.

Unified Evaluator Framework

The evaluator framework is provider-neutral. An evaluator receives an EvaluationContext containing the test case, observed sample, trace, threshold, and evaluator-specific configuration. It returns one or more normalized Score objects with a value from 0.0 to 1.0, an explanation, and optional evidence metadata.

Trusted evaluators are registered explicitly in application code:

from agenticlens.evaluation import (
    EvaluationContext,
    EvaluatorRegistry,
    LLMJudgeEvaluator,
    Score,
    evaluate_suite,
)

def judge(context: EvaluationContext) -> Score:
    # Call any hosted model, local model, or internal evaluation service here.
    result = call_model(
        input=context.case.input,
        output=context.sample.output,
        rubric=context.config.config["rubric"],
    )
    return Score(
        name="answer_quality",
        value=result.score,
        passed=False,  # AgenticLens applies the suite threshold.
        explanation=result.explanation,
        metadata={"model": result.model},
    )

registry = EvaluatorRegistry()
registry.register(LLMJudgeEvaluator("quality_judge", judge))
report = evaluate_suite(suite, samples, registry=registry)

The suite selects the evaluator and controls its acceptance threshold:

cases:
  - id: support-answer
    name: Support answer quality
    evaluators:
      - name: quality_judge
        threshold: 0.80
        required: true
        config:
          rubric: The answer must be correct, grounded, and concise.

Suite files never import or execute Python modules. Registration is performed by trusted application code, preventing untrusted suite configuration from loading arbitrary code.

CallableEvaluator supports Python rules, semantic similarity functions, safety classifiers, RAG metrics, and internal services. LLMJudgeEvaluator identifies provider-supplied LLM-as-a-Judge results in reports while leaving model selection, credentials, prompts, retries, and structured output handling under application control. See examples/custom_llm_judge.py for a complete, runnable registration example.

BusinessRuleEvaluator is a named wrapper around trusted application logic for organization-specific pass/fail rules that do not need an LLM-as-a-Judge evaluator.

Run an Evaluation

agenticlens evaluate suite.yaml samples.json \
  --save evaluation.json \
  --html evaluation.html

The JSON report is suitable for CI and further analysis. The standalone HTML report is suitable for demonstrations, release reviews, and team sharing.

Versioned Evaluation Datasets

AgenticLens can also store evaluation samples as a versioned dataset artifact that carries split assignments, tags, and human labels for judge calibration.

Summarize a dataset:

agenticlens dataset summary dataset.json

Assign deterministic train, validation, and test splits:

agenticlens dataset split dataset.json \
  --save dataset-split.json \
  --train-ratio 0.7 \
  --validation-ratio 0.15 \
  --test-ratio 0.15 \
  --seed 7

Export a split back to a plain samples.json artifact for evaluation:

agenticlens dataset export-samples dataset-split.json \
  --split test \
  --save samples-test.json

See examples/dataset_and_calibration_demo.py for a complete local example that creates a dataset from samples, assigns splits, runs evaluation, and calibrates judge labels end to end.

Run a Trusted Live Target

evaluate-live runs the same suite against a trusted live target instead of a pre-recorded sample file.

Run a Python callable:

agenticlens evaluate-live suite.yaml \
  --target-kind python \
  --target examples/live_evaluation_demo.py:run_case \
  --save evaluation-live.json

Run an HTTP target:

agenticlens evaluate-live suite.yaml \
  --target-kind http \
  --target http://localhost:8000/evaluate \
  --save evaluation-live.json

Live targets are intentionally powerful developer-facing integrations. Python targets execute local code and HTTP targets can reach arbitrary URLs, so suite files and target definitions should be treated as trusted inputs.

Example Advanced Checks

cases:
  - id: support-answer
    name: Structured support answer
    input:
      question: Where is my refund?
    required_tools: ["lookup_refund"]
    required_tool_arguments:
      lookup_refund: ["order_id"]
    output_json_schema:
      type: object
      required: ["answer", "meta"]
      properties:
        answer:
          type: string
        meta:
          type: object
          required: ["confidence"]
    required_output_fields:
      - meta.confidence
    max_turns: 3
    max_latency_ms: 1200
    max_cost_usd: 0.02

Apply a Release Gate

agenticlens gate evaluation.json \
  --min-pass-rate 0.95 \
  --min-average-score 0.98 \
  --max-failed-cases 1 \
  --max-average-latency-ms 1500 \
  --max-total-cost-usd 0.25

The command exits with status 0 when every threshold passes, 2 when the release gate fails, and 1 when the report or configuration is invalid.

max_turns requires trace.metadata.turn_count to be recorded as a positive integer. AgenticLens does not infer conversational turns from lower-level span types when that metadata is absent.

Judge Calibration

When a dataset includes human labels for a judge score, AgenticLens can compare the judge's output against those labels and report:

  • mean judge score and mean expected score
  • mean absolute error and root mean squared error
  • pass/fail agreement and verdict agreement
  • statistical confidence intervals for mean and agreement metrics
agenticlens judge-calibrate evaluation.json dataset.json \
  --score-name answer_quality \
  --confidence-level 0.95 \
  --save calibration.json

See examples/experiment_runner_demo.py for a runnable multi-variant experiment manifest example using the same evaluation layer for repeated live trials.

Labels live alongside each dataset record and can include an expected score, expected pass/fail decision, expected verdict string, optional threshold, and free-form reviewer notes.

Offline Pitch Demonstration

The repository includes a deterministic LangGraph supervisor demonstration that produces a trace, evaluation JSON, release decision, and HTML report:

uv sync --extra langgraph
uv run python -m examples.pitch_demo.run_pitch_demo

Generated artifacts are written to examples/pitch_demo/artifacts/. The demo performs real graph execution but does not require an API key or network access.

Scope

The framework supports deterministic checks, synchronous custom or model-based evaluators, versioned local datasets, and judge calibration against human labels. Built-in provider clients, asynchronous and batched evaluation, and automatic framework event adapters remain future roadmap capabilities.