Insight / Blog

AI Agent Series · 3/6

Four questions to answer before deployment

A customer asks an AI Agent to cancel a flight reservation.

The agent checks the policy and explains that the booking is not eligible for cancellation or a refund. The answer is clear. The reason is correct. Nothing important appears to be missing.

But the evaluation cannot stop there.

The team still needs to ask:

1. Does the answer match the actual system state? If the agent says the reservation was not cancelled, the booking must still exist.
2. Did the agent use only the permitted tools and procedures? A cancellation API call would be a serious failure in a read-only case.
3. Would the agent reach the same conclusion again under the same conditions? One correct run does not prove reliable behavior.

The final sentence is only one part of the evidence.

Teams also need to examine the outcome, the path the agent took, and whether the behavior remains stable across repeated trials.

Why AI Agent Evaluation Is Different

Traditional language-model evaluation often focuses on a single response.

Is the answer relevant? Is it factually correct? Is the writing clear and natural? Those are useful questions when the model's job ends with text generation.

An AI Agent does more than produce text.

It calls tools, reads customer records, modifies files or databases, and sometimes stops because a policy or permission boundary has been reached.

That changes the object of evaluation.

The response is what the agent tells the user.

The trajectory is the record of tool calls, intermediate decisions, and results produced during the run.

The outcome is the final state of the environment after the work ends.

A flight-booking agent may say that a reservation was cancelled. The outcome is whether the reservation is actually gone from the booking system. A support agent may claim that a refund was issued. The outcome is whether the transaction record changed correctly and only once.

Response quality remains important. It is simply no longer enough.

Agent evaluation has to connect what the system said, what it did, and what state it left behind.

Four Questions to Ask Before Deployment

For a work-facing AI Agent, four evaluation questions should be considered together.

1. Did the work actually finish?

This is the outcome question.

Did the agent complete the requested task? Is the final artifact correct? Does the business system reflect the intended state? Did the work finish within an acceptable time and cost range?

A natural answer cannot compensate for an incomplete or incorrect outcome.

2. Did the agent use the right capabilities efficiently?

This is about tools and procedure.

Did the agent select the right tool? Did it follow the required sequence? Did it repeat unnecessary API calls, search in the wrong system, or take a costly route when a simpler one was available?

The point is not to force one rigid path when several valid paths exist. It is to identify dangerous, wasteful, or clearly ineffective behavior.

3. Is the behavior reliable across repeated runs?

Agent behavior varies.

The same task can succeed once and fail the next time. A useful evaluation therefore runs multiple trials and examines consistency, not only the best result.

This matters most in customer-facing and operational workflows, where users expect the agent to follow the same policy every time.

4. Does the agent respect safety and policy boundaries?

The agent must stay within permissions, privacy rules, approval requirements, and security policy.

The evaluation should include cases where the agent must refuse, stop, request approval, or resist an instruction that conflicts with trusted policy.

These four questions prevent a common mistake: treating one impressive answer as proof that the whole system is ready.

A Good Evaluation Cannot Be Reduced to One Score

The evaluation environment matters as much as the questions.

First, teams may need a user simulator.

Real users do not always provide complete information at the start. They change their minds, answer ambiguously, and introduce new constraints in the middle of a task. A scripted prompt cannot reproduce every dynamic conversation. For conversational agents, a second model can simulate user behavior, but those simulations still need review and calibration.

Second, teams need the right grader for each type of evidence.

Objective state changes should be checked directly. Did a database value change? Was a file created? Was a message sent? Code-based or system-level checks are usually stronger than asking another model to guess.

Qualitative questions require a different approach. Was the policy explanation sufficient? Was the refusal reason appropriate? Did the interaction remain clear? A rubric, a model-based grader, or human review may help, but subjective grading should be calibrated against domain experts.

Third, teams need the right execution environment.

An agent that can modify files, update records, or send messages should not begin evaluation in a live production system. A sandbox or isolated test environment allows repeated trials without exposing real users or business data to avoidable risk.

No single grader or environment catches every failure.

Good evaluation design combines direct state checks, trace review, qualitative rubrics, repeated trials, and production monitoring according to the risk of the task.

Benchmarks Are Design Inputs, Not Proof of Business Readiness

Benchmarks such as GAIA, Tool Decathlon, OSWorld, and tau-bench can be useful.

But the most valuable question is not which system ranks first.

It is what kind of failure each benchmark was designed to expose.

Your organization needs to define the failures that matter in its own work.

Is the greatest risk a wrong customer explanation? An incorrect data change? An action taken without approval? A failure that appears only under a specific exception? A tool call that succeeds but leaves the workflow in an inconsistent state?

Once those failure modes are clear, teams can build an evaluation harness around real tasks, real policies, and realistic system states.

A public benchmark can inspire the design. It cannot replace the organization's acceptance criteria.

Every Failure Should Become a Future Test

Episode 2 focused on the harness around the model: tools, permissions, state, logs, recovery, and evaluation.

Evaluation is how teams learn whether that harness is actually improving.

A model changes. A prompt is revised. A new tool is added. A permission rule is tightened. The final answer may sound better while a hidden failure becomes more frequent.

That is why a production failure should not remain only an error log.

It should become a regression test.

If the agent once called a cancellation tool in a read-only case, that exact scenario should be added to the evaluation suite. If it lost context after a handoff, the next release should prove that the failure does not return. If a new model increases task completion but also increases policy violations, the tradeoff should be visible before deployment.

As AI Agents become real work systems, evaluation moves from text-quality review to operational risk management.

The question is no longer only:

Does the answer sound right?

It is:

Did the agent complete the right work, through an acceptable process, with a reliable and policy-compliant result?

When teams can answer that with evidence, they are better prepared to control review cost, recovery cost, and business risk.

The next article will examine another source of operational complexity: memory.

An agent that works across sessions must decide what to keep, what to retrieve, what to forget, and what may be shown to whom. That is why AI Agent memory should be treated as an operating policy, not simply a place to store more data.

AI Agent Series