Most companies will test an AI agent like this:
- Give it a task.
- Read the output.
- Decide whether it looks reasonable.
- Put it into a larger workflow.
That is not a test. It is an opinion.
A plausible answer can still contain the wrong customer record, an invalid accounting treatment, an unsupported sales claim or an action that should never have been taken. The output may look polished while the underlying work is wrong.
Before you add another agent, define what PASS and FAIL mean.
Visible progress is not the same as correct work
Maurice Heumann recently documented a multi-month effort to use autonomous agents to reconstruct a first-person shooter. The early setup produced visible progress, but the project still contained incorrect function signatures, types, memory layouts and invented logic. A reviewer agent did not solve the problem because workers could sometimes persuade it to accept deviations.[1]
The turning point was not simply adding another model. The team introduced a machine-checkable oracle: byte-level comparison against the original program. The system could now return an objective result instead of relying on whether the code looked convincing.
Heumann reports that the project eventually reached 99% function coverage and 83% byte-exact output. Those figures are author-reported, not independently audited, but the operating lesson is useful: once correctness became measurable, the team had a feedback loop that agents could work against.[1]
Business workflows rarely have a byte-for-byte answer. They can still have explicit acceptance conditions.
Your domain expert should design the test
The most valuable AI architect may not be the person who writes the longest prompt.
It may be the finance manager who knows why a reconciliation difference is acceptable in one ledger but not another. Or the sales operations lead who knows which evidence must exist before a CRM opportunity can move stage. Or the service manager who can distinguish a routine exception from one that needs customer contact.
That judgment is operating knowledge. It becomes useful to an AI workforce when it is converted into something the system can check.
For an invoice reconciliation agent, PASS might require:
- supplier, purchase order and invoice identities to match;
- the amount difference to remain inside a stated tolerance;
- tax treatment to follow the approved rule table;
- duplicate-invoice checks to return clear;
- any bank-detail change to enter a human review queue.
FAIL should be just as specific:
- no purchase order exists;
- the supplier record is inactive;
- the invoice appears twice;
- the agent cannot explain the mismatch using approved evidence;
- the proposed action would change master data or release payment without authorization.
Now the agent is not being judged on whether its explanation sounds intelligent. It is being judged on whether the work meets the business definition of correct.
Build four things before you automate
Choose one recurring workflow. Before selecting models or drawing a multi-agent diagram, create these four artifacts.
1. Acceptance conditions
Write the conditions that must be true before the output can move forward.
Keep them observable. “The CRM record is high quality” is too vague. “The record contains a verified work email, source, consent status, account owner and evidence for the selected industry category” can be checked.
2. Rejection rules
List the conditions that should stop the workflow immediately.
This is where domain experts protect the business from plausible-looking wrong work. A finance agent should not invent a missing cost centre. A sales agent should not manufacture qualification evidence. A service agent should not close a case when the customer has disputed the resolution.
A rejected output is not a failed system. It is proof that the system noticed a boundary.
3. Known-answer tests
Collect real examples with outcomes your team already agrees on.
Include normal cases, edge cases and cases that should stop. Remove private information before using them in a test environment. Then run the same set whenever the instructions, tools, data source or model changes.
You are not looking for a good demo. You are looking for repeatable results.
4. A failure path
Define what happens when the agent cannot pass.
The answer should not be “try harder until something works.” Route the case to a named queue with the evidence gathered, the failed condition and the proposed next action. A human can then decide whether to correct the input, update the rule or handle the exception manually.
This turns failure into usable operational data.
Better tests can change model economics
Stronger models still matter. Some work genuinely requires deeper reasoning, better tool use or more reliable judgment.
But model choice is only one part of the economics.
When a task is bounded and the system can grade the result precisely, a cheaper model may be useful for more attempts. A stronger model or human expert can handle difficult cases, design the acceptance logic and review exceptions. The right model mix depends on risk, complexity and the quality of the feedback loop.
Without a reliable test, using a cheaper model may create rework. Using a frontier model may only create more convincing rework.
The business advantage comes from knowing the difference.
Start with one workflow this week
Do not begin with an enterprise-wide agent strategy.
Pick one recurring job in finance, sales, operations or customer service. Ask the person who currently owns the outcome to write:
- five conditions for PASS;
- five conditions for automatic rejection;
- ten known-answer cases;
- the evidence every result must include;
- the person or queue that owns failures.
Then test the agent against that pack.
If the team cannot agree on the expected outcome, the workflow is not ready for more autonomy. Fix the process definition first.
Building agents is becoming easier. Defining correct work is still hard.
That is why domain experts matter more, not less. They are the people who can turn judgment into a test, a test into a reusable skill, and a reusable skill into trusted execution.
Orchestrate, don’t operate. But first, define what good work looks like.
Sources
[1] https://momo5502.com/posts/2026-10-09-game-decompilation/ — 500+ Billion Tokens Later: Letting AI Agents Decompile A First-Person Shooter