Traditional software testing is built around a reassuring idea: when a system receives a known input under known conditions, the expected output can usually be defined in advance.
Enter a valid username and password, and access should be granted. Submit an order with an expired payment method, and the transaction should be rejected. Call an API with a missing mandatory field, and the service should return the expected validation response. Calculate tax using a defined rate and taxable amount, and the result should match the rule.
The implementation may be complex, but the testing contract is relatively clear. The tester controls the input, observes the output, and compares the actual result with an expected result.
AI systems weaken this contract.
Ask a large language model the same question several times and it may produce different answers. Ask an AI agent to complete the same task twice and it may choose different tools, follow different reasoning paths, recover differently from an error, or complete the task successfully through entirely different sequences of actions. A recommendation system may change its output as user context evolves. A generative application may produce two answers that are worded differently but are equally acceptable, while a third answer appears fluent yet contains a serious factual error.
The problem is not simply that AI systems are unpredictable.
The deeper problem is that correctness itself becomes harder to define, observe, and measure.
This creates a fundamental challenge for quality engineering. Traditional software testing asks whether the actual output equals the expected output. Testing non-deterministic AI systems often requires a more difficult question:
Is this range of behavior acceptable, reliable, safe, and useful across repeated interactions and changing conditions?
That is not a small adjustment to conventional testing practice. It requires a different model of quality.
The Test Oracle Problem Becomes Much Harder
Every software test depends, explicitly or implicitly, on a test oracle.
A test oracle is the mechanism used to determine whether a test has passed or failed. In conventional software, this can be straightforward. A calculation should produce a specific number. A database operation should create a particular record. A request should return a defined status code. A business rule should allow or reject a transaction.
For many AI systems, there is no single correct output.
Consider a customer-support assistant asked:
My order arrived damaged. What should I do?
One response may apologize, explain the return process, and offer a replacement. Another may ask for the order number before explaining the next steps. A third may direct the customer to a returns page while offering to help further.
All three may be acceptable.
A fourth response may confidently invent a refund policy that does not exist.
A fifth may provide the correct policy but use language that is inappropriate for the organization’s customer-service standards.
A sixth may give a good answer but expose internal information retrieved from a source the customer was not authorized to access.
A simple exact-match assertion cannot distinguish meaningfully between these outcomes.
This is the central testing problem of generative AI: the space of acceptable outputs may be broad, while the space of unacceptable outputs is also broad and difficult to enumerate.
Research on testing LLM-based applications has identified this combination of non-determinism, ambiguous inputs, and multiple potentially valid outputs as a major challenge to conventional software-testing assumptions.
The consequence is important. AI testing cannot depend entirely on predetermined expected strings or fixed output values. Quality teams need to define properties of acceptable behavior.
The question changes from:
Did the system produce this exact answer?
to:
Did the system remain within the boundaries of acceptable behavior?
Those boundaries may include factual accuracy, relevance, completeness, groundedness, safety, policy compliance, tone, latency, cost, tool-use correctness, data privacy, and successful task completion.
An AI response can pass one dimension while failing another.
That makes AI quality inherently multidimensional.
Non-Determinism Does Not Mean “Anything Is Acceptable”
There is a dangerous misunderstanding around AI testing.
Because generative AI can produce different valid outputs, some teams treat unpredictability as an excuse for weak acceptance criteria. If the answer looks reasonable during a demonstration, the feature is considered ready.
That is not testing.
Non-determinism changes the unit of evaluation. It does not remove the need for rigor.
A weather application powered by a language model may phrase its answer differently each time, but it should not invent a storm warning. A financial assistant may explain a calculation in different ways, but it should not change the underlying arithmetic. A support agent may use different language, but it should not create a return policy. A retrieval-augmented generation system may summarize internal documents differently, but its factual claims should remain supported by authorized source material.
The output can vary while important properties remain invariant.
This distinction is one of the foundations of effective AI testing.
Instead of defining one expected sentence, a quality team can define requirements such as:
The answer must not contradict the approved policy source.
The answer must not claim that an action has been completed unless the corresponding tool call succeeded.
The system must not reveal information from unauthorized documents.
The response must acknowledge uncertainty when supporting evidence is unavailable.
The agent must obtain confirmation before performing an irreversible action.
The output may change. The quality boundaries should not.
This is where traditional QA knowledge remains highly relevant. Boundary analysis, risk-based testing, state-transition testing, exploratory testing, negative testing, security testing, and production monitoring do not become obsolete. They need to be adapted to a system in which behavior is probabilistic and quality cannot always be represented by a binary assertion.
A Single Successful Run Proves Very Little
Imagine testing a conventional login function.
A valid credential pair is submitted. Authentication succeeds. The same test is executed again tomorrow. Assuming the code and dependencies have not changed, the expected behavior remains the same.
Now imagine testing an AI agent responsible for resolving a support request.
During the first run, the agent correctly identifies the customer’s problem, retrieves the relevant policy, asks for missing information, and recommends the correct resolution.
The test passes.
Run the same scenario again and the agent may retrieve a different document. It may skip a clarification question. It may choose another tool. It may interpret an ambiguous phrase differently. It may reach the same correct outcome through a different route.
Or it may fail.
This means that a single successful execution is weak evidence.
The relevant quality property may be a distribution of outcomes, not one outcome.
For example, suppose an agent succeeds in 97 of 100 executions of a business-critical scenario. Whether that is acceptable depends on the use case. For a low-risk content-tagging assistant, the performance may be sufficient. For an agent that initiates financial transactions, a 3% failure rate may be completely unacceptable.
The testing question therefore becomes statistical as well as functional.
How frequently does the system succeed?
How severe are the failures?
Are failures concentrated around particular input types?
Does performance deteriorate over longer conversations?
Does a small change in wording produce a disproportionate change in behavior?
Does success depend heavily on one model version?
What happens to the failure distribution after a prompt change?
How stable is the system across repeated trials?
This is why repeated evaluation is essential for non-deterministic systems. Current practical guidance for evaluating production AI agents explicitly distinguishes their evaluation from traditional deterministic software testing and emphasizes repeated trials, scenario-based evaluation, and multidimensional metrics rather than relying on a single pass or fail.
Quality engineering must therefore move beyond the idea that one test case equals one execution.
For important AI behaviors, one test case may need to represent a scenario executed many times, with the resulting behavior analyzed as a distribution.
The Same Input Is Not Always the Same Test
There is another complication.
In AI applications, seemingly minor variations in input can expose major differences in behavior.
Consider these requests:
Cancel my subscription.
I don’t want this anymore.
Please stop charging me from next month.
Can you close the plan after my current billing period?
I was charged again. I told you I wanted out.
A human support professional understands that these statements belong to a related problem space, but they are not semantically identical. They may require different actions depending on policy and account state.
An AI system must interpret language, infer intent, retrieve relevant information, maintain context, and select an action.
Testing only one canonical phrase creates a false sense of coverage.
Traditional equivalence partitioning remains useful here, but the partitions are semantic rather than purely numeric or structural. Test designers need to consider paraphrases, incomplete requests, contradictory information, emotional language, spelling errors, multilingual inputs, indirect instructions, ambiguous intent, and attempts to manipulate the system.
A robust evaluation dataset should therefore contain more than clean benchmark prompts.
It should contain the messiness of actual use.
This is particularly important because benchmark performance can create an incomplete picture of production reliability. Benchmark research has documented problems including bias, implementation inconsistency, evaluator variation, prompt sensitivity, and difficulty establishing whether a benchmark measures the capability it claims to measure.
A model can perform strongly on a benchmark and still fail in a particular organization’s workflow.
The organization is not deploying a benchmark score.
It is deploying a system into an environment.
Model Testing and System Testing Are Not the Same Thing
One of the most important distinctions in AI quality engineering is the difference between evaluating a model and testing an AI system.
A model may have excellent benchmark performance.
The application built around it may still be unreliable.
Consider a retrieval-augmented generation application. The final answer depends on more than the language model. It may depend on document ingestion, text extraction, chunking strategy, metadata quality, embedding generation, vector retrieval, reranking, access control, prompt construction, context-window management, model behavior, citation generation, and post-processing.
A factual error in the final response can originate from several places.
The correct document may never have been ingested.
The document may have been parsed incorrectly.
The relevant passage may have been divided into poor chunks.
The retriever may have returned irrelevant context.
The reranker may have selected the wrong source.
The prompt may have encouraged the model to answer despite insufficient evidence.
The model may have ignored the retrieved evidence.
The citation layer may have attached the wrong source.
Testing only the model’s answer misses the system.
The same problem becomes even more pronounced with AI agents.
An agentic system may include a language model, planning logic, memory, external tools, APIs, databases, authorization controls, retry mechanisms, and environmental state. A failure can occur because the model selected the wrong tool, used the correct tool with incorrect arguments, misunderstood the tool response, repeated an action, failed to recover from a timeout, or claimed success after the underlying action failed.
An apparently good final answer can therefore hide a failed process.
Imagine an agent responding:
Your appointment has been rescheduled to Tuesday at 10:00 AM.
The sentence is linguistically perfect.
But did the booking API succeed?
Did the agent modify the correct appointment?
Was the requested time actually available?
Did the system receive confirmation from the external service?
Did it accidentally execute the operation twice after a timeout?
A response-quality evaluator looking only at the text may give the answer a high score while missing a serious transactional failure.
This is why AI testing must inspect both outcomes and trajectories.
The outcome is what happened.
The trajectory is how the system got there.
For agentic systems, both matter.
Accuracy Is Only One Dimension of AI Quality
Teams often begin AI evaluation by asking whether responses are accurate.
That is necessary, but insufficient.
A system can be factually accurate and still fail its users.
Consider an internal knowledge assistant. It may produce factually correct responses but take 40 seconds to answer. It may cite inaccessible documents. It may reveal information the requesting employee is not authorized to see. It may answer confidently when the knowledge base contains conflicting policies. It may provide a technically correct answer that is irrelevant to the user’s role or location.
Quality is contextual.
A useful evaluation framework should begin with the actual risk and purpose of the system.
For a document assistant, important dimensions might include groundedness, retrieval relevance, citation correctness, completeness, access-control compliance, and abstention quality.
For a coding assistant, the dimensions may include functional correctness, security, maintainability, dependency suitability, architectural consistency, and developer comprehension.
For a customer-service agent, the dimensions may include resolution accuracy, policy compliance, empathy, escalation behavior, tool-use success, and avoidance of unauthorized commitments.
For an autonomous workflow agent, task completion alone is not enough. The evaluation may need to examine planning quality, tool selection, argument correctness, recovery behavior, side effects, latency, and cost.
The Generative AI Profile published under the AI Risk Management Framework by the U.S. National Institute of Standards and Technology treats generative AI risk as a lifecycle management problem rather than something resolved through a single accuracy score, emphasizing the need to identify, measure, manage, and govern risks according to context and organizational priorities.
This is the right direction for quality engineering.
There is no universal AI quality score that answers every deployment question.
Quality must be decomposed into measurable properties relevant to the system’s actual function and risk.
The Evaluation Dataset Is Part of the Product
In conventional testing, test data is important.
In AI systems, the evaluation dataset becomes strategically important.
A team cannot understand system quality if its test scenarios do not represent the situations the system will encounter.
Suppose an organization launches an HR assistant and evaluates it using 200 clean questions written by the project team.
The questions may cover common policies. The answers may score well.
Then real employees begin using the system.
They ask questions containing incomplete information.
They refer to policies using informal names.
They combine multiple questions in one message.
They ask follow-up questions that depend on previous context.
They challenge the answer.
They ask about exceptions.
They use regional terminology.
They provide incorrect assumptions in the question.
They ask about a policy that changed last month.
They request information they are not authorized to access.
The production environment exposes a completely different behavioral surface from the original evaluation set.
A strong evaluation dataset should therefore evolve continuously.
It can begin with requirements and known use cases, but it should grow from production observations, support escalations, failure investigations, adversarial testing, domain-expert input, and newly discovered edge cases.
Every meaningful production failure should trigger a question:
Should this become a permanent evaluation scenario?
This is the AI equivalent of adding a regression test after discovering a defect, but the implementation is more complex because the scenario may need repeated execution and multidimensional scoring.
The evaluation suite becomes a form of institutional memory.
It records what the organization has learned about how the system can fail.
Why Exact-Match Testing Fails for Open-Ended Outputs
Exact matching remains appropriate for deterministic components inside AI systems.
If an agent calls a calculator, the arithmetic result can be checked exactly.
If an API should return a specific status code, conventional assertions remain appropriate.
If a tool call requires a particular identifier, the argument can be validated.
If a structured output must conform to a schema, schema validation should be deterministic.
The mistake is not using traditional tests.
The mistake is trying to use them for every layer of the system.
Open-ended language requires different evaluation methods.
Some outputs can be checked through rule-based methods. Required terms can be detected. Prohibited patterns can be identified. Citation formats can be validated. Structured fields can be inspected. Claims can be compared with retrieved evidence.
Other outputs require semantic evaluation.
Two answers may use entirely different wording while communicating the same correct information. Embedding similarity, natural language inference, task-specific classifiers, or model-based judges may help evaluate these cases.
Still other situations require human judgment.
A legal expert may need to determine whether an answer preserves an important nuance. A clinician may need to evaluate whether a response omits a critical consideration. A brand specialist may need to assess tone. A domain owner may need to decide whether an answer reflects actual organizational practice rather than merely plausible language.
The right evaluation architecture often combines deterministic checks, automated semantic evaluation, model-based assessment, and human review.
No single evaluator should be treated as universally authoritative.
LLM-as-a-Judge Is Useful, but It Is Not an Oracle
Using one language model to evaluate the output of another has become a common approach.
The attraction is obvious.
Human evaluation is expensive and difficult to scale. A model-based judge can evaluate large numbers of responses against a rubric, compare alternative outputs, classify failure types, and produce explanations.
This can be valuable.
But replacing one difficult-to-evaluate model with another model does not eliminate the evaluation problem.
The judge can have biases.
It can be sensitive to prompt wording.
It can prefer certain response styles.
It can be inconsistent.
It can fail to recognize domain-specific errors.
It may reward persuasive language even when the underlying answer is weak.
It may share blind spots with the system being evaluated.
Recent research on LLM-based software-engineering tools identifies the limitations of automated and model-based evaluation, alongside non-deterministic evaluation instability and the absence of stable ground truth, as major unresolved challenges.
The practical response is calibration.
Teams should compare model-based judgments against expert human judgments on a carefully selected sample. Disagreement should be investigated rather than averaged away.
If the AI judge consistently misses a particular failure class, the evaluation design needs another mechanism for detecting it.
One recent case study of a multi-agent system illustrates the value of combining evaluation modes. The reported framework used quality gates across task success, context preservation, latency, safety, and evidence coverage. Human calibration and model-based judging surfaced complementary failure modes rather than producing identical assessments.
The broader principle is more important than any single framework:
Evaluation methods should overlap where failure is costly.
Testing AI Agents Requires Testing the Journey, Not Just the Destination
AI agents create a new level of complexity because they do not merely generate content.
They act.
An agent may search a database, call an API, send a message, update a record, create a ticket, schedule an appointment, modify a configuration, or trigger another workflow.
The final response is only the visible surface.
Suppose an expense-management agent is asked to prepare a reimbursement claim.
The task appears successful because the final form is correct.
But during execution, the agent may have queried an unauthorized data source, exposed sensitive information in logs, made unnecessary tool calls, selected an outdated policy, retried an operation that should not have been retried, or consumed ten times the expected computational cost.
Outcome-only evaluation would miss these problems.
Agent testing therefore needs several levels of observation.
The team must examine whether the goal was completed, whether the correct tools were selected, whether arguments were valid, whether actions occurred in an acceptable sequence, whether permissions were respected, whether failures were handled safely, and whether the final state of the external environment is correct.
This resembles distributed-systems testing as much as chatbot testing.
The agent is not merely producing language.
It is participating in a stateful environment.
Metamorphic Testing Can Help When Exact Answers Do Not Exist
One useful approach for non-deterministic systems is metamorphic testing.
The basic idea is to test relationships between inputs and outputs rather than requiring one exact expected answer.
Suppose an AI system summarizes insurance policies.
A tester may not be able to define the perfect summary for every document. But certain relationships should hold.
Adding irrelevant formatting to the source document should not change the policy meaning reported by the system.
Rephrasing the user’s question without changing intent should not reverse the answer.
Removing the evidence supporting a claim should reduce confidence or cause the system to abstain rather than invent an answer.
Changing a policy date should cause the system to select the appropriate version.
Replacing a customer identifier should not expose information from the original customer.
These are testable relationships.
Metamorphic testing is valuable because many AI failures appear not as obviously wrong answers, but as unjustified sensitivity or insensitivity.
A system may be too sensitive to superficial wording changes.
Or it may be insufficiently sensitive to a critical contextual change.
Both are quality problems.
AI Testing Needs Adversarial Thinking
Conventional functional testing often focuses on expected use.
AI testing must devote significant attention to unexpected use.
Users will phrase requests in ways the development team did not anticipate. Retrieved documents may contain misleading instructions. Attackers may attempt prompt injection. Long conversations may create conflicting instructions. Tools may return malformed data. External services may time out after partially completing an operation.
The system may also face ordinary user behavior that looks adversarial from a testing perspective.
People contradict themselves.
They change their minds.
They provide incomplete information.
They assume things that are false.
They ask the system to do something outside its authority.
They refer to previous conversations vaguely.
They paste text containing instructions.
They mix data with commands.
The boundary between functional testing, security testing, and safety evaluation becomes less clear in AI applications.
NIST’s work on adversarial machine learning reflects the breadth of this threat landscape by providing a structured taxonomy of attacks and mitigations across AI system contexts.
For quality teams, the implication is straightforward.
Adversarial testing cannot be reserved for a final security review.
It needs to influence the design of the evaluation suite from the beginning.
Regression Testing Changes When the Model Can Change Underneath You
Traditional regression testing asks whether a software change broke existing behavior.
AI systems introduce more sources of change.
The application prompt may change.
The retrieval pipeline may change.
The embedding model may change.
The underlying foundation model may be upgraded.
A provider may alter model behavior behind a stable API.
Tool descriptions may change.
Knowledge sources may be updated.
Safety policies may be modified.
Memory logic may change.
A seemingly small adjustment can improve one group of scenarios while degrading another.
This means AI regression testing should be comparative.
The team should not ask only whether the new version passes.
It should ask where behavior improved, where it deteriorated, how much variance changed, which failure categories appeared, and whether the trade-off is acceptable.
A new prompt that raises average helpfulness from 84% to 88% may appear to be an improvement.
But if the same change reduces policy compliance from 99.5% to 96%, the release decision looks very different.
Average quality can hide critical regression.
This is why release gates for AI systems should be multidimensional and risk-sensitive.
Production Monitoring Is Part of the Test Strategy
No evaluation suite can represent the entire production environment.
Users will discover new interactions.
Data will change.
Language will shift.
Attack patterns will evolve.
Business policies will be updated.
External tools will fail in new ways.
Model behavior may drift after changes.
For these reasons, production monitoring is not merely an operations responsibility for AI systems.
It is part of quality engineering.
Teams need mechanisms to identify low-confidence behavior, failed tool calls, unusual refusal rates, unsupported claims, retrieval failures, policy violations, user corrections, escalation patterns, cost anomalies, and sudden changes in outcome distributions.
Human feedback can be useful, but it should not be treated as a complete quality signal.
Users often fail to report incorrect answers when those answers sound convincing.
A thumbs-up button does not prove factual accuracy.
A thumbs-down response may indicate that the answer is correct but inconvenient.
User satisfaction and system correctness overlap, but they are not identical.
Production monitoring should therefore combine behavioral metrics, technical telemetry, business outcomes, safety indicators, sampled human review, and feedback.
The purpose is not simply to watch dashboards.
The purpose is to convert production experience into better evaluation.
A mature AI quality loop should work in both directions.
Pre-release evaluation informs deployment.
Production evidence improves the evaluation suite.
A Better Quality Model for Non-Deterministic AI
The most useful shift for QA leaders is to stop treating AI testing as a search for a perfect replacement for the assertion.
There is no single replacement.
A robust AI quality system is layered.
Deterministic components should still have deterministic tests.
Business invariants should be explicitly checked.
Open-ended outputs should be evaluated against defined quality dimensions.
Important scenarios should be executed repeatedly.
Variations of semantically equivalent inputs should be explored.
Agent trajectories should be inspected, not merely final answers.
Model-based judges should be calibrated against human experts.
High-risk failures should have independent detection mechanisms.
Production behavior should feed back into the evaluation dataset.
Release decisions should consider distributions and failure severity, not only average scores.
This is more complicated than conventional pass-or-fail testing.
But the complexity comes from the system, not from the testing methodology.
Pretending that an AI system is deterministic does not make it easier to test.
It only makes the test results less meaningful.
The Future of QA Is the Engineering of Confidence
AI does not eliminate the fundamental purpose of testing.
It makes that purpose clearer.
Testing was never really about writing test cases.
It was about producing evidence that a system could be trusted for a particular purpose under particular conditions.
Deterministic software allowed much of that evidence to be expressed through fixed inputs, expected outputs, and binary assertions.
AI systems require a broader evidence model.
Quality engineers must reason about probability, distributions, semantic equivalence, system trajectories, failure severity, human judgment, model behavior, data quality, security boundaries, and production feedback.
The profession is therefore moving from test execution toward something more demanding: the engineering of confidence under uncertainty.
That shift will require new tools, but tools are only part of the answer.
The deeper requirement is better judgment.
Teams need to decide which behaviors must be invariant, which variations are acceptable, which failures are tolerable, which scenarios deserve repeated evaluation, when human expertise is necessary, and what evidence is strong enough to justify deployment.
A system does not become trustworthy because it passed a benchmark.
It does not become trustworthy because a demonstration worked.
It does not become trustworthy because the average evaluation score increased.
And it does not become trustworthy because the same AI system that produced an answer can explain why the answer is good.
Trustworthy AI systems require multiple forms of evidence, gathered across development, evaluation, deployment, and production.
Traditional QA is not disappearing.
Its central question remains exactly the same:
What evidence do we have that this system will behave acceptably when it matters?
For non-deterministic AI, answering that question is simply much harder.
And that is precisely why quality engineering matters more than ever.