From Trust to Continuous Assurance in Agentic AI

Artificial intelligence is moving from systems that generate outputs to systems that can increasingly take action.

A traditional AI system might answer a question, classify an image, recommend an action, or generate code. A human then decides what to do with that output.

Agentic AI changes this model. AI agents can increasingly reason about goals, plan tasks, use tools, interact with other systems, and take actions with limited human intervention.

The question then becomes not only what AI can do, but how we manage what we did not explicitly program it to do.

This raises a question that I believe will become increasingly important:

How do we continuously know that an autonomous AI system is behaving as intended after it has been deployed?

The limits of one-time assurance

Traditional AI assurance often follows a familiar pattern:

Test → Approve → Deploy

But an autonomous agent operates in a changing environment. Its inputs, context, tools, permissions, users and surrounding systems can change over time.

NIST's recent work on monitoring deployed AI systems highlights this challenge and the growing importance of post-deployment monitoring.

For agentic AI, I think we need to move toward:

Test → Deploy → Observe → Evaluate → Intervene → Reassess

This is the idea of continuous assurance.

What does continuous assurance mean?

At a basic level, continuous assurance means continuously gathering evidence that an AI system is:

operating within its intended objectives;
using its authorized tools and permissions;
behaving within defined safety and security boundaries; and
producing outcomes that remain consistent with what we intended.

This is particularly important for AI agents because we are not only evaluating an output. We are evaluating a sequence of decisions and actions.

Example:

Suppose a bank deploys an AI agent to investigate potentially fraudulent transactions. A customer makes a large purchase from a new device, shortly after their account was accessed from another country. The agent is given a simple objective: investigate the transaction and take appropriate action if fraud is suspected.

The agent reviews the customer's transaction history, checks recent login activity, examines device information, and compares the transaction against other fraud indicators. Based on the evidence available to it, the agent determines that the transaction is suspicious. It temporarily blocks the transaction, places a hold on the account, and alerts the fraud team.

At first, this looks like a successful use of autonomous AI.

But imagine that the customer contacts the bank and explains that the transaction was legitimate. They were travelling and had recently replaced their phone. The bank now needs to understand how the agent reached its decision.

It is not enough to know that the transaction was blocked. The bank needs evidence showing what information the agent considered, what decisions it made, what actions it took, and whether those actions were within its authority.

This is where continuous assurance becomes important.

The assurance process should allow the organization to reconstruct the agent's behavior from its objective and context through to its decision and eventual outcome. It should also make it possible to identify when the agent's behavior moves outside its expected boundaries.

From explaining outputs to understanding behavior

PwC's recent work on governance of agentic AI makes a related point: as agents become more autonomous, organizations need greater visibility into their objectives, actions, evidence, exceptions, and escalation paths.

This also changes how we think about transparency. With traditional AI, we often ask:

Why did the model produce this output?

With an autonomous agent, the question becomes broader:

Why did the agent behave this way?

The difference is important. The fraud agent did not simply classify a transaction as fraudulent. It gathered information, interpreted different signals, made a decision, used its authority, and took an action that affected a customer.

Understanding that sequence of behavior becomes part of assurance. The goal is not to capture every internal computation or generate thousands of logs. It is to preserve enough meaningful and trustworthy evidence to understand what happened.

This is where understandability becomes important.

Unlike Explainability that is concerned with making an AI system's outputs understandable. For instance, telling us which factors contributed to a fraud prediction; Understandability helps us make sense of an agent's behavior. It helps us understand how the agent moved from its objective to its actions.

In the fraud example, an understandable assurance record might show the agent's objective, the relevant context it received, the evidence it considered, the decision it made, the authority under which it acted, and the resulting outcome.

If we want to continuously assure autonomous systems, that broader understanding becomes essential.

A conceptual framework for continuous assurance

There is currently no single standard that completely defines continuous assurance for agentic AI. However, existing frameworks provide useful building blocks.

The NIST AI Risk Management Framework provides a foundation for managing AI risks across the system lifecycle. ISO/IEC 42001 provides an AI management-system approach, while ISO/IEC 23894 provides guidance for managing AI-specific risks. Security guidance such as OWASP's Agentic AI guidance addresses risks associated with autonomous agents.

Building on these ideas, I see continuous assurance for agentic AI as a cycle across six areas:

Define intent - Establish what the agent is expected to achieve, including objectives, constraints and conditions that require human intervention.
Establish authority - Define the agent's identity, permissions, tools and decision-making boundaries.
Observe behavior - Continuously collect meaningful evidence about the agent's context, decisions, tool use, actions and outcomes.
Evaluate - Assess whether the observed behavior remains consistent with the agent's objectives, policies, authority and acceptable risk.
Intervene - Provide mechanisms to pause, restrict, override or escalate agent actions when necessary.
Improve - Use incidents, unexpected behavior and operational evidence to improve the agent, its controls and its evaluation criteria.

The important point is that assurance is not simply about monitoring what an agent does. It is about maintaining a reliable connection between what the agent was intended to do and what it actually did. That connection is what allows us to detect deviations, investigate incidents and determine when human intervention is necessary.

From trust to continuous verification

As AI systems become more autonomous, I think we need to rethink what it means to trust them.

A pre-deployment assessment can tell us something about how a system performed under specific conditions. It cannot guarantee that the same system will behave appropriately in every future situation.

Continuous assurance provides a different approach: rather than treating trust as a one-time conclusion, we continuously gather evidence that the system remains within its intended objectives, authority and safety boundaries.

The goal is not to eliminate autonomy. It is to make autonomy observable, understandable, verifiable and controllable.

As AI moves from assisting humans to increasingly acting on their behalf, assurance needs to move with it.