AI Agent Observability: Keeping Humans in the Loop

AI agents now close tickets and trigger fixes in seconds. The organisations getting them into production have stopped trying to approve every action. They watch the agents closely enough to sign them off.

Key Takeaways

Enterprise AI has moved into stage two: agents doing real work on the organisation’s behalf. The barrier to scaling them is no longer capability. It is trust.
Agents fail quietly, on poor data, through drift or by acting confidently on a false positive. Monitoring that only asks whether a service is up will miss all three.
No person can approve every action taken at machine speed. The loop has to move: people set guardrails, review evidence and judge exceptions, and AI agent observability tells them when to step in.
Pilots stall at sign-off, not in the lab. Clear evidence of where data goes, what agents may do and how their behaviour is monitored is what gets them approved.

An AI agent picks up a tier-one support ticket, checks the user’s access history, resets a permission and closes the case. The whole exchange takes less than a minute, and the queue moves on. By every operational measure, that is a success. The harder question takes far longer to answer: whether the agent did the right thing, with the right data, for the right reason. Answering that quickly is what AI agent observability is for. It is fast becoming the condition for putting agents into production at all.

Most large Australian organisations have now run AI pilots, and many of those pilots worked. The pressure is to scale them. What stands in the way is rarely whether the technology works. It is whether anyone can show risk, legal and the board that it is working as intended.

Stage Two of AI Is Agents Doing Real Work

The first wave of enterprise AI was about individual productivity. Assistants drafted, summarised and searched for a person, who then decided what to do. The second wave is different in kind. Agents now sit inside platforms and do work on the organisation’s behalf: triaging tier-one support requests, gathering evidence for an incident, proposing a remediation. The best deployments augment people rather than replace them, taking repetitive legwork off skilled teams.

That shift changes the risk. When an assistant gets something wrong, a person usually catches it before it matters. When an agent gets something wrong, it may already have acted. Running AI at enterprise scale therefore depends on two things pilots rarely test properly. One is confidence in the data going into the model. The other is confidence in what the model does with it.

Agents Fail Quietly

An agent rarely fails with an outage. It fails quietly, in one of three ways. It acts on poor or stale data. Its behaviour drifts as the world around it changes. Or it acts with complete confidence on a false positive. None of these trips a conventional alert, because the service is up and the agent is doing exactly what it was built to do.

Data comes first. It does not matter how capable the model is if the data feeding it is wrong. ASD’s ACSC and international partners have published joint guidance on this. It recommends that organisations track data provenance, log the path data takes through an AI system, and monitor inputs and outputs to confirm the model is performing as expected (ASD’s ACSC, 2025). The same guidance treats drift as natural and expected. It starts as small, easily missed degradation and, left unchecked, becomes hard to reverse.

That is this edition’s theme in a new setting. As the opening article, The Network Is Up. The Business Is Down., argues for networks, a healthy status light says nothing about whether the outcome was right.

Human in the Loop Has to Move, Not Disappear

Every AI governance conversation reaches for “human in the loop”. The practical problem is speed. Orro’s companion piece in this edition, Eight Hours Was Optimistic, shows how far exploitation timelines have collapsed. The agents organisations deploy in response run on the same clock. When agents act in seconds, a person approving each step is not oversight. It is a queue, and a tired reviewer clicking approve is barely a control at all.

The answer is to move the human, not remove them. In the model that works in practice, agents gather information in real time, run the analysis and propose an outcome. A person reviews the evidence, can see where the data came from and why the conclusion follows, and makes the call. For well-understood, lower-risk actions, the agent acts on its own within thresholds people have set in advance. Anything that falls outside those thresholds is flagged for a human decision.

That puts people where judgement matters: setting guardrails, reviewing evidence and deciding exceptions. Regulators are pointing the same way. APRA now expects regulated entities to keep people involved in high-risk decisions and to own accountability across the AI lifecycle (APRA, 2026). The OAIC’s guidance says a person should be responsible for verifying AI outputs and able to overturn decisions made (OAIC, 2024).

None of this works without AI agent observability. A person can only judge an exception they can see. They can only trust an agent’s evidence if the lineage behind it has been recorded.

Sign-Off, Not Capability, Is Where Agents Stall

Pilots tend to succeed, then stall. Gartner forecasts that more than 40 per cent of agentic AI projects will be cancelled by the end of 2027, with inadequate risk controls among the reasons (Gartner, cited in SD Times, 2025).

AI-assisted security operations show the pattern clearly. Organisations that see it in action are enthusiastic. Those that have adopted it are resolving tier-one cases faster than their teams could alone. Then adoption slows at contract stage, when the agreement has to acknowledge that internal security data will be processed somewhere else. Without a clear account of how the agents and the data are secured, the response is consistent. It sounds great, but it can’t get signed off.

Regulators see the same gap from the other side. APRA reviewed large banks, insurers and super trustees. It found few had continuous monitoring able to catch model drift, bias or failure modes in time, and contracts often lacked terms on audit rights, model updates, incident notification and data handling (APRA, 2026).

What gets agents through is evidence, assembled before the risk review. Risk and legal teams want four answers. They want to know where data goes and who can see it. They want to know what each agent may do, and on whose authority. They want to see how its behaviour is monitored and what happens when it drifts. And they want proof that every action can be reconstructed afterwards. What an agent may do is an identity question as much as a governance one, covered in Zero Trust Wasn’t Built for This. For directors, The Board Has a New Risk on the Agenda sets out the accountability side. AI agent observability is the operational layer beneath both: the record that turns a promise about agent behaviour into proof.

Evidence Snapshot

On getting agents into production

More than 40% of agentic AI projects are forecast to be cancelled by the end of 2027, with inadequate risk controls among the reasons (Gartner, cited in SD Times, 2025)
Few of the large financial institutions APRA reviewed had continuous monitoring in place to detect model drift, bias or failure modes in a timely way (APRA, 2026)

On shadow AI

48% of Australian employees who use AI admit to using it in ways that contravene company policy, including uploading sensitive company information into free public AI tools (University of Melbourne and KPMG, 2025)
The share of breaches involving shadow AI more than doubled in a year, from 20% to 43%, and 92% of organisations with an AI-related incident lacked proper AI access controls (IBM, cited in Baker Donelson, 2026)

Agents Change the Network Too

Most enterprise networks were designed for traffic moving in and out. In Orro’s experience, roughly 80 per cent has traditionally flowed north-south and 20 per cent east-west, between systems. Agentic workloads are tipping that balance. An agent does not behave like a person typing a command and waiting for a reply. It reaches out for everything it needs, pulls it back and reaches out again, in bursts and in larger chunks. Networks built for predictable user traffic need to be designed, and watched, for that pattern too.

Shadow AI Is the Same Problem in Reverse

Sanctioned agents need watching because they act on the organisation’s data. Shadow AI is the mirror image: staff putting company data into systems nobody is watching at all. Among Australian employees who use AI, 48 per cent admit to doing so in ways that contravene company policy, including uploading sensitive company information into free public AI tools (University of Melbourne and KPMG, 2025). Consumer tiers are often cheaper because their terms let the provider use inputs to improve its models. Corporate tiers typically keep data inside the organisation’s own tenant.

Picture a staff member at a financial services firm using a personal AI account to draft a letter to a customer about poor advice they received. The customer’s details, and the admission, now sit outside the firm’s control. The OAIC recommends, as best practice, not entering personal information, particularly sensitive information, into publicly available generative AI tools (OAIC, 2024).

Blocking alone rarely works. People route around controls with personal devices, personal VPNs or a phone camera pointed at a screen. The practical answer is sanctioned tools people actually want to use. Add visibility of how AI is being used, and data loss prevention controls that follow the work. The urgency is growing. IBM’s latest research found the share of breaches involving shadow AI more than doubled in a year, to 43 per cent (IBM, cited in Baker Donelson, 2026). Orro explored controls that travel with the user in The Perimeter Is Gone.

AI Agent Observability Turns a Pilot into a Service

The organisations scaling AI agents are not the ones with the most capable models. They are the ones that can show, at any moment, what data went in and what the model did with it. They can show what action resulted and whether it was right. That is AI agent observability: production monitoring applied to a new kind of system. It is also what lets risk, legal and the board say yes.

Governance is where that work starts. Version 2.0 of NIST’s Cybersecurity Framework put governance first. An organisation that cannot say how its AI agents are governed and monitored has a gap worth measuring.

Find the gap before your risk team does

Orro’s NIST CSF self-assessment benchmarks your current security maturity in about the time it takes to review a pilot report. If you can’t yet say who owns each AI agent, what it can reach and how its behaviour is watched, that is the gap it will help you size first.

TAKE THE NIST CSF SELF-ASSESSMENT

AI Agent Observability: Common Questions

What is AI agent observability?

AI agent observability is the ability to see what an AI agent is doing in production: what data went into it and where that data came from, what the model did with it, what action resulted, and whether that action was right. It extends production monitoring from “is the service up?” to “is the agent behaving correctly?”

How do you keep a human in the loop when AI agents act at machine speed?

Move the human rather than remove them. People set the guardrails and thresholds, agents act within them, and AI agent observability flags anything outside those limits for a human decision. For higher-risk actions, the agent gathers the evidence and proposes an outcome, and a person reviews the evidence and makes the call.

Why do AI agent pilots stall before production?

Often not because the technology fails, but because risk, legal and procurement teams cannot get the evidence they need to approve it. AI agent observability supplies that evidence. Approvers need to know where data goes, what each agent is permitted to do, how its behaviour is monitored and whether its actions can be reconstructed afterwards.

What is shadow AI, and why doesn’t blocking it work?

Shadow AI is staff using AI tools the organisation has not approved, often personal or consumer accounts, so company data ends up in systems nobody is observing. Blocking alone tends to fail because people work around it. Sanctioned tools people want to use, visibility of AI use and data loss prevention controls work better together.

How do AI agents change enterprise network traffic?

Agents generate bursty, high-volume traffic, much of it moving between systems (east-west) rather than in and out of the network (north-south). Networks designed mainly for predictable user traffic may need to be redesigned and monitored for that pattern.