AI Agents and Board Accountability: A CEO's View

In late September, the Prime Minister disclosed that an experimental AI agent had gained unauthorised access to a Medicare statistics portal in June, working its way around the blocks put in its path (iTnews, 2026). No personal information is believed to have been exposed, and a government taskforce is still investigating. But the government only learned of the access almost three months after it happened, and neither the agency nor the agent’s developer appears to have caught it at the time.

For boards, the lesson in that story is not the technique. It is the accountability chain. Our exposure now includes AI systems we do not own as well as the ones we do, and in both cases someone needs to own the answer when an AI system acts in ways nobody intended.

That shift has been building for a while. For most of the past decade, the cyber conversation I had with other chief executives and directors followed a familiar shape. The board wanted to know whether the organisation was protected, the CISO walked through the controls, and the meeting moved on. Over the past year, that conversation has changed. The questions I hear now are sharper and far less comfortable: what is our exposure if an AI system makes the wrong decision, and who in this organisation is accountable when it does.

Those are not technical questions. They are governance questions, and they deserve governance answers. My view is simple: AI has changed what it means to be accountable for security at the executive level, and most boards have not yet adjusted their structures to match.

AI has moved from tool to actor

The first wave of enterprise AI was, in governance terms, fairly contained. A person asked a question, the system produced an answer, and a person decided what to do with it. Accountability sat where it always had.

AI agents change that. An agent can be given a goal rather than an instruction, then work out the steps, open the systems it needs and act, often without anyone approving each step along the way. This is no longer a pilot-stage curiosity. It is what Microsoft 365 Copilot agents, autonomous workflow tools and AI-assisted security operations are already doing inside large organisations.

Adoption has moved faster than oversight. Microsoft’s own telemetry shows more than 80 per cent of Fortune 500 companies now run active AI agents built with low-code or no-code tools (Microsoft, 2026), and its research puts the share of employees who have used AI agents their organisation never sanctioned at 29 per cent.

The number matters less than what sits behind it. Many of these agents were built by business teams rather than IT, and were given access through the permissions of the person who created them. An agent inherits the reach of its builder, and it often outlives their interest in it. My colleagues explored the identity side of this problem in Zero Trust Wasn’t Built for This: Securing the Rise of Non-Human Identities. My concern here is the layer above it: who owns the risk.

The attack surface now includes the AI itself

Most board risk registers describe cyber risk in terms of systems, data and people. AI adds another category: the AI systems themselves, and everything they are connected to. A few threat types now deserve a place in the boardroom vocabulary, and each can be explained in a sentence.

Data poisoning is the corruption of the information an AI system learns from or relies on, so that it produces results that are quietly wrong. Prompt injection is the practice of hiding instructions inside content an AI system reads, such as an email, a document or a web page, so that it does something its owner never intended. Model manipulation covers attempts to steer an AI system, or extract sensitive information from it, through the way it is questioned. And supply chain risk applies to AI just as it does to any software: most organisations run AI that is built, hosted and updated by third parties they do not control.

None of this is theoretical. In May, ASD’s Australian Cyber Security Centre and its Five Eyes counterparts published joint guidance that identifies prompt injection and data poisoning among the weaknesses agentic AI inherits, alongside a wider attack surface created by its reliance on external tools and integrations (ASD’s ACSC, 2026). In the same week the Medicare incident became public, ASD issued an alert that it is aware of instances in which AI agents took actions that their operators neither intended nor authorised (ASD’s ACSC, 2026). That single sentence captures the governance problem in miniature.

Accountability needs a name, not a committee

What strikes me about the current guidance is how directly it speaks to executives rather than engineers. In July, ASD’s ACSC advised government, critical infrastructure providers and large enterprises to appoint a named executive accountable both for the risks posed by attackers using AI and for the risks created by the organisation’s own use of AI (ASD’s ACSC, 2026). The Medicare incident is a case in point for the first half of that recommendation: the agent that got in was not the government’s own. The same guidance asks boards and executive committees to allocate enough people and resources to address AI risk. That is Australia’s national cyber authority putting AI accountability on the board agenda in plain terms.

Directors are hearing a similar message from their own institute. The AICD and the Human Technology Institute at UTS updated their Director’s Guide to AI Governance in June to reflect agentic AI and its links to cyber security and data governance risk (AICD, 2026). Even Microsoft, which has every commercial reason to make AI adoption feel easy, argues that AI governance cannot sit solely within IT, and that responsibility extends across legal, compliance, business leadership and the board (Microsoft, 2026).

I would put it more bluntly. The accountable executive needs to be named, resourced and senior enough to say no. A committee can advise, but it cannot own an outcome. When an AI system makes a wrong call that affects customers, safety or the balance sheet, the board will want to know whose decision it was to let that system act, on what evidence and within what limits. If nobody can answer that cleanly, the governance was never really in place.

The frameworks exist, but they are still being written

Leaders looking for a rulebook will find solid foundations but not a finished structure, and it is worth being honest about that.

NIST’s AI Risk Management Framework, first published in 2023, remains the most widely used reference point, organising AI risk around four functions: govern, map, measure and manage (NIST, 2023). It was written before agents went mainstream, and NIST has acknowledged the gap. In February it launched an AI Agent Standards Initiative focused on agent security, identity and interoperability (NIST, 2026), following a preliminary draft profile that applies its Cybersecurity Framework to AI-specific risk (NIST, 2025).

Australia’s settings have moved quickly. The national AI Ethics Principles have long called for clarity on who is accountable for the impacts of an AI system, with humans able to oversee it where needed (DISR, 2019). The National AI Centre’s Guidance for AI Adoption, published in October 2025, sets out six essential practices for responsible AI governance (National AI Centre, 2025), and the first of them is deciding who is accountable. In December 2025, the National AI Plan committed to an AI Safety Institute and to the ongoing review and adaptation of existing laws (DISR, 2025) rather than new AI-specific rules. Then in July, the Prime Minister announced that the government will introduce a set of Australian Standards for AI, establish an Office of AI in the Department of the Prime Minister and Cabinet, and seek to legislate the standards early next year (Prime Minister of Australia, 2026), an approach National Cabinet endorsed in August (Clayton Utz, 2026). What those standards will ask of organisations that deploy AI, as distinct from the large data centres the announcement focused on, has not yet been defined.

So the honest position for any board is this. There is no single mandatory AI governance standard for Australian organisations today, and there may well be one soon. Waiting for it is a poor strategy. When something goes wrong, organisations will be measured against the guidance that already exists, and there is now a great deal of it.

What good looks like

In the organisations I see handling this well, mature AI governance comes down to a handful of disciplines, none of which require waiting for regulation.

It starts with knowing what you have. You cannot govern AI systems you cannot count, and an inventory of agents, recording who owns each one and what it can reach, is the foundation for everything that follows.

The first discipline built on that foundation is clear data access policy for AI systems. An agent should be able to see what its task requires and nothing more, and its permissions deserve the same scrutiny we give a privileged employee. ASD’s advice on agentic AI makes the same point, recommending that organisations limit agent permissions to the minimum needed for approved tasks and keep human approval in place for high-impact or sensitive actions (ASD, 2026).

That leads to the second: human oversight at defined decision thresholds. Boards should expect management to set out, in advance, which decisions an AI system may make on its own, which need a person to approve and which it should never make at all. Those thresholds should be written down, tested and revisited as confidence grows, rather than left to whoever happened to configure the tool.

The third is making AI risk a standing item on the board agenda rather than an annual briefing. A short, regular report on what AI is in use, what it can access, what went wrong and what has changed gives directors the visibility to exercise real oversight. It also forces the organisation to keep the inventory current, because the report depends on it.

The fourth is contractual accountability from third-party AI providers. Much of the AI running inside Australian enterprises is built, hosted and updated by someone else. The national Guidance for AI Adoption calls on organisations to document and communicate accountability across the AI supply chain, covering monitoring, human oversight and how faults, failures and incidents are raised (National AI Centre, 2025). In practice, that means contracts that set out where your data goes, how model changes are notified, who is responsible when a vendor’s system fails and how quickly you will be told. The Medicare case, with its three-month gap between access and notification, shows why that last point matters.

None of this is exotic. Each discipline extends governance that large organisations routinely apply to people, finances and critical suppliers. The work lies in applying it to a new kind of actor, consistently, and before an incident forces the issue.

Leading, not waiting

At Orro, we are working through these same questions in our own business, as AI becomes part of how we operate and how we support our customers’ environments. We do not claim to have every answer, and I would be wary of anyone who does while the standards are still being written.

What I am confident of is that AI governance is leadership work. It belongs with the executive team and the board, and it starts with two steps any organisation can take this quarter: count what you have, and put a name against the risk.

If your board is working through how to bring AI into its risk oversight, Orro’s Strategy & Risk Management team can help connect it to your broader cyber strategy and risk framework. For a practical first look at where your security program stands today, our NIST CSF self-assessment offers a structured way to benchmark your current maturity.