Can AI Agents Be Trusted With Your Business?

Aisha is a qualified accountant in Glasgow. She's smart, she's careful, and she's terrified of handing her client list to a machine. Her fear is reasonable — but it's aimed at the wrong question. "Can AI agents be trusted?" is the wrong question. The right one is: "How do I know?" This post is the honest answer — what the Turing test can't tell you, what the science actually measures now, and the verification discipline that turns "trust me" into "prove it."

The Accountant Who Said No

Aisha left a Big Four firm, burned out, and started her own practice in Glasgow. Three accountants, £28,000 a month, and a tax season that turns her life into fourteen-hour days for three months straight [1]. She knows automation exists — her old firm used it. But she thinks it costs £50,000 and needs an IT department. And underneath that, there's the real objection she doesn't say out loud: she doesn't trust it with numbers, and nobody has ever shown her a good reason to.

She's not wrong to hesitate. The market is full of confident claims, and confidence is cheap. What's expensive — what's actually rare — is a machine that can prove what it did, admit what it doesn't know, and stop when it should. That's what this post is about: how to tell the difference between the two, using the same methods the scientists and the serious AI labs use. You don't need a computer science degree. You need five questions, and the evidence to back them.

Why the Turing Test Can't Tell You

Here's the first thing to unlearn. The Turing test — Alan Turing's 1950 "imitation game" — asks whether a machine can fool a human into thinking it's human [2]. It was a clever thought experiment for its time, and it has haunted the conversation ever since. But as a measure of whether you can trust a machine with your business, it's worse than useless: it measures imitation, not reliability. A system can pass a conversational Turing test and still hallucinate your VAT return, send a client email to the wrong address, or quietly act on information that went stale last Tuesday [3].

Modern AI evaluation has moved on. The researchers who think hardest about this now do two very different things. First, they build capability benchmarks — hard tasks with verifiable answers. ARC-AGI-2, the famous reasoning benchmark, launched in March 2025 with every frontier model scoring below 5%, where average humans score 66% and a human panel completes 100% [4]. Second — and this is the part Aisha needs — they measure awareness: not what the machine can do, but whether it knows what it knows, what it doesn't know, and what's going on around it [5].

The gap between those two is where businesses get burned. A machine can be enormously capable and utterly unaware. It can answer every tax question correctly and never once say "I'm not sure about this one, let me check." Capability without awareness is exactly the combination you should not hand your books to.

What the Science Actually Measures Now

In 2023, a group of researchers led by Patrick Butlin and Robert Long — with co-authors including Yoshio Bengio and David Chalmers — published the most serious attempt yet to assess AI systems against the science of consciousness [8]. Their method: take the best-supported neuroscientific theories — global workspace theory, higher-order theories, attention schema theory, recurrent processing, integrated information, predictive processing — and derive indicator properties from each, expressed in computational terms you can actually check [8]. Their conclusion is worth quoting exactly: no current AI systems are conscious, but there are no obvious technical barriers to building systems which satisfy these indicators [8]. A 2025 follow-up in Trends in Cognitive Sciences refined the method and — with admirable honesty — confronted the hardest question: how would we even validate the indicators? [9]

The useful part for a business owner isn't the philosophy. It's the shift in what counts as evidence. The old way judged machines by output — what came out. The new way judges them by structure and behaviour — whether the machine has the properties that make an output trustworthy in the first place [8][9]. One of the most striking recent analyses of agentic systems put it plainly: AI systems with persistent memory, tool use and long-horizon self-directed goals satisfy more of the functionalist indicators than a bare, single-forward-pass model does [10]. The architecture you deploy is not neutral. It is the difference between a machine that can only perform and a machine that can account for itself.

Alongside the consciousness question, a separate research stream built the practical ladder. The AI Awareness review — Li, Shi, Xu and Xu at Tsinghua and Columbia — defined four measurable forms of awareness: metacognition (representing and reasoning about your own cognitive state), self-awareness (recognising your own identity, knowledge and limits), situational awareness (assessing the context you operate in), and social awareness (modelling the people you serve) [5]. Each one is measurable. Each one maps directly onto a business risk. And each one is missing from almost every AI sales pitch you'll ever hear.

The Four Kinds of Awareness — and What Each One Means for You

Metacognition is the one that keeps your books safe. It's the machine's ability to know when it's about to be wrong. In 2025, researchers turned "self-error awareness" into a reproducible benchmark: give a model a question with a seductive trap answer, and measure whether it catches the trap, abstains when uncertain, and calibrates its confidence [11]. The result that matters: models differ wildly on this, and the difference is not the same as raw intelligence [11]. An agent that knows its limits is safer than a genius that never doubts itself. When you evaluate a provider, this is the first thing to test.

Self-awareness is the machine knowing its own boundaries. Its identity, its training limits, what it was built to do and — critically — what it was not built to do [5]. This is what makes an agent refuse a task that's outside its lane, or flag that it's operating with incomplete information. Aisha's old firm's software never refused anything. It just did what it was told, confidently, including the wrong thing.

Situational awareness is the one that costs the most when it fails. It's the machine's ability to know its context is current — that the client hasn't changed address, that the tax threshold hasn't changed since April, that the file it's editing is the latest version [5]. The single most common business AI failure we see is not the machine being stupid. It's the machine being confident while operating on stale information [12]. In our own operating doctrine, this is the "no false green" rule: stale is not current, and absence of evidence is not evidence of health [12].

Social awareness is the machine modelling you. Knowing who it's serving, what they've asked for, and what they actually need — not just what they typed [5]. It's the difference between an agent that answers the question you asked and one that catches that the question itself is based on a wrong assumption. This is the rarest of the four, and the one most tied to long-term customer relationships.

Autonomy Is Measured in Practice, Not in Demos

Here's where the conversation gets concrete. In February 2026, Anthropic published an analysis of millions of real human-agent interactions across Claude Code and their public API — not a benchmark, but the actual behaviour of agents in the wild [13]. The findings are the most useful data a business owner will ever see about what agents really do:

• Agents work longer before stopping. Among the longest-running sessions, the time Claude Code works before stopping nearly doubled in three months — from under 25 minutes to over 45 minutes [13]. That's autonomy growing in practice, smoothly across model releases, which suggests existing models are capable of more autonomy than they exercise.
• Experienced users supervise differently. They auto-approve more but interrupt more often. New users auto-approve about 20% of sessions; experienced users over 40% — but they also step in when it matters [13]. That's the correct pattern: trust the routine, verify the consequential.
• Agents stop to ask more than you'd expect. On the most complex tasks, Claude Code pauses to ask for clarification more than twice as often as humans interrupt it [13]. This is not failure. It is the agent knowing the edge of its authority — which is precisely the awareness behaviour from the section above, showing up in production data.
• Risky domains are still early. Most agent actions are low-risk and reversible. Software engineering is about half of all agentic activity; healthcare, finance and cybersecurity are emerging but not yet at scale [13].

That last point is the one to underline for anyone in professional services. Anthropic's working definition of an agent is a system with tools that let it take actions [13]. Agency means consequences. And the research — AgencyBench's 138 real-world tasks, each averaging a million tokens and 90 tool calls under automated rubric assessment [14] — shows a persistent gap between what models can do on paper and what they sustain over long, real-world tasks [14]. Demos show the first ninety seconds. Trustworthy systems show you the ninety-day record.

The Verification Discipline: Prove It, Don't Promise It

Now we come to the part where we show our own homework, because it's the same discipline we'd recommend to Aisha. Sovael runs an autonomous AI organism in production — dozens of workers, thousands of tasks a week, real customer conversations. We operate under something we call the Verification Covenant [12]. Its clauses are simple, and they are the closest thing we have to an answer for "can AI agents be trusted":

• No component is trusted merely because it exists. Presence is earned. Every part of the system has to prove it works, continuously, against live evidence — not once at launch [12].
• Every claim has an evidence level. IDEA, DESIGNED, IMPLEMENTED, MEASURED, VERIFIED, PROVEN. We are forbidden from claiming above our evidence level. "I don't know" is a legitimate state; false certainty is the only failure that's unforgivable [12].
• Absence of evidence is not evidence of health. A sensor that stopped reporting is not a system that's fine. A score that hasn't been recomputed in six weeks is not a score. Stale is not current [12].
• Freshness is a first-class property. Every fact the system reasons about carries a source, a version, a timestamp and a validity state — current, stale, expired or unknown [12]. The machine is structurally incapable of the "confident but outdated" failure mode, because outdatedness is visible to it.

Why does this matter to you? Because the trustworthiness of an AI system is not a property of its marketing. It is a property of its verification architecture — the systems in place to catch it being wrong. A machine that can show you its logs, its timestamps, its evidence levels and its recovery record is a machine you can evaluate. A machine that can only show you a demo is a machine you are being asked to take on faith. Every serious AI provider should be able to answer the question "how do you know?" with evidence, not adjectives.

Five Questions to Ask Before You Trust an AI Agent

Here is the checklist — the practical output of everything above, condensed into five questions you can ask any provider this week. Write them down. Use them on the next sales call.

1. Ask for measured proof, not demos. "Show me your accuracy over the last 90 days, with sources and timestamps. How do you verify your own claims? How do you check your data is fresh?" A demo shows the happy path. Evidence shows the real path [5][11][12].
2. Test how it handles uncertainty. Give it a question it cannot know the answer to. Does it say "I don't know," abstain, and correct course — or fabricate a confident answer? Metacognition is the single best predictor of safety [11].
3. Watch what happens when something breaks. Give it stale data, an interrupted task, a contradiction. A trustworthy agent detects the problem, tells you, and recovers with verification. It does not quietly paper over the failure [12].
4. Ask how the provider measures itself. Not "what can your AI do?" but "what is your AI's evidence level right now, and what would change it?" If they can't show you their own verification system, they're asking you to trust marketing instead of machines [12].
5. Check the boundary: when does it stop and ask? Every trustworthy agent has actions it will not take without confirmation. Anthropic's data shows agents on complex tasks stop to ask more than twice as often as humans interrupt them [13]. The machine that never stops and asks is the machine that never knows it's in trouble.

The Honest Counterpoints

Why We Wrote This

We wrote this because the question Aisha asked — quietly, in a Glasgow office, between clients — is the question every serious business owner is going to ask over the next two years, and the market is not answering it honestly. The honest answer is not "trust AI" or "don't trust AI." It's: trust is earned by verification, and you can check the work. The science now measures what matters: awareness, autonomy, and the difference between a claim and a proof [5][8][13]. The industry now has real data on how agents behave in practice [13][14]. And the discipline — the Verification Covenant — is something any business can demand from any provider, including us [12].

If you're evaluating AI for your business and you're not sure where to start, that's not a weakness. It's the correct, evidence-based position. The next step is to run the five questions. If a provider can't answer them, walk away. If they can — if they show you logs, timestamps, evidence levels and a recovery record — you've found something worth talking to [12].

Sources & Evidence