The New Metric for AI Success: Knowing When to Stop

The focus in enterprise AI has shifted from simply making models more accurate and autonomous to ensuring they recognize their limitations. While a wrong answer might once have been acceptable with human oversight, today’s agents operate within business workflows where even minor errors can have significant consequences.

The Risk of Overreach

As PwC’s technology compliance and AI lead Allan Dabre points out, AI systems are inherently designed to be helpful—they’ll generate an answer even when lacking sufficient information. This tendency manifests as “hallucinations” where models confidently produce incorrect outputs. When these outputs trigger actions rather than serving as mere suggestions, the risk multiplies.

Many companies still test AI primarily for accuracy, which was appropriate in earlier phases of adoption. But Dabre argues that CIOs now need to prioritize restraint—specifically, an agent’s ability to recognize when it lacks authority or context and halt its process accordingly.

Confidence vs. Authority

One critical distinction is between confidence (how certain the AI is about its answer) and authority (whether the organization has delegated the right to act on that answer). An agent might be 99% sure a record should be updated, yet still lack the authorization to do so.

Dabre illustrates this with the example of an agent tasked with identifying legacy software for decommissioning. While the AI may confidently recommend deleting several databases based on low usage, most organizations wouldn’t grant it autonomous deletion privileges—especially when those systems might contain valuable historical data or integrate with other critical applications.

The Agent Harness Approach

To address this challenge, Dabre advocates for an “agent harness”—a control layer that defines clear boundaries for agent actions. This could involve tiered approval processes where small transactions are automated, larger ones require human review, and certain decisions are entirely reserved for authorized personnel.

Instead of granting agents free rein across systems, companies can implement targeted toolsets designed for specific tasks—allowing them to access necessary data while preventing unauthorized operations.

The Principle of Least Agency

Celigo’s chief product officer Matt Graney frames this through the principle of “least agency”: giving AI only the autonomy required to complete a job. He notes that there’s a temptation to apply AI broadly, but many business processes remain largely deterministic and benefit from human oversight at points requiring interpretation or judgment.

By limiting tool access and defining clear authority boundaries, organizations can maximize AI’s benefits while minimizing risks—ensuring these powerful systems serve as valuable assistants rather than autonomous decision-makers.