Navigating the Risks of Unobserved AI Agents

Enterprises rushing to implement AI agents without robust governance frameworks face significant risks. Experts warn that deploying these systems with limited visibility creates vulnerabilities that could lead to major operational and reputational setbacks.

The Observability Gap

According to a recent survey by TrueFoundry, over half of organizations (54%) lack full traceability into what their AI agents are doing, while 56% have no centralized governance layer. This “governing the blind” phenomenon is compounded by the fact that traditional security tools weren’t designed to monitor autonomous systems.

Mahesh Kumar Goyal, a senior data and AI expert at Google, emphasizes this challenge: “Most enterprises are trying to govern what they can’t see.” He notes that agents often operate without clear oversight, making it difficult to detect anomalies or ensure compliance.

Beyond Observability

Effective agent governance requires more than just monitoring; it demands actionable insights. Adel El Hallak, VP of AI software at Nvidia, explains: “Observability is foundational to transparency, but just observing isn’t enough. We need to be able to take those signals and turn them into something actionable.”

This includes:

  • Implementing policy enforcement layers that mediate every interaction
  • Establishing end-to-end tracing for all prompts and actions
  • Creating feedback loops that allow continuous testing and refinement

Tiered Autonomy as a Solution

Rather than treating agents as “set it and forget it” solutions, experts recommend a tiered approach. This means granting greater autonomy on low-stakes tasks while maintaining human oversight for consequential decisions—similar to how the financial system operates with auditability and circuit breakers.

As AI continues to evolve, robust governance frameworks will be essential for unlocking its potential while mitigating risks. Enterprises that prioritize transparency and control today will be better positioned to thrive in the age of autonomous systems.