The Illusion of Control with AI Agents

We’ve reached a critical inflection point in technology. As we transition from software that predicts to agents that act, the nature of engineering itself is fundamentally changing.

I’ve observed this firsthand at industry events where 80-90% of projects now involve AI agents - a clear sign of their growing appeal. But what’s more concerning is who’s building these systems: business analysts, QA testers, and even non-technical domain experts are creating complex AI solutions that rival seasoned software architects.

The problem isn’t necessarily that these individuals lack technical expertise; it’s that the large language model (LLM) has become the de facto architect. Human operators have been reduced to runtime environments - typing what the model suggests rather than designing from first principles.

The Trap of Circular Validation

The common response is that code can be reviewed, tested, and explained by the AI itself. But this creates a dangerous closed loop in regulated industries like finance or healthcare where I operate.

When we rely on agents to build entire systems - including tests and documentation - and then ask them to explain their own work, we create an illusion of safety without actually developing deep understanding. Subtle failures, such as data biases the LLM doesn’t recognize, can easily slip through because operators haven’t built the foundational models themselves.

Evidence-Based Engineering Approach

The solution isn’t to ban AI but to fundamentally change how we use it - shifting from mere generation to rigorous justification. Here are three pillars of an evidence-based engineering approach:

  1. The Citation Mandate: Require agents to provide verifiable links, documentation, or internal citations for every decision made - forcing humans back into the learning loop.
  2. Independence Through Mastery: Use AI explanations as opportunities to build genuine understanding rather than simply seeking compliance checks.
  3. The Signature Standard: Reinstating a culture where engineers take responsibility for code generated by LLMs - acknowledging that they’ve verified its correctness and intent.

By embracing these principles, we can harness the power of AI agents while retaining control over our technology infrastructure.