Not Every Problem Needs an AI Agent

As generative AI (GenAI) surged in popularity, many companies rushed to integrate it into their products. However, experience shows that applying powerful technologies indiscriminately can lead to unreliable results and wasted resources.

When I led a team of data scientists at one company, we faced this challenge firsthand. Despite board pressure to incorporate GenAI everywhere, I advocated for careful selection based on technical merit. Our recommender system was initially considered a prime candidate, but I argued that an LLM—trained to predict text rather than user behavior—would struggle with our proprietary engagement data.

The outcome validated this approach: the GenAI integration proved less effective than our existing ML models, while consuming significantly more compute power.

This isn’t just a one-off experience. Engineering teams at companies I’ve consulted often find themselves replacing reliable ML systems with agentic AI that delivers inconsistent outputs and lacks confidence scores—all while incurring prohibitive token costs.

The Right Tool for the Job

The key takeaway is this: even the most powerful technology fails when applied to the wrong problem. LLMs excel at tasks requiring creative generation or complex reasoning, but they’re not always the answer.

Practical Applications:

• Use ML models for calibrated predictions where accuracy matters (e.g., fraud detection, risk assessment)
• Leverage LLMs for explanations, content generation, and conversational interfaces
• Prioritize deterministic logic for critical operations like data retrieval and financial calculations

A Real-World Example:

Our team once built a content moderation system that would flag inappropriate user posts. Initially, we favored GenAI’s ability to provide explanations alongside its classifications. However, we discovered that the LLM’s probability scores were uncalibrated—rendering them unusable for routing content to human reviewers.

The solution? We split the task: our ML model provided accurate probability scores for automated decisions, while the LLM generated contextual explanations when human review was required.

Beyond the Hype

As organizations navigate this technological landscape, a strategic approach is essential. Instead of chasing every AI trend, focus on aligning technology with business needs and prioritizing solutions that deliver measurable value—not just buzz.