From hallucinations to hardware: Lessons from a real-world computer vision project gone sideways | VentureBeat

Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy. Learn more

Computer vision projects rarely go exactly as planned, and this one was no exception. The idea was simple: Build a model that could look at a photo of a laptop and identify any physical damage — things like cracked screens, missing keys or broken hinges. It seemed like a straightforward use case for image models and large language models (LLMs), but it quickly turned into something more complicated.

Along the way, we ran into issues with hallucinations, unreliable outputs and images that were not even laptops. To solve these, we ended up applying an agentic framework in an atypical way — not for task automation, but to improve the model’s performance.

In this post, we will walk through what we tried, what didn’t work and how a combination of approaches eventually helped us build something reliable.

Our initial approach was fairly standard for a multimodal model. We used a single, large prompt to pass an image into an image-capable LLM and asked it to identify visible damage. This monolithic prompting strategy is simple to implement and works decently for clean, well-defined tasks. But real-world data rarely plays along.

We ran into three major issues early on:

This was the point when it became clear we would need to iterate.

One thing we noticed was how much image quality affected the model’s output. Users uploaded all kinds of images ranging from sharp and high-resolution to blurry. This led us to refer to research highlighting how image resolution impacts deep learning models.

We trained and tested the model using a mix of high-and low-resolution images. The idea was to make the model more resilient to the wide range of image qualities it would encounter in practice. This helped improve consistency, but the core issues of hallucination and junk image handling persisted.

Encouraged by recent experiments in combining image captioning with text-only LLMs — like the technique covered in The Batch, where captions are generated from images and then interpreted by a language model, we decided to give it a try.

Here’s how it works:

While clever in theory, this approach introduced new problems for our use case:

It was an interesting experiment, but ultimately not a solution.

This was the turning point. While agentic frameworks are usually used for orchestrating task flows (think agents coordinating calendar invites or customer service actions), we wondered if breaking down the image interpretation task into smaller, specialized agents might help.

We built an agentic framework structured like this:

This modular, task-driven approach produced much more precise and explainable results. Hallucinations dropped dramatically, junk images were reliably flagged and each agent’s task was simple and focused enough to control quality well.

As effective as this was, it was not perfect. Two main limitations showed up:

We needed a way to balance precision with coverage.

To bridge the gaps, we created a hybrid system:

This combination gave us the precision and explainability of the agentic setup, the broad coverage of monolithic prompting and the confidence boost of targeted fine-tuning.

A few things became clear by the time we wrapped up this project:

What started as a simple idea, using an LLM prompt to detect physical damage in laptop images, quickly turned into a much deeper experiment in combining different AI techniques to tackle unpredictable, real-world problems. Along the way, we realized that some of the most useful tools were ones not originally designed for this type of work.

Agentic frameworks, often seen as workflow utilities, proved surprisingly effective when repurposed for tasks like structured damage detection and image filtering. With a bit of creativity, they helped us build a system that was not just more accurate, but easier to understand and manage in practice.

Shruti Tiwari is an AI product manager at Dell Technologies.

Vadiraj Kulkarni is a data scientist at Dell Technologies.

If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.

Read our Privacy Policy

Thanks for subscribing. Check out more VB newsletters here.

An error occured.