95% of AI Pilots Fail. Here's What the Other 5% Got Right.
- Bill Donofrio
- Aug 8
- 6 min read
Generative AI is one of the most hyped and misunderstood technologies we have seen in a long time. According to MIT's 2025 State of AI in Business report, 95% of generative AI pilots never make it to production. I've spent the last three years finding out why, and more importantly, what the surviving 5% actually have in common. This article walks through the use cases that are genuinely ideal for AI, and the ones that aren't, so you don't have to learn the difference the expensive way.
Where I Started
Back in 2023, I was first introduced to OpenAI's GPT-3.5 model and the Ada embedding model. Our principal architect was fully bought in. He explained how learning this technology would change your career and open up endless new possibilities. The early models had issues with hallucinations and limits on token counts. The models were a lot of fun to experiment with but could not be productionized due to a lack of quality.
I remember asking how we could sell this to clients. Databases are fundamental to everything a business does. If set up properly, your CRM or ERP system runs your business. There will always be work building data warehouses. However, we were never going to build models for self-driving cars. The key was to unlock data from your unstructured PDFs, video files, and audio files. But what use cases make sense for a bank, insurance company, or law firm?
The Hype vs. the Reality
I have spent the last three years reading articles, watching videos, and going to every meetup in Chicago. I went to the Databricks conference two years in a row. Jerry Liu, CEO of LlamaIndex, showed a beautiful presentation of how AI could create financial reports at one of those conferences. The only problem was the accuracy was less than 90%. This would translate to millions in SEC fines. At Microsoft's AI Tour events, Judson Althoff claimed that you could open a business, create a business plan, and then create a sales presentation for investors in a matter of minutes. These claims help inflate the stock price of the Magnificent Seven but are poor use cases.
The reality is that AI expedites the build process. You can build proof of concepts very quickly. This is what executives are excited about. However, delivering a production-grade product that you can provide to users takes just as long. Your financials cannot be 88% accurate. Your clients can't have a chatbot provide 95% accurate data. Data leakage into internet models cannot happen even 0.01% of the time.
Where AI Should Never Be the Answer
Here is what I have learned. Projects that need to be precise, like financial reports, should never be built using AI. Processes that flow from one step to another in a deterministic fashion should never be built using AI. Sure, you can use code assistance to help write the code, but real humans need to be in the loop. Governance needs to be determined early on. Actual human testing and user acceptance need to exist. A security expert needs to enforce proper controls. Otherwise, it is a lawsuit waiting to happen.
Here is the hard truth that every executive needs to understand before firing their development team. All AI models are machine learning models built on predictions. In the past, a machine learning model that was 80% accurate was seen as successful. They are not perfect because anomalies exist. I talked to an AI developer at Rush Hospital who is using AI to build models to predict spinal issues. They are taking thousands of images and classifying them. What they realized is that one doctor would mark an image as low severity while another would mark it as medium severity. If we can't agree on one standard, we cannot expect the model to be accurate. Because of this, that project is still in the testing phase, far from production.
Where Generative AI Actually Makes Sense
So, what are the projects that actually make sense? These are projects that help your team without completely replacing them.
Creating exam questions for standardized tests can be a good use case. Models can read entire college textbooks in seconds. This makes it easy to build a database of questions. It saves your team hours of reading every book and setting up countless discovery meetings. With that being said, not all the questions will be appropriate. This still requires a human in the loop. AI will give you citations, but someone needs to review those for accuracy. While not perfect, it still saves hours of manual work.
Helping with legal documentation prep is also a great use case. The legal process in the US is full of paperwork. This includes entire reams of documentation for each case. Suppose you are an immigration law firm. You have to review all of the documents your clients provide to help them get citizen status. Your case relies on checking their work history and running a full background check. What profession are they in, how many employees will your company hire, and how can they contribute to the US economy? Do they have a valid asylum case? This documentation needs to be thorough. A glossed-over summary will not make it past USCIS. AI can scan hundreds of documents and build a summary with a little prompting. Instead of a legal writer handling 2 or 3 cases a week, they could handle 10. Now that's real productivity, and real ROI.
Creating an internal knowledge base also makes sense. AI can review all your company documents. It can rewrite them so they're all consistent. It can organize them into proper categories. I have worked with companies that would hire a full-time employee and have them work on a project like this for six months. Now, it can be done in a week.
Creating your own personal agent to handle routine tasks is a great use case. Set up agents to write emails, read your Teams chats, and prioritize your calendar. You can personalize it to keep you up to date with current trends by linking it to industry websites. You and your team become more organized, less stressed, and much more productive. Why spend your Monday morning getting ready for the week when AI can do this for you?
Summarizing meetings and tracking action items is one of the highest-adoption use cases out there, and for good reason. AI can sit in on a call, transcribe it, and produce a clean summary with clear owners and next steps. Nobody's job depends on that summary being perfect. If it misses a nuance, someone catches it in thirty seconds and fixes it. That's exactly the kind of low-stakes, high-frequency task generative AI is well suited for. It creates notes where none would have existed otherwise and keeps your project manager on task.
Extracting data from unstructured documents and forms is another strong fit, and it circles back to where this article started. Invoices, intake forms, scanned contracts, and handwritten notes are exactly the kind of unstructured PDFs that used to require manual data entry. AI can pull the relevant fields into a structured format in seconds. A human still needs to spot-check the output before it feeds into a downstream system, but that review is far faster. This can also provide another source of data to understand your clients or competitors.
Finally, building a chatbot for internal issues or common customer issues is a strong use case. This should replace your current 800 number. It is simple to program and much more personalized to your customers' needs. Humans will still need to resolve outlier-type issues, but 90% of issues will go away overnight.
The Bottom Line
That's the throughline behind every use case above, and the real answer to what the surviving 5% get right. More than 80% of companies have already piloted generative AI, but almost none of them ask it to be perfect. They ask it to be fast, and they keep a person in the loop to catch what it misses. Using AI to replace a worker, or to complete a task that must be 100% accurate, is a terrible idea. Using it to help your employees and customers discover things faster is a perfect one. Before you greenlight your next AI project, ask one question: can a person catch the mistake before it matters? If the answer is no, rethink your approach.
Reference
MIT NANDA, "The GenAI Divide: State of AI in Business 2025," August 2025
Comments