The 95% Number Is a Signal, Not the Conclusion
When I first saw the “95%” figure, the number itself was not the part that stayed with me. A failure rate that dramatic is easy to repeat, but it can also flatten a much more useful question: why do capable models, serious vendors, and well-funded pilots so often fail to become part of a working business system?
MIT Project NANDA’s 2025 report, The GenAI Divide: State of AI in Business 2025, found that only a small share of integrated enterprise AI pilots were producing measurable P&L impact. That finding is often compressed into the headline that “95% of AI projects fail.”
I would not read it that way.
The 95% figure is not a universal failure rate for every AI initiative, nor is it evidence that generative AI itself does not work. What interests me is the structural problem behind the number: a model can perform well and still fail to become useful inside the way a business actually operates.
I have seen variations of this problem long before generative AI. A CRM can contain the right customer data and still fail to help a salesperson make the right decision. A process can be technically well defined and still break because the context needed at a handoff lives in an email, a meeting note, a spreadsheet, or someone’s memory.
Adding an LLM does not automatically solve that fragmentation. In some cases, it simply gives the fragmentation a more intelligent interface.
That is why I think the better starting question for enterprise AI is not:
“Which model should we deploy?”
It is:
“What part of this workflow actually needs AI judgment, and what context must be assembled before that judgment can be trusted?”
That shift—from choosing a model to designing the decision point—is where enterprise AI architecture starts to become interesting.
Better Models Do Not Reassemble Broken Workflows
I came to this view partly from spending years in B2B manufacturing sales, long before I started designing enterprise AI systems.
Customer context rarely lived in one place. A CRM might contain the account, opportunity, and activity history. The latest issue might be buried in an email thread. A meeting note could explain why the customer had changed direction. A proposal might contain assumptions that never made it back into the CRM. And some of the most important context could still exist only in the memory of the salesperson who had been working the account.
None of those systems was necessarily broken on its own. The problem appeared when someone had to make a decision.
Should we escalate this issue? Is this opportunity actually progressing? Has the customer’s priority changed? Which previous proposal is relevant? What should the salesperson know before the next meeting?
At that point, the real work was not simply retrieving more information. It was assembling the right context from several systems, understanding what mattered now, and applying that context to a specific business decision.
This is why I am cautious when enterprise AI is introduced mainly as a new interface on top of existing information silos.
A better model can summarize a longer email thread. It can extract more entities from documents. It can search more naturally across internal knowledge. Those capabilities are useful—but they do not decide which information should be trusted, which user is allowed to see it, which business state is current, or what action the surrounding system should permit.
If the workflow does not assemble those boundaries and that context deliberately, a more capable model is still operating inside a poorly designed system.
In other words, the bottleneck is often not intelligence in the model. It is the architecture around the judgment.
The Better Question: What Should AI Be Allowed to Decide?
Once I started looking at enterprise AI this way, the question I asked during solution design changed.
The usual starting point is:
“Where can we add AI?”
That question sounds practical, but it tends to push the architecture in the wrong direction. It encourages us to start with a capability—summarization, classification, search, generation, an agent—and then look for places to attach it.
I prefer to start with a narrower question:
“What decision or judgment in this workflow is difficult enough to benefit from AI?”
Then I ask a second set of questions.
What information is required to make that judgment? Which parts of that information are authoritative? What is the current business state? What is the user actually allowed to see? Which rules cannot be violated? And what must be validated before the result is allowed to change anything in the system?
These questions separate two very different kinds of work.
Some parts of an enterprise workflow should be deterministic. Permissions, policy rules, record state, transaction boundaries, validation, auditability, and the final execution of business actions should normally remain under application control.
Other parts benefit from probabilistic judgment. Understanding intent, interpreting ambiguous language, comparing evidence, summarizing context, identifying likely meaning, recommending an option, or drafting a response are exactly the kinds of tasks where an LLM can add value.
The architecture becomes much clearer when the system owns the boundaries and the AI owns only the judgment that genuinely requires uncertainty.
This is a small distinction in wording, but a large distinction in system design. Instead of asking an LLM to operate a process, we design the process first—and deliberately create a place inside it where AI judgment is useful.
Deterministic Systems and Probabilistic Judgment Need Different Jobs
This distinction becomes easier to design when the responsibilities of the application and the LLM are treated separately.
The application should remain responsible for the parts of the workflow that need to be reproducible, enforceable, and auditable. The LLM should be used where interpretation or judgment is genuinely useful—but within boundaries the surrounding system has already defined.
When I draw this boundary in an enterprise architecture, the goal is not to make the LLM less capable. It is to make its capability usable.
Consider permissions. An LLM may be excellent at interpreting a customer request, but it should not decide whether the current user is authorized to see a particular account, document, or commercial condition. That decision belongs to the application and its access-control model.
The same applies to workflow state. An AI may recommend that an opportunity should be escalated, that an inbound message represents a new issue, or that a case deserves a different classification. But whether that recommendation is allowed to change a CRM record—and what downstream process that change triggers—should remain deterministic.
I treat cost and execution boundaries the same way. The model can help decide what appears to be the right thing to do. The surrounding application should decide whether it is allowed to happen, under which conditions, and how the action is recorded.
This separation also changes how I think about AI output. Instead of asking the model to directly “do the process,” I prefer the LLM to return a bounded result that the application can validate: a classification, a recommendation, a structured interpretation, a proposed response, or another explicitly defined judgment.
The system can then check that result against required fields, allowed values, business rules, permissions, and current state before anything is executed.
The probabilistic part can be powerful precisely because it does not have to control everything around it.
Once that boundary is clear, the next design question becomes much more specific: where, inside an otherwise deterministic workflow, should that probabilistic judgment actually occur?
The Agentic Moment
I use the term “Agentic Moment” for a specific point inside a deterministic workflow where probabilistic AI judgment is genuinely useful.
It is not the entire workflow, and it is not an autonomous agent taking control of the business process. It is a bounded moment where the application has already assembled the relevant context, established the rules and permissions, and reached a point where interpretation or judgment is required.
Before that moment, the system should do as much deterministic work as possible: receive the event, normalize the input, identify the user and business object, retrieve the relevant state, assemble the permitted context, and remove information that does not belong in the decision.
Then the AI performs the part that is difficult to express as a fixed rule.
It might classify the intent of an inbound message, determine whether the meaning has changed from the previous interaction, compare several pieces of evidence, summarize the situation, recommend an action, or draft a response.
After that judgment, control returns to the application.
The output should be validated, constrained to an expected structure, checked against business rules, and only then translated into a system action.
In that sense, the “agentic” part of the architecture can be surprisingly small. The surrounding workflow is what makes it reliable.
A Salesforce Email-to-Case Example
A simple Salesforce example shows why this separation matters.
Imagine an inbound customer email arriving on an existing Case. The obvious AI implementation would be to send the message to an LLM, ask it to determine the intent or category, and then update the Case.
Technically, that can work.
Architecturally, I would be uncomfortable with it.
The incoming email is only one piece of the decision. The customer may be replying to a thread that already contains several exchanges. A forwarded message may include quoted content from an earlier issue. The current Case category may reflect a problem that has since changed. The sender, account, entitlement, product, owner, and current Case state may all affect what the system is actually allowed to do.
So I would not design the transaction as:
Email → LLM → Update Case
I would design it more like this:
Inbound Event → Normalize → Assemble Context → AI Judgment → Validate Output → Apply System Action
The normalization step can remove duplicated quoted text, separate the newly written message from historical content, and establish a consistent representation of the conversation.
Context assembly can then add only what the classification decision actually requires: the recent conversational turns, current Case state, relevant CRM attributes, selected metadata, and any business constraints the model needs to understand.
Only then does the workflow reach the Agentic Moment.
The LLM might return something deliberately narrow, for example:
{
"intent": "technical_issue",
"category": "product_support",
"category_changed": true,
"summary": "Customer reports a new product malfunction after the previous billing issue was resolved.",
"confidence": 0.93
}That response is still not permission to modify Salesforce.
The application can validate whether the category is allowed, whether required fields are present, whether the confidence or evidence is sufficient for the intended action, and whether the current Case state permits the transition.
If the result passes those checks, deterministic logic can update the appropriate record, route the Case, write an audit event, or trigger the next workflow step. If it does not, the system can preserve the current state and send the result to a fallback or human-review path.
The LLM makes the judgment. Salesforce remains the system that decides what that judgment is allowed to change.
Context Engineering Is Also a Routing Problem
Once the judgment point is defined, another architectural question appears: what context should the model actually receive?
My answer is rarely “everything we can retrieve.”
In enterprise systems, context is not a single body of knowledge waiting to be searched. Different questions depend on different kinds of evidence. Some answers already exist as authoritative structured data. Others depend on recent external information. Still others require interpretation across documents, meeting notes, previous cases, or accumulated history.
Treating all of those questions as the same retrieval problem usually creates unnecessary cost, noise, and ambiguity.
Suppose a salesperson asks, “What stage is this opportunity in?” The authoritative answer should probably come directly from CRM data. There is little reason to perform semantic search over documents and ask an LLM to infer something the system already knows.
Now consider a different question: “What has changed in this customer’s priorities since our last proposal?” A single database lookup is unlikely to be enough. The workflow may need recent meeting notes, proposal history, relevant emails, and selected CRM state before the model can compare the evidence.
A third question—“What recent market development could affect this deal?”—may require a different route again, using current external information rather than internal RAG.
These are all “questions to AI,” but they should not travel through the same retrieval path.
Good context engineering is therefore partly a routing problem: determine what kind of question is being asked, select the appropriate evidence sources, and assemble only the context needed for that judgment.
This is also why I do not think of RAG simply as “searching more.” Retrieval is useful only when the retrieval strategy matches the decision the system is trying to support.
Workflow Redesign Is the Architecture
Taken together, these patterns lead me to a broader conclusion: enterprise AI architecture is often less about inserting a model and more about redesigning the workflow around a new kind of judgment.
That redesign includes deciding where deterministic control must remain, where probabilistic reasoning is useful, what context must be assembled at that point, how the result should be validated, and what the surrounding application is allowed to do with it.
This is one reason I find the discussion around workflow redesign more useful than model comparisons alone.
McKinsey’s 2025 survey found that, among 25 organizational attributes it examined, workflow redesign had the strongest relationship with reported EBIT impact from generative AI.
MIT Sloan has made a related point from a different direction: rather than treating a business process as one large task for generative AI, organizations can break workflows into individual tasks and evaluate where AI actually performs well, where human verification is required, and where the economics of checking the output still make sense.
I think both observations point to the same architectural reality.
The model is not the operating model.
An enterprise system still needs ownership of permissions, policy, state, evidence, validation, execution, and accountability. The AI capability becomes useful when the workflow is redesigned so that the model receives the right context at a point where judgment genuinely adds value.
That is why I would frame the architecture less like this:
Business Process → AI
and more like this:
Business Process → Bounded Context → AI Judgment → Validation → Controlled Action
The second design may look less “agentic” at first glance. In practice, I find it much closer to what an enterprise can actually operate, audit, improve, and scale.
What I Would Design First
If I were starting an enterprise AI use case from scratch, I would not begin with the model, the agent framework, or even the prompt.
I would begin with four questions.
1. What decision is actually being made?
“Summarize this email” is a capability. “Determine whether this customer is raising a new issue or continuing the existing one” is a decision. The second is much easier to design around because it tells us what the AI is actually responsible for.
2. What context is required for that decision?
This is where data architecture becomes part of AI architecture. The answer may require CRM state, recent conversation history, a policy record, selected documents, current external information, or some combination of them. More context is not automatically better context.
3. Which constraints must remain deterministic?
Permissions, allowed transitions, mandatory fields, budgets, transaction boundaries, compliance rules, and system actions should not become probabilistic simply because an LLM is involved.
4. How will the output be validated before action?
A useful AI response is not necessarily an executable response. The workflow still needs to know what constitutes a valid result, what happens when information is missing, when a human should review the output, and what should remain unchanged when the model is uncertain.
These questions sound less ambitious than asking how to build an autonomous agent. I think that is often an advantage.
They force the architecture to identify the exact point where AI judgment creates value—and everything the surrounding system must do to make that judgment safe, useful, and repeatable.
The Workflow Still Matters
The 95% figure may continue to attract attention, but I do not think the most useful lesson is that enterprise AI “fails.”
The more useful lesson is that a capable model does not remove the need for system design. It makes the quality of that design more important.
Enterprises still need to decide what data is authoritative, what context is relevant, what users are allowed to access, what actions are permitted, and where uncertainty is acceptable.
The LLM belongs inside that architecture—not above it.
AI should not replace the workflow. It should make the right judgment at the right point inside it.
References
- MIT Project NANDA, The GenAI Divide: State of AI in Business 2025, Aditya Challapally, Chris Pease, Ramesh Raskar, and Pradyumna Chari, July 2025. Preliminary findings.
- McKinsey & Company, The State of AI: How Organizations Are Rewiring to Capture Value, March 2025.
- MIT Sloan School of Management, How to Find the Right Business Use Cases for Generative AI, June 2025.