Making AI Agents Business-Ready [Part 1]

What AI Deployment Incidents Reveal

— What to design now to make AI adoption business-ready (a three-part series)

Hello! This is the Qualiteg Consulting Team.

Even as more companies adopt AI agents,

the difficulty of moving from the "trying it out" stage to the "running it in the business" stage

is coming into sharp focus.

In this series, we frame AI agent adoption not merely as a technology question but as a problem of business design — one that encompasses the division of responsibility, auditability, contracts, and operational controls.

Across the three installments, we will think not about "whether the AI is smart" but about "what must be designed to put AI to work in the business."

In this first installment, starting from two incidents that occurred in 2025, we look at why "accountability design" has become the issue of the moment.

The figure above shows the full landscape of topics this series covers.

Deploying an AI agent is not something you complete through technical model selection alone; it requires design that extends to authority, contracts, auditing, quality monitoring, insurance, and incident response.

In Part 1, we begin by looking at real incidents to understand why this kind of design has come to be required.

Note that the emblematic cases this series draws on are mostly from outside Japan.
That does not mean the same problems do not exist in Japan. On the contrary, as AI agent adoption gathers pace, issues such as authority design, quality assurance, auditability, and the division of responsibility are just as important for Japanese companies. For now, overseas reporting offers the most accessible, well-documented examples of these structural problems surfacing at scale, so this series uses those cases as its guideposts.

Two incidents caused by AI agents

Case 1: Replit — a production database deleted, then covered up

In 2025, SaaStr founder Jason Lemkin had his production database deleted by an AI coding agent (Replit) running on his own infrastructure. It happened during a designated "code and action freeze" period.

TechTarget's reporting describes how the agent fabricated reports and data, and even generated a deception algorithm to "make it look like things were still working." It did what it was not supposed to do, during a period when it was not supposed to act — and then concealed it. This is a case where the absence of permission controls, environment separation, and approval design turned an AI's deviant behavior into real damage.

Case 2: Deloitte — a government report riddled with fabricated citations

The same year, it came to light that a 237-page welfare compliance review report Deloitte had delivered to the Australian government was full of references to nonexistent scholars and nonexistent papers. The discovery began when University of Sydney researcher Chris Rudge noticed the fabricated citations.

Deloitte later acknowledged it had used Azure OpenAI GPT-4o to produce part of the report, and submitted a corrected version. Serious errors, including fabricated citations, had crept in during the GPT-4o-assisted drafting process — a case showing that disclosure of AI use and quality assurance can become real points of contention in practice (Business Standard, October 2025).

Two incidents, two different risks

The two cases introduced above differ in nature.

The figure below organizes which risk each case symbolizes.

A text-based comparison table follows.

The Replit case is, at heart, a question of what the AI should be allowed to do — a problem of authority design — while
the Deloitte case is a question of how to verify and disclose what the AI produced — a problem of quality assurance.

In both cases, the problem lay not in the AI's own performance but in the design surrounding how the AI was embedded into the business.

The common essence: an absence of design

What these two cases have in common is not so much that the AI erred, but that when the AI was embedded into the business,

• who holds how much authority
• what gets monitored
• where control returns to humans when something goes wrong

— all of this had never been designed.

The flowchart below shows
how this "absence of design" leads to deployments being halted.

An absence of design halts deployment projects regardless of how well the AI performs.

A December 2025 analysis by BCG cites the AI Incidents Database, which shows AI-related incidents increased 21% from 2024 to 2025.

Likewise, ISACA's December 2025 review concluded,
of the biggest AI failures of 2025, that they were
"not technical. They were organizational: weak controls, unclear ownership, and misplaced trust"
.

The essence of the problem is not AI accuracy.

It lies in the design of authority, auditing, responsibility, and human intervention points when AI is embedded into business operations.

Where Qualiteg can walk alongside you, from concept to operational adoption

In this installment, using real incidents from AI agent deployments as our guide, we saw that the essence of the problem is not mere model performance but something else:
authority design, quality assurance, auditability, human intervention points — in short, the absence of design in these areas.

When you embed AI into business operations, the question is not only "which model to use."

How far authority is delegated; where it stops; what is monitored; how control is handed back when something goes wrong.

Without that foundational design, you may get through a PoC, but you are likely to stall at the stage of putting it into the business.

These are questions it is too late to consider after an incident; they need to be structurally organized from the earliest stage of adoption.

Drawing on the implementation expertise gained from developing and operating our own AI platform, together with strategy and business consulting spanning AI transformation, BPR, and new business development, we provide end-to-end support — concept, design, implementation, and operational adoption — so that AI initiatives do not end at the "trial" stage but are truly put to work in the business.

Deploying AI agents is not just about model selection and PoCs; it is important to connect business process redesign, authority design, auditability, information management, quality control, and ROI visibility. Qualiteg addresses these challenges with a structure that does not separate strategy from technology.

If this resonates with you, please feel free to reach out. From organizing the initial concept to governance design, evaluating the execution platform, and post-deployment operational adoption, we can engage according to your needs.

Contact

https://qualiteg.com/contact?inquiry=consulting_business

Coming up next

As the cases in this installment suggest, when such incidents occur, how might legal and contractual responsibility be pursued?

And why is it that responsibility is harder to divide for AI agents than for conventional software?

In Part 2, we take a structural look at why dividing responsibility for AI agents is so difficult, from the legal, contractual, and organizational perspectives.

See you in the next installment!