Taking AI Agents into the Business, Part 3: What to Design First in Practice So AI Adoption Doesn't Stall
— What to design now to take AI adoption "into the business" (a three-part series)
Hello! This is the Qualiteg Consulting Team.
Our series "What to design now to take AI adoption into the business" reaches its third and final installment.
In Part 1, we looked at real AI-adoption incidents and saw that the risks of AI agents arise not from mere technical defects, but from the absence of authority and operational design.In Part 2, we examined where responsibility should sit when an incident occurs, and laid out the difficulty of decomposing responsibility from the legal, contractual, and organizational perspectives.
So what, in practical terms, should be designed first in order to keep AI adoption moving forward without stopping it?
In this article — drawing on the shift in quality assurance, the limits of human review, and the changes underway in overseas insurance markets — we lay out the five areas to design before deploying AI agents, and the three questions management must answer first.
1. The quality assurance shift: assume "AI gets things wrong with confidence"
Traditional software quality assurance rested on the idea that, at minimum, the same input could be expected to yield the same result, and that quality is verified along the axes of specification, testing, and reproducibility.
In its December 2025 review, ISACA pointed out that "hallucination is not a quirk but a safety risk: every high-impact AI system should be designed on the assumption that it will sometimes be confidently wrong."
Multiple studies and reviews have flagged that AI, even when wrong, can adopt expressions that appear confident and authoritative. Wrong yet convincing — this property is why quality assurance must change at its foundations.
The figure below shows the difference between traditional quality assurance and quality assurance for AI in operation.
An IBM commentary from January 2026 recommends, as a framework for putting this shift into practice, tracking metrics such as accuracy, drift, context relevance, and cost; capturing reasoning traces in real time; and stress-testing before production.
Quality assurance does not end with model evaluation. It must be designed through to which outputs are monitored and which anomalies count as operational incidents.
2. The pitfall of human review: automation bias
"As long as a human checks the AI's output, we're safe" — this idea, too, has its limits.
A systematic review published in AI & Society in July 2025 (Romeo & Conti, 2025), reviewing 35 peer-reviewed studies, concluded that explainable AI (XAI) and transparency mechanisms — when the explanations are overly technical or, conversely, too simplistic — can reinforce misplaced trust, particularly among practitioners with low AI literacy. The finding that more explanation does not necessarily improve judgment accuracy carries important implications for how review structures should be designed.
A paper published in the European Journal of Risk Regulation in July 2025 analyzes that Article 14(4)(b) of the EU AI Act refers explicitly to automation bias. This is a regulatory context premised on the EU's high-risk AI systems, but for Japanese companies as well, it is instructive as a design argument for keeping human review from becoming a mere formality of approval. That regulation has begun to recognize the limits of human review means companies, too, must go as far as designing the review itself — frequency, method, reviewer selection and training, and routing rules keyed to the AI's confidence level.
3. Insurance-market restructuring abroad: operational controls and documentation matter more in Japan too
The insurance industry's moves reflect the nature of AI risk with unusual clarity.
The industry's moves show that AI risk is beginning to be treated as a risk that existing insurance products struggle to accommodate. Note, however, that the developments introduced below are mainly cases from overseas markets such as the US and UK. Insurance products covering generative-AI risk have appeared in Japan as well, but at this point it cannot be said that broadly excluding AI risk from existing policies has become the general practice here.
Overseas, two movements are advancing in parallel: insurers working to carve AI-related losses out of existing coverage, and the development of new insurance products targeting AI-specific risks. According to an analysis by Hunton Andrews Kurth published on the Harvard Law School Forum on Corporate Governance, insurers are concerned that AI-related losses may be covered by existing policies and are moving to introduce AI-specific exclusion clauses. According to another article from the same firm, Berkley introduced the first "Absolute" AI exclusion clause.
Meanwhile, according to a report by Hunton Andrews Kurth, the AI liability insurance offered by Armilla Insurance Services was reported in spring 2025 as a product underwritten by Lloyd's insurers. It covers AI-specific risks such as hallucination, model performance degradation, and algorithmic failure.
What deserves attention is the emerging view that, at least in underwriting and coverage discussions, the maturity of operational controls and documentation is being weighted more heavily than before.
In Japan as well, insurance products have appeared covering intellectual-property infringement, information leaks, and hallucination-driven defamation arising from generative-AI use. At this point, however, the kind of broad AI exclusion clauses and full-scale restructuring of the AI liability insurance market seen overseas cannot be said to have become general practice in Japan. That is precisely why what matters for Japanese companies is not "letting insurance handle it," but putting in place — ahead of time — the operational controls, logs, and documentation that can withstand insurance and compliance scrutiny.
4. Building governance: the six pillars
The trends in quality assurance, human review, and insurance covered so far all point in the same direction. The greatest risk in operating AI agents is not the technology's imperfection, but governance left undesigned.
Based on a commentary published by Mayer Brown in February 2026, a practical AI governance framework can be organized into six major components. The figure below visualizes them.
What the commentary emphasizes above all is the importance of documentation. Holding auditable records helps demonstrate, in a regulatory investigation, that the organization acted reasonably. Conversely, without such documents, your defensive position weakens considerably.
5. The five areas to design in practice
What management must decide first is: which tasks to delegate, up to what level of authority, who owns the exceptions, and how much failure cost is acceptable. Proceed to a PoC with these left vague, and the adoption decision itself starts to wobble — before implementation success or failure even enters the picture.
Nor is responsibility decomposition settled by organizational theory and contract theory alone. In particular — which inputs and outputs to audit, which information to block or mask, which model usage to permit, which operations to route back to human approval, and how far the reasoning path can be reconstructed in an anomaly — these are also questions of platform design and operational design.
The figure below organizes the five areas that should be designed before deployment.
First: authority design.
The starting point is to define explicitly what the agent may and may not do. As the Lemkin case in Part 1 showed, the risk of "doing the forbidden thing during the forbidden window" arises from deficient authority design.
Second: ensuring traceability.
As we saw in Part 2, if the chain of decisions cannot be reconstructed after the fact, neither root-cause analysis nor responsibility decomposition is possible. IBM's recommended "real-time capture of reasoning traces" is one practical approach to this requirement.
Third: designing quality monitoring.
As ISACA points out, standing on the premise that "AI gets things wrong with confidence" requires a design that includes periodic benchmarks, sampling inspections, threshold alerts, and fallback procedures.
Fourth: contractualizing the division of responsibility among stakeholders.
The AI-specific issues organized in Part 2 — kill-switch clauses, shadow-mode clauses, retraining-window clauses, and contributory-negligence frameworks, all under active discussion and adoption in advanced contracting practice — should be built into your contracts to fit your own deployment.
Fifth: connecting to insurance and compliance.
Now that AI-specific risk assessment is beginning in the insurance market, the maturity of your operational controls and documentation can influence insurance discussions. In Japan, this means building into contracts and operations a response that takes into account the AI Act and the AI Business Operator Guidelines — and, in finance, the FSA's AI discussion paper — alongside the EU AI Act, US state laws, and industry-specific regulation.
The three questions management must answer first
A company considering AI agent adoption needs to answer at least three questions in advance. First, how much loss can this business process tolerate when a malfunction occurs? Second, where do we draw the boundary of the authority delegated to the AI? Third, have we built the ability to trace causes when an incident occurs and explain them inside and outside the company?
As long as these three remain vague, adoption tends to stall midway — no matter how accurate the model selection or the PoC.
The real contest is not the design, but embedding it in operations
Even if a design document exists — if authority settings are vague, logs are not being captured, the review process is not running, and frontline training has not happened — governance effectively does not exist.
The success of AI agent adoption is determined not by whether a design exists, but by whether the design is embedded in operations.
Series conclusion: AI adoption is not tool adoption — it is the design of responsibility and control
As we have seen across all three installments, deploying AI agents is not completed by evaluating model performance alone.
In Part 1, real incidents showed that the problem with AI lies not in the technology's imperfection but in the absence of design. In Part 2, we examined, from the legal, contractual, and organizational perspectives, why responsibility decomposition is hard and how contracts and operations can compensate for that difficulty. In Part 3, building on the quality assurance shift, the limits of human review, and the restructuring of the insurance market, we presented the five areas to design first in practice and the three questions management must answer.
The question of AI agents is not merely "is the AI smart?" It lies in how you design authority, auditability, contracts, operational controls, and responsibility decomposition. Deploying an AI agent is no longer a mere PoC — it is business process design itself, encompassing authority design, auditability, contracts, and operational controls.
Thinking about technical implementation and governance design as one — not as separate exercises — will be what matters in the AI adoption ahead.
Where Qualiteg can walk alongside you, from concept to operational adoption
In this installment, we organized what should be designed first in practice so that AI adoption does not stall. Taking into account the quality assurance shift, the limits of human review, and the changes in the insurance market, it becomes clear that deploying AI agents is no longer a mere PoC but business process design encompassing concept, design, control, and operations.
What matters most is connecting authority design, traceability, quality monitoring, contractual division of responsibility, and the link to insurance and compliance — not as scattered topics, but as a single adoption architecture. A design document alone is not enough; only when it is embedded in operations does AI move from the "trial" stage to the "in the business" stage.
The topics we have covered throughout this series are exactly what should be designed in advance for that purpose.
Qualiteg supports AI adoption end to end — concept, design, implementation, and operational adoption — so that it does not end at the "trial" stage but truly lands in the business, drawing on both the implementation expertise gained from developing and operating our own AI platform and our strategy and operations consulting spanning AI Transformation, BPR, and new business development.
In deploying AI agents, what matters is connecting not just model selection and PoCs, but business process redesign, authority design, auditability, information management, quality management, and ROI visibility. Qualiteg approaches these challenges with an organization that does not separate strategy from technology.
If this speaks to your situation, please feel free to reach out.
From organizing the initial concept and designing governance to evaluating the execution platform and embedding operations after deployment, we will work with you according to your needs.
