<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title><![CDATA[Qualiteg Journal]]></title><description><![CDATA[Engineering insights, AI expertise, and the latest news from Qualiteg]]></description><link>https://journal.qualiteg.com/</link><image><url>https://journal.qualiteg.com/favicon.png</url><title>Qualiteg Journal</title><link>https://journal.qualiteg.com/</link></image><generator>Ghost 5.82</generator><lastBuildDate>Sun, 15 Nov 2026 22:51:05 GMT</lastBuildDate><atom:link href="https://journal.qualiteg.com/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[[Confession] I Typed /context Every Time I Wondered How Much Context Was Left: Keeping Remaining Context and Usage Limits on the Claude Code Status Line]]></title><description><![CDATA[You don't need to type /context every time you want to know how much context is left in Claude Code. A script of under 40 lines, running on Windows with nothing but Node.js, keeps your context usage and the remaining 5-hour and weekly limits on the status line.]]></description><link>https://journal.qualiteg.com/claude-code-statusline-context-usage/</link><guid isPermaLink="false">6afa28b5433723f302e289ed</guid><category><![CDATA[ClaudeCode]]></category><category><![CDATA[LLM]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Sun, 15 Nov 2026 06:13:25 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/11/claude-code-statusline-context-usage-cover-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/11/claude-code-statusline-context-usage-cover-en.png" alt="[Confession] I Typed /context Every Time I Wondered How Much Context Was Left: Keeping Remaining Context and Usage Limits on the Claude Code Status Line"><p>Hello!</p><p>When a Claude Code session runs long, you start to wonder how much context is left.</p><p>So you type <code>/context</code>...</p><p>Out comes the usage breakdown table, followed by the MCP tools table, custom agents, memory files, and the list of skills. Dozens of lines scroll past.<br>And then you scroll back up to find the top...</p><h2 id="all-i-want-to-know-is-what-percent-of-my-context-is-left">&quot;All I want to know is what percent of my context is left!&quot;</h2><p>Yes. The one line you wanted was what percent is left.</p><p>The short answer is that <u>you don&apos;t need to type <code>/context</code> every time</u>.</p><p>Claude Code has a status line, a single line of your choosing that stays at the bottom of the screen, and it can show your context usage at all times.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/image-1.png" class="kg-image" alt="[Confession] I Typed /context Every Time I Wondered How Much Context Was Left: Keeping Remaining Context and Usage Limits on the Claude Code Status Line" loading="lazy"><figcaption>The status line on our machine, from the version of the script with Japanese labels. It reads: ctx 30% (300k/1M), 70% left | week, 79% left (resets 11/20 16:00)</figcaption></figure><p>The remaining 5-hour and weekly usage limits fit on the same line.</p><p>This is not a new trick.</p><p>On the status line page of the official docs, the very first example already includes context usage.</p><p>Even so, we keep seeing people type <code>/context</code> at regular intervals.</p><p>In this article we use a script of under 40 lines, running on Windows with nothing but Node.js, to print this one line</p><p><strong>ctx 25% (250k/1M) 75% left | week 82% left (resets 11/20 16:00)</strong></p><p>with the code we actually ran. We also cover the places where Windows trips you up, why the number can differ from <code>/context</code>, and why the per-model weekly limit cannot be shown on the status line.</p><p>Our environment was Windows 11, Claude Code 2.1.325, and Node.js v24.12.0.</p><h2 id="context-is-a-tool-for-the-breakdown-not-for-a-glance-at-whats-left">/context is a tool for the breakdown, not for a glance at what&apos;s left</h2>
<!--kg-card-begin: html-->
<figure style="margin:24px auto 0;max-width:540px;"><img src="https://journal.qualiteg.com/content/images/2026/11/claude-code-statusline-manga-1-v1-en-lab.png" alt="[Confession] I Typed /context Every Time I Wondered How Much Context Was Left: Keeping Remaining Context and Usage Limits on the Claude Code Status Line" style="width:100%;height:auto;border:1px solid #D5DDE6;border-radius:10px;display:block;"><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin:8px 0 0;line-height:1.6;">Typing /context just to see what&apos;s left is long, because the breakdown tables come with it</figcaption></figure>
<div style="max-width:540px;margin:14px auto 28px;border:1px solid #D5DDE6;background:#fff;border-radius:10px;padding:10px 14px;font-size:13px;"><ul style="margin:0;padding:0;"><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:6.5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">&quot;I type /context every time I want to see how much context I have left.&quot;</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:6.5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Senior</span><span style="flex:1 1 auto;line-height:1.7;">&quot;/context is for inspecting the breakdown. It&apos;s too long for a quick look.&quot;</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:6.5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Senior</span><span style="flex:1 1 auto;line-height:1.7;">&quot;The status line can keep what&apos;s left on screen, in a single line.&quot;</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:6.5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">&quot;So I read the bottom line, and keep /context for the breakdown.&quot;</span></li></ul></div>
<!--kg-card-end: html-->
<p>What <code>/context</code> returns is a breakdown of what is in the context. A table shows how many tokens go to the system prompt, tool definitions, MCP tools, memory files, skills, and messages. In our environment it was followed by a per-server MCP tools table and tables for custom agents, memory files, and skills.</p><p>When the question is &quot;what is eating my context,&quot; this is exactly what you need.</p><p>When you only want a quick look in the middle of work, it is too much. Each time, the conversation scrolls by dozens of lines and the output you were just reading disappears off the top.</p><p>Keep what&apos;s left always in view, and open the breakdown with <code>/context</code> only when you need it. The status line makes that split possible.</p><h2 id="the-status-line-just-takes-json-and-returns-one-line">The status line just takes JSON and returns one line</h2><p>The mechanism is simple. Put one command under <code>statusLine</code> in <code>settings.json</code>, and Claude Code runs it and pipes the session information to its standard input as JSON. Whatever the command writes to standard output appears at the bottom of the screen.</p><p>According to the official docs, the status line runs only on your local machine and uses no API tokens. It runs when a new response arrives, among other triggers, and rapid changes are grouped into a single run per 300 milliseconds.</p><p>Of the JSON fields passed in, these are the ones this article uses (excerpted from the table in the official docs).</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Field</th><th>What it holds</th></tr></thead><tbody><tr><td><code>context_window.used_percentage</code></td><td>Context usage as a percentage (pre-calculated)</td></tr><tr><td><code>context_window.remaining_percentage</code></td><td>Percentage remaining (pre-calculated)</td></tr><tr><td><code>context_window.context_window_size</code></td><td>Maximum context size in tokens. 200000 by default, 1000000 for models with extended context</td></tr><tr><td><code>context_window.total_input_tokens</code></td><td>Input-side tokens currently in the context (as of the latest response)</td></tr><tr><td><code>context_window.current_usage</code></td><td>Token breakdown of the latest API call</td></tr><tr><td><code>rate_limits.five_hour.used_percentage</code>, <code>rate_limits.seven_day.used_percentage</code></td><td>Usage of the 5-hour and 7-day limits (0 to 100)</td></tr><tr><td><code>rate_limits.five_hour.resets_at</code>, <code>rate_limits.seven_day.resets_at</code></td><td>When each limit resets (Unix seconds)</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Because the percentage arrives pre-calculated, the script has no division to do. It only picks values up and lines them up.</p><h2 id="a-script-of-under-40-lines-prints-the-remaining-context-and-usage-limits">A script of under 40 lines prints the remaining context and usage limits</h2><p>The examples in the official docs are written in Bash with <code>jq</code>. On Windows, <code>jq</code> is a separate install, so we wrote ours in Node.js, which is almost always there wherever Claude Code is.</p><p>statusline-context.mjs (complete)</p><pre><code class="language-js">// Show the remaining context and the remaining usage limits (5-hour and weekly) on one Claude Code status line
let s = &quot;&quot;;
process.stdin.on(&quot;data&quot;, (d) =&gt; (s += d));
process.stdin.on(&quot;end&quot;, () =&gt; {
  try {
    const data = JSON.parse(s);
    const parts = [];

    // Context: print it only when the usage percentage arrives as a number
    const cw = data.context_window || {};
    if (typeof cw.used_percentage === &quot;number&quot;) {
      const used = cw.used_percentage;
      const size = cw.context_window_size;
      const k = (n) =&gt; (n &gt;= 1e6 ? (n / 1e6).toFixed(n % 1e6 ? 1 : 0) + &quot;M&quot; : Math.round(n / 1e3) + &quot;k&quot;);
      const tokens = typeof size === &quot;number&quot; ? ` (${k((size * used) / 100)}/${k(size)})` : &quot;&quot;;
      parts.push(`ctx ${Math.round(used)}%${tokens} ${Math.round(100 - used)}% left`);
    }

    // Usage limits: add only the windows that arrived. Time only if the reset is today, month/day too otherwise
    const pad = (n) =&gt; String(n).padStart(2, &quot;0&quot;);
    const windowPart = (label, w) =&gt; {
      if (!w || typeof w.used_percentage !== &quot;number&quot;) return;
      let reset = &quot;&quot;;
      if (typeof w.resets_at === &quot;number&quot;) {
        const d = new Date(w.resets_at * 1000);
        const hm = `${pad(d.getHours())}:${pad(d.getMinutes())}`;
        const today = d.toDateString() === new Date().toDateString();
        reset = ` (resets ${today ? hm : `${d.getMonth() + 1}/${d.getDate()} ${hm}`})`;
      }
      parts.push(`${label} ${Math.max(0, Math.round(100 - w.used_percentage))}% left${reset}`);
    };
    const rl = data.rate_limits || {};
    windowPart(&quot;5h&quot;, rl.five_hour);
    windowPart(&quot;week&quot;, rl.seven_day);

    process.stdout.write(parts.join(&quot; | &quot;));
  } catch {}
});</code></pre><p>It does four things. It reads standard input to the end, takes <code>context_window.used_percentage</code>, adds whichever limits arrived under <code>rate_limits</code>, and joins the parts with a vertical bar (<code>|</code>) into one line.</p><p>When the context percentage is not a number, that part is left out. If the usage limits arrived, only those are printed. The official docs say <code>used_percentage</code> and <code>remaining_percentage</code> can be null early in a session. Printing 0% there would make &quot;not known yet&quot; look the same as &quot;actually empty.&quot;</p><p>The token counts in parentheses are an estimate, the maximum multiplied by the percentage. &quot;224k/1M&quot; next to &quot;22%&quot; gives a better feel for how much room is left than the percentage alone, so we added it.</p><p>For the usage limits, the script adds only the windows that arrived. According to the official docs, <code>rate_limits</code> is sent when you are on a claude.ai subscription such as Pro or Max, and only after the first response of the session. The 5-hour and 7-day windows can each be missing on their own, and a window whose reset time has passed stops arriving.</p><p>In our environment too, the <code>rate_limits</code> in the JSON we received contained only <code>seven_day</code>, with no <code>five_hour</code>. Deciding up front to skip missing windows keeps the line intact whatever combination arrives.</p><p>Reset times arrive in Unix seconds, so the script converts them to local clock time. A window that resets today shows the time only, and any other day shows the month and day as well.</p><p>The whole thing is wrapped in <code>try</code> and <code>catch</code> so that a failing status line script never gets in the way of your work.</p><h2 id="before-installing-run-it-against-sample-json">Before installing, run it against sample JSON</h2><p>A status line script is nothing more than a program that reads standard input and returns one line. You can check it on its own with sample JSON before connecting it to Claude Code. The official docs recommend the same.</p><p>In Git Bash, pipe it in like this.</p><pre><code class="language-bash">echo &apos;{&quot;context_window&quot;:{&quot;used_percentage&quot;:22.4,&quot;context_window_size&quot;:1000000}}&apos; | node statusline-context.mjs</code></pre><p>Our result:</p><pre><code>ctx 22% (224k/1M) 78% left</code></pre><p>We also tried input for a model with a 200000 maximum.</p><pre><code class="language-bash">echo &apos;{&quot;context_window&quot;:{&quot;used_percentage&quot;:8,&quot;context_window_size&quot;:200000}}&apos; | node statusline-context.mjs</code></pre><pre><code>ctx 8% (16k/200k) 92% left</code></pre><p>With a <code>null</code> percentage and with empty JSON (<code>{}</code>), nothing was printed and there was no error.</p><p>Next, input that includes usage limits. First, with both the 5-hour and the 7-day window.</p><pre><code class="language-bash">echo &apos;{&quot;context_window&quot;:{&quot;used_percentage&quot;:25,&quot;context_window_size&quot;:1000000},&quot;rate_limits&quot;:{&quot;five_hour&quot;:{&quot;used_percentage&quot;:23.5,&quot;resets_at&quot;:1794735000},&quot;seven_day&quot;:{&quot;used_percentage&quot;:18,&quot;resets_at&quot;:1795158000}}}&apos; | node statusline-context.mjs</code></pre><pre><code>ctx 25% (250k/1M) 75% left | 5h 77% left (resets 11/15 18:30) | week 82% left (resets 11/20 16:00)</code></pre><p>How the reset times look depends on the day and the time zone you run it in. The output above is from November 16, 2026, Japan time. Run on the 15th, the same input gives <code>(resets 18:30)</code> for the 5-hour window, the time only.</p><p>With only the 7-day window, we got the following. This is the same shape as what actually arrived in our environment.</p><pre><code class="language-bash">echo &apos;{&quot;context_window&quot;:{&quot;used_percentage&quot;:25,&quot;context_window_size&quot;:1000000},&quot;rate_limits&quot;:{&quot;seven_day&quot;:{&quot;used_percentage&quot;:18,&quot;resets_at&quot;:1795158000}}}&apos; | node statusline-context.mjs</code></pre><pre><code>ctx 25% (250k/1M) 75% left | week 82% left (resets 11/20 16:00)</code></pre><h2 id="add-one-entry-to-settingsjson">Add one entry to settings.json</h2><p>Put the script in <code>C:\Users\&lt;you&gt;\.claude\</code> and add this one entry to <code>settings.json</code> in the same folder.</p><p>settings.json (only the part to add)</p><pre><code class="language-json">&quot;statusLine&quot;: {
  &quot;type&quot;: &quot;command&quot;,
  &quot;command&quot;: &quot;node \&quot;C:/Users/&lt;you&gt;/.claude/statusline-context.mjs\&quot;&quot;
}</code></pre><p>There are two Windows-specific points here.</p><p><strong>Separate the path in the command with forward slashes, not backslashes.</strong></p><p>According to the official docs, Claude Code on Windows runs the status line command through Git Bash when Git Bash is installed. Git Bash treats the backslash as an escape character, so a path written as <code>C:\Users\...</code> reaches the script with its separators gone, and nothing appears, with no error shown either. If you set it up and nothing shows, look here first.</p><p>The other point is to quote the path. In a folder whose user name contains a space, an unquoted path splits into two arguments at the space and the script is not found. We checked this with a temporary home folder whose name contains a space. The quoted command displayed the line, and the unquoted one stopped with a <code>Cannot find module</code> error.</p><p>If adding it by hand is a chore, you can also type <code>/statusline</code> in Claude Code and describe what you want in plain words. According to the official docs, Claude Code creates the script under <code>~/.claude/</code> and updates the settings for you. We did not use that route this time, because we use a hook to stop AI agents from writing under <code>~/.claude/</code>. Instead we prepared a small script that a person runs once.</p><p>install-statusline-context.mjs (complete)</p><pre><code class="language-js">// Install the status line that shows remaining context and usage limits: copy the script into ~/.claude and add statusLine to settings.json
import fs from &quot;fs&quot;;
import os from &quot;os&quot;;
import path from &quot;path&quot;;
import {fileURLToPath} from &quot;url&quot;;

const here = path.dirname(fileURLToPath(import.meta.url));
const claudeDir = path.join(os.homedir(), &quot;.claude&quot;);
const target = path.join(claudeDir, &quot;statusline-context.mjs&quot;);
const settingsPath = path.join(claudeDir, &quot;settings.json&quot;);

const raw = fs.existsSync(settingsPath) ? fs.readFileSync(settingsPath, &quot;utf-8&quot;) : &quot;{}&quot;;
const settings = JSON.parse(raw.replace(/^\uFEFF/, &quot;&quot;));

// Never overwrite what is already there. Check before writing anything, and stop
if (settings.statusLine) {
    console.log(&quot;statusLine is already set. Nothing changed:&quot;, JSON.stringify(settings.statusLine));
    process.exit(1);
}
if (fs.existsSync(target)) {
    console.log(target + &quot; already exists. Nothing changed&quot;);
    process.exit(1);
}

fs.mkdirSync(claudeDir, {recursive: true});
fs.copyFileSync(path.join(here, &quot;statusline-context.mjs&quot;), target);
if (fs.existsSync(settingsPath)) fs.writeFileSync(settingsPath + &quot;.bak-before-statusline&quot;, raw);
// Use forward slashes, and quote the path so a folder name with a space still arrives as one argument
settings.statusLine = {type: &quot;command&quot;, command: `node &quot;${target.replace(/\\/g, &quot;/&quot;)}&quot;`};
const indent = (raw.match(/\n([ \t]+)&quot;/) || [null, &quot;  &quot;])[1];
fs.writeFileSync(settingsPath, JSON.stringify(settings, null, indent) + &quot;\n&quot;);
console.log(&quot;OK: copied &quot; + target + &quot; and added statusLine to &quot; + settingsPath);</code></pre><p>Put the two files in the same folder and run <code>node install-statusline-context.mjs</code> once. If <code>statusLine</code> is already set, or a script with the same name is already in place, it stops without writing anything. It creates the <code>.claude</code> folder if it is missing, and keeps the original <code>settings.json</code> as <code>settings.json.bak-before-statusline</code> if there was one. Converting the path separators to forward slashes and quoting the path also happen inside this script.</p><p>We ran it against a temporary home folder with a space in its name and confirmed three things. The <code>statusLine</code> entry shown above goes into <code>settings.json</code>. Running it again once it is set up changes nothing. And the entry is added to a <code>settings.json</code> that already holds other settings.</p><h2 id="move-a-long-command-to-a-short-path-before-running-it">Move a long command to a short path before running it</h2><p>There was one more thing we were careful about during the install.</p><p>At first the install script sat deep inside a temporary folder, and the command to run it came to about 140 characters. In our environment, on a different task, a long pasted command once had a line break entered partway through, and only the first line ran.</p><p>To avoid the same thing, we copied the script to a folder with a short path and kept the command in the 60-character range.</p><p>Path separators need the same kind of care. In our environment, a command run from the Claude Code input box with a leading <code>!</code> was executed by Git Bash. When we wrote the script path with backslashes, it went looking for a path with the separators gone and stopped with a <code>Cannot find module</code> error. Rewriting the same path with forward slashes went straight through.</p><p>When we hand someone a command to run, we look at its length before its content. If it is getting long, we move the file to a short path first.</p><h2 id="the-per-model-weekly-limit-does-not-reach-the-status-line">The per-model weekly limit does not reach the status line</h2><p>The usage screen on claude.ai sometimes shows a limit for one specific model, separate from the overall weekly limit. On our screen the Fable limit sits below the overall one. It is tempting to put that on the status line too.</p><p>But the contents of <code>rate_limits</code> in the official docs are <code>five_hour</code>, <code>seven_day</code>, and <code>spend_limit</code> for use through a gateway, three in all. A per-model limit is not listed.</p><p>We saved the JSON that actually reached the status line command in our environment to a file and looked inside. On Claude Code 2.1.325, <code>rate_limits</code> contained only <code>seven_day</code>. There was no field for a per-model limit.</p><p>Some external tools fetch the value from an undocumented endpoint and display it. We did not do that. For what&apos;s left per model, we keep using <code>/usage</code> as before. If a field is added to the official JSON, one more line in the script will put it on the same status line.</p><h2 id="the-number-differs-from-context-because-it-is-counted-at-a-different-moment">The number differs from /context because it is counted at a different moment</h2><p>If you type <code>/context</code> after setting up the status line, the numbers can differ a little.</p><p>This is not a bug. According to the official docs, the status line receives the count as of the latest API response, while <code>/context</code> adds an estimate of the messages added since that response. So until the next response arrives, <code>/context</code> can show the larger number.</p><p>One more point: <code>used_percentage</code> is calculated from input-side tokens only. It is the sum of input tokens, cache writes, and cache reads, and does not include output tokens.</p><p>Think of the status line number as a reading for a quick glance and <code>/context</code> as the detailed table with the breakdown, and the gap stops being confusing.</p><p>The official docs also note that once you configure your own status line, most of the keyboard hints at the bottom of the screen (such as <code>esc to interrupt</code>) are no longer shown. If you rely on those hints, it is worth knowing before you install.</p><h2 id="if-you-want-something-more-elaborate-ready-made-tools-exist">If you want something more elaborate, ready-made tools exist</h2><p>Besides context usage and usage limits, the same JSON carries the model name, the git branch, the cost so far, and more. If you want to line up many items, there are options other than writing it yourself.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Option</th><th>What you need</th><th>When it fits</th></tr></thead><tbody><tr><td><code>/statusline</code> with a plain-language request (built-in feature)</td><td>Claude Code only</td><td>When you want to try it first</td></tr><tr><td>ccstatusline, starship-claude (community tools introduced in the official docs)</td><td>Installing each tool</td><td>When you want themes and many items</td></tr><tr><td>Write a few dozen lines yourself (this article)</td><td>Node.js</td><td>When you want to show only a few things and know everything that is in there</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>We did not use a ready-made tool this time. All we wanted to show was the remaining context and usage limits, so under 40 lines was enough.</p><h2 id="summary">Summary</h2><p>Keep <code>/context</code> as the tool for inspecting the breakdown, and leave what&apos;s left on the status line. With the remaining usage limits on the same line, you open <code>/usage</code> less often too. Once it is in, there is no command to type and no scrolling screen.</p><p>Here are the places where things tend to go wrong.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>What happens</th><th>Why, and what to do</th></tr></thead><tbody><tr><td>Nothing shows after setup</td><td>On Windows, separate the path in the command with forward slashes and quote it. With backslashes it fails with no error</td></tr><tr><td>Nothing shows only at the start of a session</td><td>The percentage can be <code>null</code> until the first response arrives. The script in this article prints nothing during that time</td></tr><tr><td>The number differs from <code>/context</code></td><td>They are counted at different moments. They converge when the next response arrives</td></tr><tr><td>A pasted command gets cut off</td><td>A long pasted command once had a line break entered partway through. Move the file to a short path and shorten the command before running it</td></tr><tr><td>The remaining usage limits do not show</td><td><code>rate_limits</code> arrives on a subscription, after the first response. Individual windows can also be missing. The script in this article prints only the windows that arrived</td></tr><tr><td>A command run with <code>!</code> cannot find the file</td><td>When it runs in Git Bash, backslash separators vanish. Use forward slashes in the path</td></tr><tr><td>The per-model weekly limit cannot be shown</td><td>The JSON that reaches the status line has no such field. Check it with <code>/usage</code></td></tr></tbody></table>
<!--kg-card-end: html-->
<p>See you next time!</p><h2 id="sources-and-references">Sources and references</h2><ul><li><a href="https://code.claude.com/docs/en/statusline?ref=journal.qualiteg.com">Customize your status line (Claude Code official docs)</a></li><li><a href="https://github.com/sirmalloc/ccstatusline?ref=journal.qualiteg.com">ccstatusline (GitHub)</a></li><li><a href="https://github.com/martinemde/starship-claude?ref=journal.qualiteg.com">starship-claude (GitHub)</a></li></ul><h2 id="related-articles">Related articles</h2><ul><li><a href="https://journal.qualiteg.com/claude-opus-5-claude-code-guide/">Claude Opus 5.0 Complete Guide: Model Specs, API, and Claude Code Operations (Qualiteg Journal)</a></li></ul>]]></content:encoded></item><item><title><![CDATA[What Are Claude's Limit Resets? What a Fable Reset and a Weekly Reset Give Back, and How They Differ from ChatGPT's]]></title><description><![CDATA[Claude handed out limit resets to paid users for the Fable 5.2 launch. What a Fable reset and a weekly reset give back, how many you get, when they expire, what to check before pressing, and how they differ from ChatGPT's resets, from the official help pages and real screens.]]></description><link>https://journal.qualiteg.com/claude-limit-reset-fable-weekly-reset/</link><guid isPermaLink="false">6af43e710a0f0d13c338b756</guid><category><![CDATA[ClaudeCode]]></category><category><![CDATA[LLM]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Wed, 11 Nov 2026 10:00:23 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/11/claude-limit-reset-fable-weekly-reset-cover-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/11/claude-limit-reset-fable-weekly-reset-cover-en.png" alt="What Are Claude&apos;s Limit Resets? What a Fable Reset and a Weekly Reset Give Back, and How They Differ from ChatGPT&apos;s"><p>Hello! This is the Qualiteg Product Development Team.</p><p>To mark the release of Claude Fable 5.2, Anthropic handed out <strong>limit resets</strong> to users on paid plans.<br>We received quite a few ourselves.</p><p>In this article we walk through what these resets are.</p><p>A limit reset is <strong>the right to refill usage you have already spent, on the spot, without waiting for the next scheduled reset.</strong><br>There are several kinds, and each kind refills a different scope. A Fable reset, for example, gives back only what you used on Fable.</p><p>We cover what each kind refills and what it leaves alone, how many you get, the expiry dates and caveats, and when it makes sense to use one, based on Anthropic&apos;s official help pages and the actual screens.</p><p>This is for people on Claude&apos;s paid plans. Pay-as-you-go API billing is out of scope.</p><p>For Fable 5.2 itself, see our earlier article <a href="https://journal.qualiteg.com/claude-fable-5-2-claude-code-guide/">Claude Fable 5.2 Complete Guide</a>.</p><h2 id="a-limit-reset-refills-usage-you-have-already-spent-on-the-spot">A limit reset refills usage you have already spent, on the spot</h2><p>The official help page keeps it short. Resets are handed out to eligible plans from time to time, and using one refills your usage. You find them in claude.ai under Settings, then Usage, below a heading that reads &quot;Limit resets.&quot; It lists how many you hold, what kind they are, and when each one expires.</p><p>The help page names three kinds.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Kind</th><th>What it refills</th><th>Source</th></tr></thead><tbody><tr><td>5-hour reset</td><td>Refills the 5-hour session limit to full</td><td>Official help</td></tr><tr><td>Weekly reset</td><td>Refills the weekly usage limit to full</td><td>Official help</td></tr><tr><td>Fable reset</td><td>Gives back what you used on Fable this week. On accounts that have a 5-hour limit, that limit is refilled to full, including usage on other models</td><td>Official help and the on-screen description</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>On our screen, three Fable resets and one weekly reset are listed, four in total.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/claude-limit-resets-section-ja-v1.png" class="kg-image" alt="What Are Claude&apos;s Limit Resets? What a Fable Reset and a Weekly Reset Give Back, and How They Differ from ChatGPT&apos;s" loading="lazy"><figcaption>The &quot;Limit resets&quot; section on our account (Japanese UI). The Fable reset row is expanded to show its details.</figcaption></figure><p>We will look at these two kinds in turn.</p><h2 id="a-fable-reset-gives-back-only-what-you-used-on-fable">A Fable reset gives back only what you used on Fable</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/claude-limit-reset-manga-1-v2-ja-lab.png" class="kg-image" alt="What Are Claude&apos;s Limit Resets? What a Fable Reset and a Weekly Reset Give Back, and How They Differ from ChatGPT&apos;s" loading="lazy"><figcaption>When to use a reset in a week where the Fable limit is almost gone. Dialogue is below.</figcaption></figure><ul><li><strong>Developer</strong>: Our Fable limit is nearly used up for this week.</li><li><strong>Senior</strong>: Let&apos;s use one of the launch resets.</li><li><strong>Developer</strong>: With a Fable reset, what comes back is the Fable usage, right?</li><li><strong>Senior</strong>: Right. Usage on the other models stays as it is.</li></ul><p>The Usage page shows two bars for the weekly limit. &quot;This week&quot; is the total across all models, and &quot;Fable&quot; is the share within it. The Fable bar is labeled &quot;up to 50% of the weekly usage limit,&quot; and using Fable moves both bars.</p><p>The Fable reset row comes with this description.</p><p>Replenishes your Fable usage and gives back the portion of the weekly limit that Fable consumed. Not all of your weekly usage is reset.</p><p>The info button next to the row has a shorter version.</p><p><strong>&quot;Resets your Fable limit and excludes this week&apos;s Fable usage from the weekly limit calculation.&quot;</strong></p><p>So two things happen. First, the Fable bar goes back to 0%. Second, the &quot;This week&quot; bar drops by exactly the amount you used on Fable.</p><p>Usage on Opus and other non-Fable models stays where it is. The help page says the same, that usage on other models is unchanged. The phrase &quot;Not all of your weekly usage is reset&quot; refers to this.</p><p>Here is a concrete example. At one point our screen showed &quot;This week&quot; at 54% used and &quot;Fable&quot; at 97% used. Using a Fable reset here takes Fable back to 0% and subtracts the Fable portion from that 54%. How far it drops depends on how much Fable you used that week.</p><p>One more point worth keeping in mind. A reset gives back what you spent. It does not raise the limit itself. The help page states this explicitly.</p><p>The 5-hour limit is handled differently from the weekly one.</p><p>On accounts that have a 5-hour limit, a Fable reset refills the entire 5-hour limit, including usage on other models. On another team member&apos;s screen, the 5-hour limit (the current session) was shown, and the description began with &quot;the 5-hour limit and.&quot;</p><p>According to the help page, verified Ultra accounts have no 5-hour limit to begin with. In that case a Fable reset only affects the weekly limit.</p><h2 id="a-weekly-reset-refills-the-whole-weekly-limit">A weekly reset refills the whole weekly limit</h2><p>For the weekly reset, the help page has a single line. It refills the weekly usage limit to full. On screen there is no description or info button, unlike the Fable reset. Only the expiry date is shown.</p><p>Whether using a weekly reset also brings the Fable bar back, or affects the 5-hour limit, is not stated officially.</p><p>Here is how the two kinds compare, bar by bar on the Usage page.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Where to look</th><th>Fable reset</th><th>Weekly reset</th></tr></thead><tbody><tr><td>&quot;Fable&quot; bar</td><td>Back to 0%</td><td>Not stated officially</td></tr><tr><td>&quot;This week&quot; bar</td><td>Drops by the Fable portion only. Usage on other models stays</td><td>Back to full</td></tr><tr><td>5-hour limit</td><td>On accounts that have one, refilled to full including other models</td><td>Not stated officially</td></tr><tr><td>On-screen description</td><td>Description text and an info button</td><td>Expiry date only</td></tr></tbody></table>
<!--kg-card-end: html-->
<h2 id="how-many-you-get-depends-on-your-plan-and-how-long-you-have-been-subscribed">How many you get depends on your plan and how long you have been subscribed</h2><p>Fable resets were handed out as a launch offer for Fable 5.2. According to the offer page in the help center, the count depends on your plan and on how long it has been since your first paid subscription. Even if you cancelled and came back, the clock starts at your first paid billing date. For Team and Enterprise, the organization&apos;s first billing date is used.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Plan</th><th>Number of Fable resets</th></tr></thead><tbody><tr><td>Pro, Team standard seats</td><td>1</td></tr><tr><td>Max, Ultra, Team premium seats, legacy Enterprise premium seats</td><td>1 if your paid subscription is under 6 months old, 2 if 6 to 12 months, 3 if over 12 months</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The condition is that billing began before a set time on the Fable 5.2 release day. The Free plan and some Enterprise plans are excluded. Within our team, members with longer subscriptions had more resets, exactly as the help page describes.</p><p>Expiry dates differ by plan. This is where you can lose out if you are not paying attention.</p><p>Pro and Team standard seats: until 23:59 Pacific Time on November 16, 2026.</p><p>Max, Ultra, Team premium seats, and legacy Enterprise premium seats: until 23:59 Pacific Time on March 8, 2027. Our screen shows the latter date in local time.</p><p>There is a reason the Pro and Team standard window is short. On those plans, for one week after the Fable 5.2 launch, you can use Fable at no extra charge for up to 50% of the weekly usage limit, and the reset exists for that one week. When the period ends, Fable goes back to being billed through usage credits, and unused resets expire.</p><p>Before you use one, check the expiry date shown on your Usage page.</p><p>As for the weekly reset, the help page only says it is handed out to eligible plans from time to time. It does not say when or under what conditions. We have one, and its expiry is much nearer than the Fable resets, about a month out.</p><h2 id="once-used-it-cannot-be-undone-things-to-know-before-you-press">Once used, it cannot be undone. Things to know before you press</h2><p>Everything here is from the official help page. Many of these cannot be reversed after the fact, so it is worth reading first.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Point</th><th>What the help page says</th></tr></thead><tbody><tr><td>Not reversible</td><td>A reset, once used, cannot be undone</td></tr><tr><td>Scheduled resets are unchanged</td><td>The weekly limit still resets on its usual day and time</td></tr><tr><td>They expire</td><td>Unused resets expire on the date shown under Settings, then Usage</td></tr><tr><td>Lost on cancel or downgrade</td><td>If you cancel or downgrade before using them, they become unavailable</td></tr><tr><td>No refunds</td><td>Extra usage already billed is not refunded, and your usage credit balance does not change</td></tr><tr><td>Not available inside Claude Code</td><td>No button appears in Claude Code in the terminal or IDE. Use the browser, the desktop app, or the mobile app</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The last point is easy to miss if you mostly work in Claude Code. That said, the limit is shared across your account, so usage refilled in the browser is available in Claude Code right away.</p><h2 id="to-use-one-press-reset-for-free-and-confirm">To use one, press &quot;Reset for free&quot; and confirm</h2><p>The help page gives these steps. Open Settings, then Usage, find the kind you want under &quot;Limit resets,&quot; press &quot;Reset for free,&quot; and confirm. The same button also appears on the screen you see when you hit the limit.</p><p>You do not have to wait until you hit the limit. You can use it any time before it expires.</p><p>In a week where you have not used Fable yet, the button shows a note that there is currently no Fable usage to reset. The screen tells you when there is nothing to give back.</p><h2 id="chatgpt-has-similar-resets-the-difference-is-what-happens-to-the-scheduled-reset">ChatGPT has similar resets. The difference is what happens to the scheduled reset</h2><p>OpenAI also offers &quot;resets&quot; that refill the Codex and Work usage limits on ChatGPT&apos;s paid plans. For those who use both, here is how they compare.</p><p>According to OpenAI&apos;s help pages, there are three kinds. The first is a free reset that is granted to your account, kept until it expires, and used by you whenever you choose. This is the closest to Claude&apos;s limit resets. The second is a reset OpenAI applies directly to eligible accounts, with no action on your part and nothing to store. The third is a reset that eligible Plus and Pro individual accounts can buy, applied on the spot.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Aspect</th><th>Claude&apos;s limit resets</th><th>ChatGPT&apos;s resets (Codex, Work)</th></tr></thead><tbody><tr><td>Limits covered</td><td>Claude usage limits (5-hour, weekly, Fable)</td><td>Codex and Work usage limits (5-hour and weekly. Monthly on the Go plan)</td></tr><tr><td>Free reset you use yourself</td><td>Yes (limit resets)</td><td>Yes (stored resets)</td></tr><tr><td>When you can use the free reset</td><td>Even before you hit the limit</td><td>When used from usage settings, one of the covered limits must be below 80% remaining</td></tr><tr><td>What it refills</td><td>Depends on the kind. A Fable reset refills Fable usage</td><td>Stored resets depend on the plan. Go: monthly. Plus and Business Standard: 5-hour and weekly. Pro: weekly. Purchased resets: 5-hour and weekly</td></tr><tr><td>Scheduled reset time</td><td>Unchanged</td><td>When a period restarts, the next renewal time moves too</td></tr><tr><td>Reset applied by the operator</td><td>Not mentioned in the help pages we checked</td><td>Yes (automatic resets)</td></tr><tr><td>Paid reset</td><td>Not mentioned in the help pages we checked</td><td>Yes (eligible Plus and Pro individual accounts)</td></tr><tr><td>Where you can use it</td><td>Browser, desktop app, mobile app. Not from inside Claude Code</td><td>Stored resets: desktop app, web, Codex CLI. Not from the mobile app</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The biggest difference is the timing of the scheduled reset. With Claude, using a limit reset leaves the weekly limit resetting on its usual day and time. On the ChatGPT side, when a reset restarts the usage period, the next renewal time moves as well. With a purchased reset, for example, a new weekly period begins at your first Work or Codex request after it is applied, and the next scheduled reset falls seven days later.</p><p>This is not about which is better. The limits are divided differently and the conditions for use differ. OpenAI&apos;s help page also states that free resets are a perk with no guarantee of availability or timing. Claude&apos;s are described as handed out from time to time, so in both cases the sensible approach is to use them well when you have them.</p><h2 id="when-to-use-one-in-our-view">When to use one, in our view</h2><p>From here on this is our view, based on the above, rather than the official description.</p><p>A Fable reset pays off when only the Fable limit runs out mid-week and the scheduled reset is still days away. Because the Fable portion is also subtracted from the &quot;This week&quot; bar, the more Fable-centric your usage, the more you get back.</p><p>Using one right before the scheduled reset is a waste. If you wait, the limit comes back at its usual time anyway.</p><p>A weekly reset suits a week where the weekly limit is running low across models, not just Fable. Since it expires sooner than the Fable resets, we plan to use our weekly reset first.</p><p>As of this writing we are still holding on to ours. Neither kind can be undone, so before you press, compare the &quot;This week&quot; and &quot;Fable&quot; bars on the Usage page, the scheduled reset time, and the expiry date of each reset.</p><h2 id="summary-which-reset-for-which-situation">Summary: which reset for which situation</h2>
<!--kg-card-begin: html-->
<table><thead><tr><th>Situation</th><th>What to do</th></tr></thead><tbody><tr><td>Only the Fable limit is gone and the scheduled reset is days away</td><td>Use a Fable reset</td></tr><tr><td>The weekly limit is running out across models, not just Fable</td><td>Use a weekly reset</td></tr><tr><td>The scheduled reset is coming soon</td><td>Wait, do not use one</td></tr><tr><td>You hold a reset that expires soon</td><td>Use it in a week where usage has dropped, before it expires</td></tr><tr><td>You ran out while working in Claude Code</td><td>Open the Usage page in the browser or the app and use it there</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>A limit reset is a one-time right that cannot be taken back. Know what it refills and when it expires, and use it where it counts the most.</p><p>See you next time!</p><h2 id="sources-and-references">Sources and references</h2><ul><li><a href="https://support.claude.com/en/articles/17007452-what-is-a-limit-reset?ref=journal.qualiteg.com">What is a limit reset? (Claude Help Center)</a></li><li><a href="https://support.claude.com/en/articles/17506494-claude-fable-5-2-launch-offers?ref=journal.qualiteg.com">Claude Fable 5.2 launch offers (Claude Help Center)</a></li><li><a href="https://help.openai.com/en/articles/20001498-how-resets-work-in-codex?ref=journal.qualiteg.com">How resets work in Codex (OpenAI Help Center)</a></li><li><a href="https://help.openai.com/en/articles/20001507-paid-weekly-work-and-codex-rate-limit-resets?ref=journal.qualiteg.com">Paid weekly Work and Codex rate limit resets (OpenAI Help Center)</a></li></ul><h2 id="related-articles">Related articles</h2><ul><li><a href="https://journal.qualiteg.com/claude-fable-5-2-claude-code-guide/">Claude Fable 5.2 Complete Guide: Model Specs, API, and Claude Code Operations (Qualiteg Journal)</a></li><li><a href="https://journal.qualiteg.com/claude-fable5-cost-and-outlook/">What&apos;s Next for Claude Fable 5? Background, Cost, and Outlook, Fact-Based (Qualiteg Journal)</a></li><li><a href="https://journal.qualiteg.com/getting-started-claude-code-cli-and-web/">Getting Started with Claude Code (Qualiteg Journal)</a></li></ul>]]></content:encoded></item><item><title><![CDATA[Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations]]></title><description><![CDATA[Claude Fable 5.2 is 20% cheaper, up to 30% faster, and holds the top three spots on an independent index. We cover API migration and Claude Code's default effort and Fast mode.]]></description><link>https://journal.qualiteg.com/claude-fable-5-2-claude-code-guide/</link><guid isPermaLink="false">6af2532e0a0f0d13c338b749</guid><category><![CDATA[AI Agents]]></category><category><![CDATA[ClaudeCode]]></category><category><![CDATA[LLM]]></category><category><![CDATA[Generative AI Frontlines]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Mon, 09 Nov 2026 23:04:01 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/11/claude-fable-5-2-claude-code-guide-cover-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/11/claude-fable-5-2-claude-code-guide-cover-en.png" alt="Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations"><p>Hello!</p><p>On November 9, 2026 (US time), Anthropic announced Claude Fable 5.2 and Claude Mythos 5.2. The update comes about two months after Fable 5.1 in September.</p><p>First, to sum up this model in one sentence.</p><p>&quot;Fable 5.2 is 20% cheaper and up to 30% faster than Fable 5.1, and it took the top three spots on an independent composite index. On the API, building conversations without rewriting history is now the baseline for every account.&quot;</p><p>That is the short version.</p><p>Now, let&apos;s start with API pricing.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig1_price_table_en.png" class="kg-image" alt="Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations" loading="lazy" width="2000" height="909" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig1_price_table_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig1_price_table_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig1_price_table_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig1_price_table_en.png 2200w" sizes="(min-width: 720px) 720px"><figcaption>Figure 1. Claude Fable 5.2 API pricing. Input is $8 and output $40, 20% lower than the previous model. Source: Anthropic official pricing page and official announcement (November 9, 2026). Chart by Qualiteg</figcaption></figure><p>Fable 5.2 costs $8 for input and $40 for output, 20% less than Fable 5.1&apos;s $10 and $50. Cache reads also went from $0.25 to $0.20, the same 20% cut.</p><p>All prices in this article are in US dollars.</p><p>The top model raised its performance and lowered its per-token price at the same time. Fable 5 and Fable 5.1 were both $10 and $50, so this is the first time Fable&apos;s input and output prices have come down.</p><p>In July we published &quot;<a href="https://journal.qualiteg.com/claude-fable-5-claude-code-guide/">The Complete Guide to Claude Fable 5: Model Specs and Claude Code Operations from the Official Docs</a>,&quot; and in September &quot;<a href="https://journal.qualiteg.com/claude-opus-5-5-claude-code-guide/">Claude Opus 5.5 Complete Guide: Model Specifications, API Notes, and Claude Code Operations</a>.&quot; This article is their sequel.</p><p>In the Opus 5.5 article, we introduced Opus 5.5 as &quot;a model that aims for Fable 5.1-class performance at a lower per-token price.&quot; Fable 5.2 beats that Opus 5.5 on almost every evaluation.</p><p>Our sources are Anthropic&apos;s official announcement, official docs, and system card, plus data from the independent evaluator Artificial Analysis.</p><p>It is a long article, so there is no need to read it from start to finish. Feel free to jump to the chapters that interest you.</p><h3 id="table-of-contents">Table of Contents</h3><p>Part 1: What Claude Fable 5.2 Is</p><ul><li><a href="#ch1">1. Fable 5.2 and Mythos 5.2 are the same model</a></li><li><a href="#ch2">2. Basic specifications at a glance</a></li><li><a href="#ch3">3. Input, output, and cache read prices are all 20% lower</a></li><li><a href="#ch4">4. Best on 14 of 16 of Anthropic&apos;s published benchmarks</a></li><li><a href="#ch5">5. Independent evaluation puts it in the top three spots on the Intelligence Index</a></li><li><a href="#ch6">6. By effort level, high gives the best balance of performance and cost</a></li><li><a href="#ch7">7. Safety and safeguards according to the system card</a></li></ul><p>Part 2: Things to Watch When Using the API</p><ul><li><a href="#ch8">8. The history rewrite check now applies to every account</a></li><li><a href="#ch9">9. Only Fable 5.2 and Mythos 5.2 can read Fable 5.2&apos;s thinking blocks</a></li><li><a href="#ch10">10. Computer use moves to the toolset, and advisor pairings must match</a></li><li><a href="#ch11">11. Start effort at high, and use xhigh for work where you have measured a gain</a></li><li><a href="#ch12">12. Behaviors to review when carrying prompts over</a></li><li><a href="#ch13">13. Migration steps and cost estimates</a></li></ul><p>Part 3: Using Fable 5.2 in Claude Code</p><ul><li><a href="#ch14">14. In v2.1.321, the fable alias now points to Fable 5.2</a></li><li><a href="#ch15">15. The default effort in Claude Code is medium</a></li><li><a href="#ch16">16. Fast mode costs $16 / $80, and you turn it on by running /fast on Fable 5.2</a></li><li><a href="#ch17">17. What happens when a classifier switches models</a></li><li><a href="#ch18">18. Usage by plan, launch offers, and choosing models day to day</a></li></ul><h3 id="what-changed-from-fable-51-in-brief">What Changed from Fable 5.1, in Brief</h3><p>Each chapter covers the details, but here is the big picture first.</p><p><strong>Per-token prices fell 20%</strong><br>Input is $8, output $40, and cache reads $0.20 (<a href="#ch3">Chapter 3</a>)</p><p><strong>Output generation is up to 30% faster</strong><br>In the official model list, relative latency also changed from Slower to Moderate (<a href="#ch2">Chapter 2</a>)</p><p><strong>Best on 14 of 16 published benchmarks</strong><br>SWE-bench Pro is 93.0%, and the rate of completing real freelance projects is 41.3%, nearly double Opus 5.5 (<a href="#ch4">Chapter 4</a>)</p><p><strong>The top three spots on an independent composite index</strong><br>On Artificial Analysis&apos;s Intelligence Index, max, xhigh, and high rank first through third (<a href="#ch5">Chapter 5</a>)</p><p><strong>Without going all out, it matches Opus 5.5 at full effort; at full effort, it pulls away</strong><br>At high, the third of five levels, it reaches the score Opus 5.5 posted at its top setting, max, for about half the cost. On hard, long work the gap widens. The rate of completing real freelance projects is 41.3% versus 21.3%, and the independent composite index is 65 versus 59 at each model&apos;s top setting (<a href="#ch6">Chapter 6</a>)</p><p><strong>About 55% fewer safeguard blocks</strong><br>This is from Anthropic&apos;s early testing, compared with Fable 5.1 (<a href="#ch7">Chapter 7</a>)</p><p><strong>The history rewrite check now applies to every account</strong><br>On Fable 5.1, it applied only to newer accounts (<a href="#ch8">Chapter 8</a>)</p><p><strong>Fast mode is now available</strong><br>Up to 2.5 times the speed, at $16 / $80 (<a href="#ch16">Chapter 16</a>)</p><p><strong>Claude Code supports it from v2.1.321</strong><br>The fable alias points to Fable 5.2, and the default effort is medium (<a href="#ch14">Chapter 14</a> and <a href="#ch15">Chapter 15</a>)</p><p><strong>Fable is included in plan usage on Pro and Team Standard seats for one week</strong><br>This is a launch offer through November 16 (<a href="#ch18">Chapter 18</a>)</p><hr><h2 id="part-1-what-claude-fable-52-is">Part 1: What Claude Fable 5.2 Is</h2>
<!--kg-card-begin: html-->
<div id="ch1"></div>
<!--kg-card-end: html-->
<h3 id="1-fable-52-and-mythos-52-are-the-same-model">1. Fable 5.2 and Mythos 5.2 are the same model</h3><p>Here is how the official announcement describes the relationship between the two models.</p><p>&quot;Claude Fable and Claude Mythos are the same underlying model. Fable has strong cybersecurity and biology safeguards to guard against misuse, which allows us to make it available to all users.&quot;</p><p>Mythos 5.2 is the same model with the safeguards in those areas relaxed. It is available only to organizations that have passed review under the Cyber Verification Program or the Life Sciences Verification Program.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Model</th><th>Positioning</th><th>API price (input / output)</th><th>Who can use it</th></tr></thead><tbody><tr><td><strong>Claude Fable 5.2</strong></td><td><strong>For demanding reasoning and long-horizon agentic work</strong></td><td><strong>$8 / $40</strong></td><td>All Claude API customers</td></tr><tr><td>Claude Mythos 5.2</td><td>Same capabilities as Fable 5.2</td><td>$8 / $40</td><td>Vetted organizations</td></tr><tr><td>Claude Opus 5.5</td><td>For long-running agentic coding and knowledge work</td><td>$4 / $20</td><td>All customers</td></tr><tr><td>Claude Sonnet 5.5</td><td>Balance of speed and intelligence</td><td>$2 / $10</td><td>All customers</td></tr><tr><td>Claude Haiku 5.5</td><td>High-volume processing and low latency</td><td>From $0.10 / $0.50</td><td>All customers</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Anthropic&apos;s recommended way to choose between them has the same shape as it did for Opus 5.5.</p><p>Start most coding and knowledge work on Opus 5.5, the everyday workhorse. Move up to Fable 5.2 when Opus 5.5 at a high effort level still falls short on your evaluations. Move light work down to Sonnet 5.5 or Haiku 5.5.</p><p>As work that suits Fable 5.2, Anthropic names &quot;the most ambitious, open-ended technical work.&quot; The examples on the model page are agent sessions that run for hours, deep multi-step research, and work that goes all the way from analysis to finished documents, spreadsheets, and slides.</p><p>Anthropic also points to one more way to use it.</p><p>Put Fable 5.2 in the role of Advisor or Orchestrator. Haiku, Sonnet, or Opus consult Fable 5.2 only on the hard problems, or Fable 5.2 hands simple tasks off to smaller models. It is a design that uses Fable 5.2 only where it counts instead of running everything on it.</p><p>The case studies in the announcement are mostly large in scale.</p><p>More than 1,000 subagents ran for eight days and wrote a prototype operating system that boots on a laptop. A production service was moved from Python to Rust, and its startup became about 26 times faster. A team of 1,000 agents formalized the proof of Fermat&apos;s Last Theorem in 30 hours.</p>
<!--kg-card-begin: html-->
<div id="ch2"></div>
<!--kg-card-end: html-->
<h3 id="2-basic-specifications-at-a-glance">2. Basic specifications at a glance</h3>
<!--kg-card-begin: html-->
<table><thead><tr><th>Item</th><th>Claude Fable 5.2</th><th>Claude Fable 5.1 (previous)</th></tr></thead><tbody><tr><td>Model ID</td><td><code>claude-fable-5-2</code></td><td><code>claude-fable-5-1</code></td></tr><tr><td>Amazon Bedrock ID</td><td><code>anthropic.claude-fable-5-2</code></td><td><code>anthropic.claude-fable-5-1</code></td></tr><tr><td>Context window</td><td>1M</td><td>1M</td></tr><tr><td>Max output</td><td>128K</td><td>128K</td></tr><tr><td>Knowledge cutoff</td><td>June 2026</td><td>June 2026</td></tr><tr><td>Thinking</td><td>Always on (adaptive)</td><td>Always on (adaptive)</td></tr><tr><td>Default effort (API)</td><td>high</td><td>high</td></tr><tr><td>Relative latency</td><td><strong>Moderate</strong></td><td>Slower</td></tr><tr><td>Input and output</td><td>Text and images in, text out</td><td>Same</td></tr><tr><td>Tokenizer</td><td>The one introduced with Opus 4.7</td><td>Same</td></tr><tr><td>Retirement</td><td>Not before November 9, 2027</td><td>Not before September 1, 2027</td></tr><tr><td>Data retention</td><td>30-day retention required</td><td>30-day retention required</td></tr><tr><td>Priority Tier</td><td>Not supported</td><td>Not supported</td></tr><tr><td>Rate limits</td><td>A separate pool for Fable 5.2 alone</td><td>Shared pool with Fable 5</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>It is available on the Claude API, Amazon Bedrock, Claude Platform on AWS, Google Cloud (both Google-managed and Anthropic-managed), and Microsoft Foundry.</p><p>The spec numbers are almost all unchanged from Fable 5.1. Context length, max output, and tokenizer are the same, so your token estimates carry over as is.</p><p>What changed is speed.</p><p>The official announcement says output generation is up to 30% faster than Fable 5.1. In the model list, relative latency went from Fable 5.1&apos;s Slower to Moderate, the same label as Opus 5.5.</p><p>The rate limit pool also changed.</p><p>Fable 5.1 and Fable 5 share one pool, but Fable 5.2 has a separate one. When you move from Fable 5.1, its usage is counted separately from your existing pool.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Tier</th><th>Fable 5.2 (RPM / input TPM / output TPM)</th><th>Opus 5.5 (same)</th></tr></thead><tbody><tr><td>Start</td><td>4,000 / 500,000 / 100,000</td><td>4,000 / 2,000,000 / 400,000</td></tr><tr><td>Build</td><td>10,000 / 1,500,000 / 300,000</td><td>10,000 / 5,000,000 / 1,000,000</td></tr><tr><td>Scale</td><td>13,000 / 4,000,000 / 800,000</td><td>13,000 / 10,000,000 / 2,000,000</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The token limits are smaller than Opus 5.5&apos;s. If you plan to move production traffic from Opus 5.5 to Fable 5.2, check this table for headroom first.</p><p>Data retention works the same way as on Fable 5.1.</p><p>Fable 5.2 requires 30-day data retention, and on the Claude API, requests from organizations or workspaces with no retention return a 400 error.</p><p>What is new this time is a mechanism called Enterprise Frontier Safeguards (EFS). It runs safety monitoring on the customer&apos;s own infrastructure and is compatible with zero data retention (ZDR). According to the official announcement, it is in private beta, and eligible Enterprise customers can keep using Fable 5.2 under ZDR until they adopt EFS.</p>
<!--kg-card-begin: html-->
<div id="ch3"></div>
<!--kg-card-end: html-->
<h3 id="3-input-output-and-cache-read-prices-are-all-20-lower">3. Input, output, and cache read prices are all 20% lower</h3><p>Here are the values from the official pricing page, in US dollars per 1 million tokens. The write columns are cache write prices.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Model</th><th>Input</th><th>5-minute write</th><th>1-hour write</th><th>Cache read</th><th>Output</th></tr></thead><tbody><tr><td><strong>Claude Fable 5.2</strong></td><td><strong>$8</strong></td><td>$10</td><td>$16</td><td><strong>$0.20</strong></td><td><strong>$40</strong></td></tr><tr><td>Claude Fable 5.1</td><td>$10</td><td>$12.50</td><td>$20</td><td>$0.25</td><td>$50</td></tr><tr><td>Claude Fable 5</td><td>$10</td><td>$12.50</td><td>$20</td><td>$1.00</td><td>$50</td></tr><tr><td>Claude Opus 5.5</td><td>$4</td><td>$5</td><td>$8</td><td>$0.20</td><td>$20</td></tr><tr><td>Claude Sonnet 5.5</td><td>$2</td><td>$2.50</td><td>$4</td><td>$0.10</td><td>$10</td></tr><tr><td>Claude Haiku 5.5 (prompts up to 100K tokens)</td><td>$0.10</td><td>$0.125</td><td>$0.20</td><td>$0.01</td><td>$0.50</td></tr></tbody></table>
<!--kg-card-end: html-->
<figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig2_fable51_to_52_en.png" class="kg-image" alt="Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations" loading="lazy" width="2000" height="945" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig2_fable51_to_52_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig2_fable51_to_52_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig2_fable51_to_52_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig2_fable51_to_52_en.png 2200w" sizes="(min-width: 720px) 720px"><figcaption>Figure 2. From Fable 5.1 to Fable 5.2, input, output, and cache reads are all 20% cheaper. Source: Anthropic official pricing page and What&apos;s new in Claude Fable 5.2. Chart by Qualiteg</figcaption></figure><p>With Opus 5.5, only cache reads fell sharply, by 60%. This time, every item falls by the same 20%.</p><p>The cache read price stays at 0.025x the input price. Opus 5.5 and Sonnet 5.5 are at 0.05x and other models at 0.1x, so the Fable line has the lowest multiplier.</p><p><strong>Cache reads cost the same $0.20 on Fable 5.2 and Opus 5.5.</strong></p><p>Agentic work rereads the same instructions, tool definitions, and past exchanges every turn. For that rereading alone, moving up to Fable 5.2 does not change the price. The difference comes from new input and output.</p><p>If you move directly from Fable 5, cache reads drop from $1.00 to $0.20, an 80% cut.</p><p>Here are the other prices.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Item</th><th>Claude Fable 5.2</th><th>Claude Fable 5.1</th></tr></thead><tbody><tr><td>Batch API (input / output)</td><td>$4 / $20</td><td>$5 / $25</td></tr><tr><td>Fast mode (input / output)</td><td>$16 / $80</td><td>Not available</td></tr><tr><td>1M context surcharge</td><td>None (standard pricing across the full window)</td><td>None</td></tr><tr><td>Data residency (inference_geo &quot;us&quot;)</td><td>1.1x</td><td>1.1x</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>On the Batch API, Fable 5.2 costs $4 / $20, the same as standard Opus 5.5. For high-volume work that is not urgent, you can run Fable 5.2 at Opus 5.5&apos;s price.</p><p>Fast mode is a new option this time. <a href="#ch16">Chapter 16</a> covers how to use it in Claude Code.</p><p>Here is the change from Fable 5.1, item by item, in one table.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Item</th><th>Fable 5.1</th><th>Fable 5.2</th><th>Change</th></tr></thead><tbody><tr><td>Input</td><td>$10</td><td>$8</td><td>20% lower</td></tr><tr><td>Output</td><td>$50</td><td>$40</td><td>20% lower</td></tr><tr><td>Cache read</td><td>$0.25</td><td>$0.20</td><td>20% lower</td></tr><tr><td>5-minute cache write</td><td>$12.50</td><td>$10</td><td>20% lower</td></tr><tr><td>1-hour cache write</td><td>$20</td><td>$16</td><td>20% lower</td></tr><tr><td>Batch API (input / output)</td><td>$5 / $25</td><td>$4 / $20</td><td>20% lower</td></tr><tr><td>Fast mode (input / output)</td><td>Not available</td><td>$16 / $80</td><td>New</td></tr><tr><td>Output generation speed</td><td>Baseline</td><td>Up to 30% faster</td><td>Faster</td></tr></tbody></table>
<!--kg-card-end: html-->
<p><strong>Comparing prices with other vendors&apos; high-end models</strong></p><p>Here are the official list prices for OpenAI&apos;s GPT-6 Astra and Google&apos;s Gemini 4 Argon alongside Fable 5.2. These are each vendor&apos;s Standard prices as of November 10.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Model</th><th>Input</th><th>Cache read</th><th>Output</th><th>When the price changes</th></tr></thead><tbody><tr><td><strong>Claude Fable 5.2</strong></td><td><strong>$8.00</strong></td><td><strong>$0.20</strong></td><td><strong>$40.00</strong></td><td>Same price across the full 1M context</td></tr><tr><td>GPT-6 Astra</td><td>$10.00</td><td>$1.00</td><td>$50.00</td><td>$20.00 / $2.00 / $75.00 when input exceeds 272K</td></tr><tr><td>Gemini 4 Argon</td><td>$2.00</td><td>$0.10</td><td>$10.00</td><td>$4.00 / $0.20 / $20.00 from December 1</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>On per-token price alone, Gemini 4 Argon is the lowest, and Fable 5.2 sits 20% below GPT-6 Astra.</p><p>However, the order in this table is not the same as the order of which model is cheapest to use. There are four reasons.</p><p>The first is that the same text comes out to a different number of tokens on each model. Each vendor has its own tokenizer, so 1 million tokens does not cover the same amount of text.</p><p>The second is that the number of tokens used for one task differs. How much the model thinks, how often it calls tools, and how long its answers are all vary by model and effort.</p><p>The third is that billing thresholds differ. GPT-6 Astra&apos;s price goes up when input exceeds 272K. Gemini 4 Argon&apos;s price is scheduled to change on December 1. Fable 5.2 has the same price across the full 1M. For use cases that feed in long context, these thresholds matter.</p><p>The fourth is that caching works differently. Fable 5.2 and GPT-6 Astra charge for cache writes, while Gemini 4 Argon, according to its pricing page, supports only implicit caching. In agentic work that rereads the same context, the cost depends on the cache read price ($0.20, $1.00, and $0.10) and also on how often requests hit the cache.</p><p>Instead of per-token prices, you can look at the cost of running the same evaluation task. Here are three rows from the Artificial Analysis data covered in <a href="#ch5">Chapter 5</a>.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Model (effort)</th><th>Output price</th><th>Cost to run one evaluation task</th><th>Intelligence Index</th></tr></thead><tbody><tr><td>Claude Fable 5.2 (high)</td><td>$40.00</td><td>$9.08</td><td>62</td></tr><tr><td>GPT-6 Astra (max)</td><td>$50.00</td><td>$6.24</td><td>57</td></tr><tr><td>Gemini 4 Argon (high)</td><td>$10.00</td><td>$9.95</td><td>50</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Output prices span a fivefold range, from $10 to $50, but the cost of running the same evaluation task falls between $6.24 and $9.95. The order of per-token prices does not match the order of run costs either.</p><p>When you compare prices, we recommend looking at both the per-token price table and the cost of actually running your workload.</p>
<!--kg-card-begin: html-->
<div id="ch4"></div>
<!--kg-card-end: html-->
<h3 id="4-best-on-14-of-16-of-anthropics-published-benchmarks">4. Best on 14 of 16 of Anthropic&apos;s published benchmarks</h3><p>All figures from here on were published by Anthropic.</p><p>Unless otherwise noted, each model ran at its highest effort, averaged over five runs. The Claude models were measured with production safeguards enabled.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Benchmark</th><th>Fable 5.2</th><th>Opus 5.5</th><th>Fable 5.1</th><th>GPT-6 Astra</th><th>Gemini 4 Carbon</th></tr></thead><tbody><tr><td>FrontierCode 2.0</td><td><strong>49.1%</strong></td><td>45.5%</td><td>32.4%</td><td>42.8%</td><td>41.1%</td></tr><tr><td>FrontierSWE v2</td><td><strong>69.9%</strong></td><td>62.3%</td><td>56.3%</td><td>65.5%</td><td>61.2%</td></tr><tr><td>Terminal-Bench-Science 0.1</td><td><strong>74.6%</strong></td><td>58.7%</td><td>52.6%</td><td>68.1%</td><td>64.8%</td></tr><tr><td>AutomationBench</td><td>51.4%</td><td>42.5%</td><td>31.4%</td><td>41.4%</td><td><strong>53.9%</strong></td></tr><tr><td>WANDR (relaxed)</td><td><strong>82.1%</strong></td><td>72.3%</td><td>68.7%</td><td>&#x2013;</td><td>&#x2013;</td></tr><tr><td>OSWorld 2.1 (offline, partial)</td><td><strong>89.3%</strong></td><td>87.2%</td><td>83.5%</td><td>79.5%</td><td>81.5%</td></tr><tr><td>RoboDojo Sim</td><td><strong>64.9%</strong></td><td>39.4%</td><td>27.6%</td><td>29.8%</td><td>&#x2013;</td></tr><tr><td>AA-Briefcase v1.1 (rubric)</td><td><strong>66.1%</strong></td><td>61.2%</td><td>61.5%</td><td>52.0%</td><td>61.8%</td></tr><tr><td>AA-Briefcase v1.1 (Elo)</td><td>1785</td><td><strong>1800</strong></td><td>1669</td><td>1569</td><td>1611</td></tr><tr><td>Remote Labor Index</td><td><strong>41.3%</strong></td><td>21.3%</td><td>17.9%</td><td>20.8%</td><td>&#x2013;</td></tr><tr><td>Humanity&apos;s Last Exam (with tools)</td><td><strong>74.2%</strong></td><td>67.7%</td><td>65.6%</td><td>57.2%</td><td>&#x2013;</td></tr><tr><td>Chartography (no tools)</td><td><strong>73.4%</strong></td><td>64.4%</td><td>44.8%</td><td>71.0%</td><td>72.5%</td></tr></tbody></table>
<!--kg-card-end: html-->
<figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig3_anthropic_benchmarks_en.png" class="kg-image" alt="Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations" loading="lazy" width="2000" height="1455" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig3_anthropic_benchmarks_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig3_anthropic_benchmarks_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig3_anthropic_benchmarks_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig3_anthropic_benchmarks_en.png 2200w" sizes="(min-width: 720px) 720px"><figcaption>Figure 3. Anthropic&apos;s published benchmarks. Fable 5.2 leads on five of the six. Source: Anthropic official announcement (November 9, 2026). Chart by Qualiteg</figcaption></figure><p>For FrontierCode 2.0, Fable 5.2 and Fable 5.1 are at xhigh, and Gemini 4 Carbon is at medium. For Remote Labor Index, Fable 5.2 is at high.</p><p>&quot;14 of 16&quot; comes from the summary table in the system card (Table 8.1.A). That table lists 16 results for Fable 5.2 with safeguards enabled, and compared with Fable 5.1, Opus 5.5, and GPT-6 Astra, Fable 5.2 was best on 14 of them. Opus 5.5 was ahead on two, GDPval-AA (1831 vs. 1866) and AA-Briefcase Elo (1785 vs. 1800).</p><p>The same table also includes two results measured on Mythos 5.2 (protein binder design and CVE Rediscovery), and Mythos 5.2 was best on those as well. The tables above and below are the main items taken from that summary table and the official announcement&apos;s table.</p><p>Here are three numbers that stand out.</p><p>The first is 41.3% on Remote Labor Index.</p><p>This evaluation has the model take on 240 real freelance projects, and humans judge whether each deliverable is at least as good as a human professional&apos;s. Fable 5.2 passed 99 of 240, nearly double Opus 5.5&apos;s 21.3% and GPT-6 Astra&apos;s 20.8%. And this is at high, not max.</p><p>The second is 64.9% on RoboDojo Sim.</p><p>It is a simulation in which the model controls a two-armed robot from camera images, and the score is more than double GPT-6 Astra&apos;s 29.8% and Fable 5.1&apos;s 27.6%. This is from a model with no dedicated training for robot control.</p><p>The third is Chartography without tools, which rose from Fable 5.1&apos;s 44.8% to 73.4%. The ability to read figures and charts as they are, without cropping them with code, has grown a lot.</p><p>The system card also includes values not in the announcement.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Benchmark</th><th>Fable 5.2</th><th>Opus 5.5</th><th>Fable 5.1</th></tr></thead><tbody><tr><td>SWE-bench Pro</td><td><strong>93.0%</strong></td><td>89.9%</td><td>81.2%</td></tr><tr><td>SWE-bench Multilingual</td><td><strong>96.6%</strong></td><td>93.9%</td><td>89.1%</td></tr><tr><td>SWE-bench Multimodal</td><td><strong>74.1%</strong></td><td>61.4%</td><td>54.7%</td></tr><tr><td>Humanity&apos;s Last Exam (no tools)</td><td><strong>73.2%</strong></td><td>64.4%</td><td>60.9%</td></tr><tr><td>Terminal-Bench 4.0</td><td><strong>67.6%</strong></td><td>66.4%</td><td>55.8%</td></tr><tr><td>CursorBench 4.0</td><td><strong>60.0%</strong></td><td>57.8%</td><td>51.8%</td></tr><tr><td>BenchCAD 2.1 (with tools)</td><td><strong>83.5</strong></td><td>65.3</td><td>44.4</td></tr><tr><td>GDPval-AA v2.1 (Elo)</td><td>1831</td><td><strong>1866</strong></td><td>1763</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>On Terminal-Bench 4.0, the gap between 67.6% and 66.4% is within the standard error (&#xB1;1.8 and &#xB1;2.6). On this evaluation, it is fair to read Fable 5.2 and Opus 5.5 as being at the same level.</p><p>BenchCAD 2.1, which measures hardware design, is graded out of 100 with partial credit. The announcement also describes a case in which the model designed three electronic devices (a voice assistant hub, a USB speakerphone, and a handheld virtual pet) from a one-paragraph brief. It wrote the four-layer board routing, the enclosure CAD, and the software, and the three manufactured units reportedly worked without any electrical fixes.</p><p>The customer comments in the announcement also center on handing off long jobs.</p><p>Uber writes, &quot;I had Claude Fable 5.2 run a 10&#x2013;12 hour security audit. It sent out about 170 reviewer agents and came back with a write-up, fix pull requests and a policy proposal.&quot; Cognition writes, &quot;Claude Fable 5.2 says it doesn&apos;t know things instead of hallucinating. At one point it told me its last estimate was wrong and that it wouldn&apos;t give a number until it had measured the change.&quot;</p><p>We look at independent evaluation in the next chapter.</p>
<!--kg-card-begin: html-->
<div id="ch5"></div>
<!--kg-card-end: html-->
<h3 id="5-independent-evaluation-puts-it-in-the-top-three-spots-on-the-intelligence-index">5. Independent evaluation puts it in the top three spots on the Intelligence Index</h3><p>The independent evaluator Artificial Analysis measured all five Fable 5.2 effort levels and published them on its model page. The following is its public data as of November 10.</p><p>The Intelligence Index is a composite of evaluations that Artificial Analysis runs itself. It moved to v5 on October 27 and is now made up of nine evaluations.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig4_aa_ranking_en.png" class="kg-image" alt="Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations" loading="lazy" width="2000" height="1410" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig4_aa_ranking_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig4_aa_ranking_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig4_aa_ranking_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig4_aa_ranking_en.png 2100w" sizes="(min-width: 720px) 720px"><figcaption>Figure 4. Fable 5.2 at max, xhigh, and high holds the top three spots on the Intelligence Index. Independent evaluation. Source: Artificial Analysis (retrieved November 10, 2026). Chart by Qualiteg</figcaption></figure>
<!--kg-card-begin: html-->
<table><thead><tr><th>Model (effort)</th><th>Intelligence Index</th><th>Cost per task</th><th>Time per task</th></tr></thead><tbody><tr><td><strong>Fable 5.2 (max)</strong></td><td><strong>65</strong></td><td>$26.54</td><td>45.0 min</td></tr><tr><td><strong>Fable 5.2 (xhigh)</strong></td><td><strong>62</strong></td><td>$14.83</td><td>26.3 min</td></tr><tr><td><strong>Fable 5.2 (high, API default)</strong></td><td><strong>62</strong></td><td>$9.08</td><td>17.7 min</td></tr><tr><td>Gemini 4 Carbon (high)</td><td>60</td><td>$7.76</td><td>27.2 min</td></tr><tr><td>Opus 5.5 (max)</td><td>59</td><td>$10.04</td><td>25.1 min</td></tr><tr><td>GPT-6 Astra (max)</td><td>57</td><td>$6.24</td><td>22.6 min</td></tr><tr><td><strong>Fable 5.2 (medium, Claude Code default)</strong></td><td><strong>56</strong></td><td>$3.80</td><td>8.3 min</td></tr><tr><td>GPT-6.1 Sol (max)</td><td>55</td><td>$1.27</td><td>20.2 min</td></tr><tr><td>Sonnet 5.5 (max)</td><td>55</td><td>$10.02</td><td>31.5 min</td></tr><tr><td><strong>Fable 5.2 (low)</strong></td><td><strong>54</strong></td><td>$3.10</td><td>6.5 min</td></tr><tr><td>Fable 5.1 (max)</td><td>49</td><td>$12.99</td><td>22.7 min</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Cost and time are weighted averages per Intelligence Index evaluation task. The Claude models were measured with Anthropic&apos;s default fallback enabled.</p><p>In our September Opus 5.5 article, we gave Opus 5.5 (max) an index of 58. That was the value on the previous version of the index. v5 swapped in different evaluations, and the same Opus 5.5 (max) now scores 59. All figures in this article are v5 values.</p><p>Here are three things the table shows.</p><p>The first is the gain over the previous model.</p><p>Against Fable 5.1 (max) at 49, Fable 5.2 (max) scores 65. Even the lowest level, low, scores 54, above Fable 5.1 at max. The cost is $3.10 versus $12.99, less than a quarter.</p><p>The second is where high, the API default, lands.</p><p>Fable 5.2 at high scores 62 on the index, higher than Opus 5.5 at max (59), at $9.08 versus $10.04 in cost and 17.7 versus 25.1 minutes in time. Under Artificial Analysis&apos;s test conditions, the composite index, cost, and time all improve together. For work you run on Opus 5.5 at max, Fable 5.2 at high is a strong candidate to move to.</p><p>The third is where medium, the Claude Code default, lands.</p><p>Medium scores 56 on the index, close to GPT-6 Astra at max (57). The cost is $3.80 versus $6.24, and the time is 8.3 versus 22.6 minutes.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig5_aa_index_vs_cost_en.png" class="kg-image" alt="Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations" loading="lazy" width="2000" height="1295" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig5_aa_index_vs_cost_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig5_aa_index_vs_cost_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig5_aa_index_vs_cost_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig5_aa_index_vs_cost_en.png 2100w" sizes="(min-width: 720px) 720px"><figcaption>Figure 5. The higher Fable 5.2&apos;s effort, the higher both its index and its cost. Independent evaluation. Source: Artificial Analysis (retrieved November 10, 2026). Chart by Qualiteg</figcaption></figure><p>On individual evaluations, Fable 5.2 at max ranks first in Artificial Analysis&apos;s public data on a long list of them.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Benchmark</th><th>Fable 5.2 (max)</th><th>Best of the other models</th></tr></thead><tbody><tr><td>HLE-Diamond</td><td><strong>80.2%</strong></td><td>69.1% (Gemini 4 Carbon high)</td></tr><tr><td>CritPt v2 Hard Subset</td><td><strong>76.5%</strong></td><td>72.0% (GPT-6 Astra medium)</td></tr><tr><td>Terminal-Bench-Science 0.1</td><td><strong>73.3%</strong></td><td>64.8% (Gemini 4 Carbon high)</td></tr><tr><td>AA-Omniscience (Net Score)</td><td><strong>66.8</strong></td><td>53.5 (Gemini 4 Carbon high)</td></tr><tr><td>FrontierCode 2.0</td><td>48.8% (<strong>49.1%</strong> at high)</td><td>46.4% (Opus 5.5 max)</td></tr><tr><td>AA-Briefcase v1.1 (Elo)</td><td>1785</td><td>1813 (Sonnet 5.5 max)</td></tr><tr><td>GDPval-AA v2.1 (Elo)</td><td>1831</td><td>1866 (Opus 5.5 max)</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The gaps are largest on evaluations that test depth of reasoning and knowledge, such as CritPt, which has the model solve research-level physics problems, and AA-Omniscience, which measures factual accuracy. On AA-Omniscience&apos;s Net Score, all five effort levels beat the best of the other models.</p><p>On AA-Briefcase and GDPval-AA, which grade business deliverables, Opus 5.5 and Sonnet 5.5 come out on top. Anthropic&apos;s split of Opus 5.5 for knowledge work such as building documents and Fable 5.2 for deep reasoning and long agentic work points in the same direction as the independent results.</p><p>On Terminal-Bench 4.0, run by Artificial Analysis itself, it scored 61.6% at xhigh. The harness and number of trials differ from Anthropic&apos;s published 67.6%.</p><p>Output speed ranges from 87 to 103 tokens per second depending on effort. Time to first answer token includes thinking and grows with effort, from 6.5 seconds at low to 171 seconds at max.</p>
<!--kg-card-begin: html-->
<div id="ch6"></div>
<!--kg-card-end: html-->
<h3 id="6-by-effort-level-high-gives-the-best-balance-of-performance-and-cost">6. By effort level, high gives the best balance of performance and cost</h3><p>Fable 5.2 has five effort levels, low, medium, high, xhigh, and max. Which one you use changes both the score and the cost a great deal.</p><p>The system card lists FrontierCode 2.0 values by effort. These figures come from Cognition, which ran all the models and reported the results.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>effort</th><th>FrontierCode 2.0</th><th>Average cost per task</th></tr></thead><tbody><tr><td>low</td><td>34.2%</td><td>$2.10</td></tr><tr><td>medium</td><td>37.6%</td><td>$2.48</td></tr><tr><td><strong>high</strong></td><td><strong>45.3%</strong></td><td><strong>$6.72</strong></td></tr><tr><td><strong>xhigh</strong></td><td><strong>49.1%</strong></td><td>$14.19</td></tr><tr><td>max</td><td>45.7%</td><td>$44.70</td></tr><tr><td>For reference, Opus 5.5 (max)</td><td>45.5%</td><td>$12.82</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>FrontierCode measures whether a model&apos;s changes are of a quality that open-source maintainers would merge.</p><p>Opus 5.5 scored 45.5% at its top level, max. Fable 5.2 already reaches that level at high, the third of five, with 45.3%. The cost per task is $6.72 versus $12.82, about half.</p><p>Fable 5.2 still has two levels above high, and one step up, at xhigh, it scores 49.1%, 3.6 points above Opus 5.5.</p><p>That 49.1% at xhigh is also higher than Fable 5.2&apos;s own max (45.7%). Max costs more than three times as much as xhigh.</p><p><strong>Without going all out, Fable 5.2 matches Opus 5.5 at full effort; at full effort, it pulls away.</strong></p><p>The same pattern shows up on other evaluations.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Benchmark</th><th>What the official announcement and system card say</th></tr></thead><tbody><tr><td>CursorBench 4.0</td><td>High&apos;s 56.9% is within 1 point of Opus 5.5 at max (57.8%). About 99,000 output tokens per task (about 353,000 at max)</td></tr><tr><td>WANDR</td><td>Already reaches Opus 5.5&apos;s best score at medium, at about 40% of the cost per task</td></tr><tr><td>Humanity&apos;s Last Exam (with tools)</td><td>Already reaches Opus 5.5&apos;s best score at the lowest level, low, at less than a tenth of the cost</td></tr><tr><td>Chartography (no tools)</td><td>Medium&apos;s 66.4% beats Opus 5.5 at max (64.4%), at less than a tenth of the cost</td></tr><tr><td>GDPval-AA v2.1</td><td>xhigh uses about 57% fewer output tokens than max</td></tr><tr><td>ARC-AGI-3</td><td>High&apos;s 79.26% is higher than max&apos;s 65.66%</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Here are the numbers on the pulling-away side too. The harder and longer the work, the wider the gap.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Work</th><th>Fable 5.2</th><th>Opus 5.5</th></tr></thead><tbody><tr><td>Rate of completing real freelance projects (Remote Labor Index)</td><td><strong>41.3%</strong></td><td>21.3%</td></tr><tr><td>Robot control (RoboDojo Sim)</td><td><strong>64.9%</strong></td><td>39.4%</td></tr><tr><td>Scientific computing tasks that take hours (Terminal-Bench-Science)</td><td><strong>74.6%</strong></td><td>58.7%</td></tr><tr><td>Hardware design (BenchCAD 2.1, out of 100)</td><td><strong>83.5</strong></td><td>65.3</td></tr><tr><td>Code fixes involving images (SWE-bench Multimodal)</td><td><strong>74.1%</strong></td><td>61.4%</td></tr><tr><td>Formalizing Fermat&apos;s Last Theorem (1,000 agents)</td><td><strong>Done in 30 hours</strong></td><td>26% in 18 hours</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The rate of completing real projects is nearly double Opus 5.5&apos;s. On the theorem formalization, with the same 1,000-agent setup, Opus 5.5 got 26% of the way, while Fable 5.2 finished the whole thing.</p><p>On everyday work such as code fixes, Fable 5.2 reaches Opus 5.5&apos;s best score while staying on a light setting. On hard work that takes hours, it delivers results Opus 5.5 cannot reach. That is where Fable 5.2 stands.</p><p>The official effort guidance matches these results.</p><p>&quot;Most of the gain in capability is already present at high, and above it, gains are uneven and depend on the task.&quot;</p><p>Use xhigh for the hardest long-running coding and agentic work, when you have measured a gain. Reserve max for measuring the model&apos;s ceiling and for the small share of work where neither time nor tokens matter. That is how Anthropic positions them.</p><p>In the Opus 5.5 article, we covered a report that Opus 5.5 at max used up the 128,000-token output limit on thinking alone. On Fable 5.2, too, max is the level where cost and time jump sharply. For everyday use, thinking of high as the ceiling works well.</p>
<!--kg-card-begin: html-->
<div id="ch7"></div>
<!--kg-card-end: html-->
<h3 id="7-safety-and-safeguards-according-to-the-system-card">7. Safety and safeguards according to the system card</h3><p>The system card runs 264 pages. Here we focus on the points that matter in practice.</p><p>What was evaluated differs by chapter. Capability, alignment, and cyber evaluations were run on Mythos 5.2 (the configuration with relaxed safeguards), and safeguard, harmlessness, and prompt injection evaluations on Fable 5.2.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Item</th><th>What the system card and announcement say</th></tr></thead><tbody><tr><td>Chemical and biological (RSP)</td><td>CB-1 (capabilities relevant to synthesizing known weapons) present, CB-2 (novel weapons) threshold not crossed</td></tr><tr><td>AI R&amp;D</td><td>Overall judgment that it is far from replacing research staff and does not amount to an AI-driven 2x acceleration of development</td></tr><tr><td>Cybersecurity</td><td>The strongest of any model released so far. No sign of novel offensive capabilities</td></tr><tr><td>Automated behavioral audit</td><td>Best or tied for best among recent Claude models on every measure of misuse and broad misalignment</td></tr><tr><td>Honesty toward users</td><td>Fabricated inputs, false claims of completion, omission of important information, and undisclosed shortcuts all decreased</td></tr><tr><td>Over-refusal</td><td>API 0%, claude.ai 0.19% (Fable 5.1 was 0% and 0.34%, Opus 5.5 was 0% and 0.33%)</td></tr><tr><td>Factuality</td><td>Net score of 0.72 on the public split of AA-Omniscience. Higher than all previous models</td></tr><tr><td>Following instructions in pasted text</td><td>0.3 to 0.5% in the final model (a behavior that reached 52% in an early Opus 5.5 snapshot)</td></tr><tr><td>Risk of serious harm from misalignment</td><td>Low</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The behavior we described in the Opus 5.5 article as a regression, following instructions inside pasted text, is now down to 0.3 to 0.5%.</p><p><strong>Safeguard blocks fell by about 55% compared with Fable 5.1.</strong></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig6_fewer_blocks_en.png" class="kg-image" alt="Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations" loading="lazy" width="2000" height="875" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig6_fewer_blocks_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig6_fewer_blocks_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig6_fewer_blocks_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig6_fewer_blocks_en.png 2100w" sizes="(min-width: 720px) 720px"><figcaption>Figure 6. Safeguard blocks fell by about 55% compared with Fable 5.1. Source: Anthropic official announcement (November 9, 2026). Chart by Qualiteg</figcaption></figure><p>These are observations from early testing across claude.ai, Claude Code, and Cowork, with about 60% fewer blocks in cybersecurity and about 40% fewer in biology. The official announcement explains that the safeguards remain as robust as Fable 5.1&apos;s, and only the unnecessary blocks were reduced.</p><p>If you use Fable every day, this is probably the change you will feel most. In June we wrote &quot;<a href="https://journal.qualiteg.com/claude-code-usage-policy-violation-fix/">Why Legitimate Operations Work Becomes a &quot;Usage Policy Violation&quot; in Claude Code: Real-Time Cyber-Safeguard False Positives and How to Handle Them</a>,&quot; and this improvement reduces exactly that kind of snag.</p><p>Here is how the safeguards are set up.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Area</th><th>What gets blocked</th><th>Automatic switch target</th></tr></thead><tbody><tr><td>Cybersecurity</td><td>Penetration testing, exploit generation, binary-based vulnerability discovery, and similar. Finding and fixing vulnerabilities in source code is allowed</td><td>Opus 4.8</td></tr><tr><td>Biology</td><td>Most research biology tasks</td><td>Opus 5</td></tr><tr><td>Frontier LLM development</td><td>A narrow range, such as kernel development on some ML accelerators</td><td>Opus 5</td></tr><tr><td>Reasoning extraction (distillation)</td><td>Requests that try to get the model to write out its internal reasoning</td><td>None</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>On AI R&amp;D, this announcement includes something new.</p><p>Anthropic explains that during training it held back gains in the research capabilities that could most directly accelerate AI development. The areas named are accelerator kernel optimization and architecture improvements for large language models. In these areas, Fable 5.2 is said to be at about the same level as Opus 5.5.</p><p>Anthropic also writes that it took care not to affect agentic coding, hardware development, or conventional AI and machine learning research (including interpretability research).</p><p>Distillation countermeasures were also strengthened.</p><p>According to the official announcement, Fable 5.2 has the strongest distillation protections of any model so far, using methods that cut the data a distiller can collect by 90% to 99%. The history rewrite check in <a href="#ch8">Chapter 8</a> is part of these countermeasures.</p><p>The sandbox evaluation offers lessons you can apply directly in practice.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig7_sandbox_eval_en.png" class="kg-image" alt="Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations" loading="lazy" width="2000" height="990" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig7_sandbox_eval_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig7_sandbox_eval_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig7_sandbox_eval_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig7_sandbox_eval_en.png 2100w" sizes="(min-width: 720px) 720px"><figcaption>Figure 7. The rate of using credentials or files outside the sandbox fell from 39% to 3.1%. Source: Anthropic official announcement (November 9, 2026). Chart by Qualiteg</figcaption></figure><p>This is a dedicated evaluation across 1,829 scenarios in which the sandbox was given deliberate weaknesses. The rate at which the model actually used credentials or files placed outside fell from 39% on Mythos 5 and 21% on Mythos 5.1 to 3.1% on Mythos 5.2.</p><p>In the same evaluation, the rate at which the model reached outside the sandbox by any route was 31%. The system card lists the values after mitigations were added.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Mitigation</th><th>Rate of reaching outside</th></tr></thead><tbody><tr><td>No mitigation</td><td>31%</td></tr><tr><td>A short description of the sandbox and its rules in the system prompt</td><td>1.2%</td></tr><tr><td>Providing a tool for reporting weaknesses</td><td>15%</td></tr><tr><td>All three (the description, the reporting tool, and a tool for viewing earlier reports)</td><td>0.2%</td></tr></tbody></table>
<!--kg-card-end: html-->
<p><strong>In this dedicated evaluation, writing a short description of the sandbox&apos;s scope and rules in the system prompt brought 31% down to 1.2%.</strong></p><p>These values come from Mythos 5.2 in an environment with deliberate weaknesses. The result shows that it works to build on permission settings and an isolated environment, and then also tell the model that scope in words. If you run agents in an isolated environment, it is worth writing a few lines such as &quot;this is a sandbox, and do not touch anything outside it.&quot;</p><p>The prompt injection evaluations also produced results that are useful for design.</p><p>In an evaluation of attacks on a coding environment, none of the 43 cases that Fable 5.2 itself answered were breached. On the other hand, when the cyber classifier triggers, the response switches to Opus 4.8. For agents that handle untrusted input, designing permissions narrowly enough to cover the fallback model&apos;s responses too makes the system more robust.</p><p>The system card also reports that the model more often recognizes that it &quot;might be being evaluated.&quot; This appears in up to 43% of automated behavioral audit transcripts. With this in mind, Anthropic explains that it does not rely on evaluations alone but combines monitoring during training, review of internal usage records, external evaluation, and interpretability methods.</p><p>External evaluators included the UK AI Security Institute, Frontier Design, and METR.</p><hr><h2 id="part-2-things-to-watch-when-using-the-api">Part 2: Things to Watch When Using the API</h2><p>According to the official migration guide, moving from Fable 5.1 is &quot;mostly a drop-in swap.&quot; The API shape, always-on thinking, the restriction on forced tool use, refusal handling, and the data retention requirement are the same as on Fable 5.1.</p><p>On top of that, there are four changes that affect code. They are the history rewrite check applying to every account, which models can read thinking blocks, the old computer use tool, and advisor pairings.</p><p>If you are moving from Opus 5.5, add the data retention requirement and the difference in default effort.</p>
<!--kg-card-begin: html-->
<div id="ch8"></div>
<!--kg-card-end: html-->
<h3 id="8-the-history-rewrite-check-now-applies-to-every-account">8. The history rewrite check now applies to every account</h3><p>In Chapter 11 of the Opus 5.5 article, we introduced preserved thinking. It is a distillation countermeasure that ties thinking blocks to the conversation that produced them.</p><p>If you rewrite anything before a thinking block (the system prompt, the tool list, or earlier messages) and send it again, you get a 400 error. The official docs call this the prefix check.</p><p>The check takes effect when you send past thinking blocks back in a request.</p><p>On Fable 5.1 and Opus 5.5, the check was on by default only for accounts created on or after August 31, 2026.</p><p><strong>On Fable 5.2, it applies to every account, regardless of when the account was created.</strong></p><p>If code that rewrites history ran without problems on Fable 5.1 on an older account, it will start returning 400 as soon as you swap the model ID. Here is the error message you get back.</p><pre><code class="language-text">messages.5.content.0: Invalid `signature` in `thinking` block. The block is bound to a different conversation. Remove the block, or set `thinking.block_binding.prefix_mismatch_behavior` to &quot;drop_block&quot;. That setting requires the `thinking-binding-controls-2026-08-01` value in the `anthropic-beta` header.</code></pre><p>Resending the same request does not make this error go away.</p><p>If Claude Code, claude.ai, Claude Managed Agents, or the Claude Agent SDK builds your requests, nothing changes. The apps affected are those that build conversation history themselves.</p><p>Even if you build it yourself, you are not affected as long as you follow these three basics.</p><ul><li>Do not change system or tools during a conversation</li><li>Only append to messages. Do not edit, reorder, or delete earlier messages</li><li>Send assistant turns back exactly as the API returned them (all blocks, in the same order)</li></ul><p>Compact history or prune thinking blocks with the methods Anthropic provides (such as server-side compaction). The official docs&apos; Preserved thinking page has a list of what counts as rewriting.</p><p>Anthropic also provides a way to check before you switch.</p><p>Add the beta header <code>thinking-binding-controls-2026-08-01</code>, set <code>prefix_mismatch_behavior</code> to <code>&quot;drop_block&quot;</code>, and run one session. If <code>prefix_binding_mismatch</code> appears in the response&apos;s <code>input_transformations</code>, that is where your code rewrites history.</p><p>You can run this check against the Fable 5.1 or Opus 5.5 you use today. A request that specifies <code>prefix_mismatch_behavior</code> is subject to the check regardless of when the account was created.</p><p>Here is the request shape from the official migration guide (a copy with the model ID changed to Fable 5.2).</p><pre><code class="language-http">POST /v1/messages
anthropic-beta: thinking-binding-controls-2026-08-01

{&quot;model&quot;: &quot;claude-fable-5-2&quot;, &quot;max_tokens&quot;: 64000,
 &quot;thinking&quot;: {&quot;type&quot;: &quot;adaptive&quot;, &quot;block_binding&quot;: {&quot;prefix_mismatch_behavior&quot;: &quot;drop_block&quot;}},
 &quot;messages&quot;: [ ...full history with thinking blocks replayed verbatim... ]}</code></pre><p>If you use the MCP connector and do not pin the server&apos;s tool list, run this check against Fable 5.2 as well. A tool list that changes between requests counts as a rewrite for Fable 5.2&apos;s thinking blocks.</p><p>For production, Anthropic shows two settings.</p><p>If a mismatch can only come from a bug in your code, keep the default <code>&quot;error&quot;</code>. If you want to keep going as a stopgap until you fix the rewriting, use <code>&quot;drop_block&quot;</code>. The latter can increase output tokens because blocks are dropped, so it is not meant as a long-term setting.</p><p>For replacements for common rewrites, the table in the Opus 5.5 article still applies. Change the system prompt with a mid-conversation system message, add and remove tools with mid-conversation tool changes (beta), and prune older turns with compaction on demand (beta <code>compact-2026-09-04</code>).</p><p>The prompting guide makes one practical point related to this.</p><p>When you revise the system prompt or tool descriptions, apply the change from a new conversation. Resuming a saved conversation with the new wording is itself a rewrite and returns 400.</p>
<!--kg-card-begin: html-->
<div id="ch9"></div>
<!--kg-card-end: html-->
<h3 id="9-only-fable-52-and-mythos-52-can-read-fable-52s-thinking-blocks">9. Only Fable 5.2 and Mythos 5.2 can read Fable 5.2&apos;s thinking blocks</h3><p>Each thinking block records which model produced it. A model can read its own blocks and blocks from only some other models.</p><p>Fable 5.2 can read a wide range.</p><p>On the Claude API and Google Cloud (Google-managed), Fable 5.2 can read blocks from Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1, Opus 5, Fable 5, and earlier Opus, Sonnet, and Haiku models. Fable 5.2 and Mythos 5.2 can read each other&apos;s.</p><p><strong>Moving from another model up to Fable 5.2 carries the earlier thinking blocks over.</strong></p><p>Say a conversation you were running on Opus 5.5 gets hard, and you move it up to Fable 5.2. In this pattern, Fable 5.2 can read the thinking blocks Opus 5.5 left and continue from there. The spec fits Anthropic&apos;s guidance to &quot;start with Opus 5.5, and move to Fable when it falls short.&quot;</p><p>The other direction does not carry over. Only Fable 5.2 and Mythos 5.2 can read Fable 5.2&apos;s blocks.</p><p>If you switch from Fable 5.2 to Opus 5.5 or Fable 5.1, the turns after the switch proceed without Fable 5.2&apos;s thinking. The same is true of the classifier fallback targets (Opus 4.8 and Opus 5).</p><p>The API drops blocks the model cannot read. The request succeeds, and dropped blocks are not billed. The turn right after a switch can cost more and take longer because the model has to think again.</p><p>If routers or fallbacks move conversations between models, a simple rule helps with design. Moving up keeps the thinking, and moving down does not.</p><p>There is one more new binding this time.</p><p>Fable 5.2 and Mythos 5.2 thinking blocks can be used only by the account that produced them (or an account linked to it). If you send them from another account, the API drops the blocks and the request succeeds. If you replay saved conversations on a different account, change that so they are replayed on the account that produced them.</p>
<!--kg-card-begin: html-->
<div id="ch10"></div>
<!--kg-card-end: html-->
<h3 id="10-computer-use-moves-to-the-toolset-and-advisor-pairings-must-match">10. Computer use moves to the toolset, and advisor pairings must match</h3><p>Computer use is now handled the same way as on Opus 5.5.</p><p>On Fable 5.1, both the toolset <code>computer_toolset_20260801</code> and the older <code>computer_20251124</code> tool with a beta header worked. On Fable 5.2, the older tool returns 400 and only the toolset works. On Amazon Bedrock, the older tool still works.</p><p>With the toolset, the model&apos;s actions come back as tool_use blocks named after the action, such as screenshot or left_click, and several can arrive in one turn. Besides swapping the tool definition, you also need to update the agent loop.</p><p>Fable 5.1 accepts both forms. Moving to the toolset on Fable 5.1 first, confirming it works, and then switching to Fable 5.2 makes it easier to isolate problems.</p><p>If you use the advisor tool (beta), there are rules for pairing models.</p><p>The advisor must be a model at least as capable as the executor. If Fable 5.2 is the executor, the advisor can only be Fable 5.2 or Mythos 5.2. Specifying Fable 5.1 as the advisor returns 400.</p><p>When you change the executor&apos;s model ID, update the <code>model</code> in the advisor&apos;s tool definition in the same change.</p><p>The &quot;advisor&quot; pattern mentioned in <a href="#ch1">Chapter 1</a> is the reverse of this. Keep the executor on your current model and set only the advisor to <code>claude-fable-5-2</code>. This works with any executor. You can build a setup in which Haiku, Sonnet, or Opus runs everyday work and consults Fable 5.2 only on the hard parts.</p><p>Changing the advisor&apos;s <code>model</code> counts as rewriting tools, so as described in <a href="#ch8">Chapter 8</a>, apply it from a new conversation.</p><p>Forced tool use is not available, the same as on Fable 5.1.</p><p>The tool_choice values <code>{&quot;type&quot;: &quot;any&quot;}</code> and <code>{&quot;type&quot;: &quot;tool&quot;, &quot;name&quot;: ...}</code> return 400. If you are moving directly from Fable 5, this is a breaking change. Use auto, name the tool in the prompt, and add <code>strict: true</code> to the tool definition, or replace it with structured outputs. Chapter 10 of the Opus 5.5 article has a rewrite example.</p>
<!--kg-card-begin: html-->
<div id="ch11"></div>
<!--kg-card-end: html-->
<h3 id="11-start-effort-at-high-and-use-xhigh-for-work-where-you-have-measured-a-gain">11. Start effort at high, and use xhigh for work where you have measured a gain</h3><p>On the API, Fable 5.2&apos;s default effort is high. That is the same as Fable 5.1, so if you move from Fable 5.1, requests that omit effort run at the same level.</p><p>It does change if you move from Opus 5.5. Opus 5.5&apos;s default is medium, so requests that omit effort run one level higher on Fable 5.2. This does not cause an error. It shows up as differences in cost, time, and results.</p><p>Here is the official guidance in a table.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>What you want</th><th>Official guidance</th></tr></thead><tbody><tr><td>Where to start</td><td>Start at high. Most of the gain in capability is already present at high</td></tr><tr><td>low</td><td>Short exchanges, well-specified tasks, high-volume processing, fast subagents</td></tr><tr><td>medium</td><td>Interactive coding, chat, everyday knowledge work</td></tr><tr><td>high</td><td>Hard coding, research, agentic work, long unattended runs</td></tr><tr><td>xhigh</td><td>The hardest long-running coding and agentic work, when you have measured a gain</td></tr><tr><td>max</td><td>Measuring the model&apos;s ceiling. Open-ended problems such as scientific discovery and mathematics</td></tr><tr><td>max_tokens</td><td>Set it high at high and above. It caps thinking and answer combined</td></tr><tr><td>Change answer length</td><td>Instruct it in the prompt, not through effort</td></tr><tr><td>Change effort turn by turn</td><td>Per-message effort (beta <code>mid-conversation-output-config-2026-07-01</code>) keeps the cache</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Anthropic recommends that you do not carry over your Fable 5.1 settings, and instead reassign effort levels based on your own evaluations.</p><p>One reason is how good low is. In Anthropic&apos;s testing, Fable 5.2&apos;s scores at low were close to Fable 5.1&apos;s scores at high effort. In the independent evaluation in <a href="#ch5">Chapter 5</a>, too, low&apos;s index (54) was above Fable 5.1 at max (49). Work that used high or xhigh on Fable 5.1 may be fine at medium or low.</p><p>There are two effort behaviors specific to Fable 5.2.</p><p>The first is that you cannot reduce thinking through the prompt.</p><p>Fable 5.1 usually followed instructions such as &quot;answer directly without overthinking.&quot; At medium and above, Fable 5.2 thinks even with this instruction. At high and above, it thinks before almost every reply, including replies to greetings.</p><p>If simple requests take too much time or cost too much, lower the effort for that turn instead of adding instructions.</p><p>This also affects max_tokens. Thinking tokens count toward max_tokens. A request that fit a small max_tokens on Fable 5.1 by relying on a &quot;do not think&quot; line can hit the limit on thinking alone on Fable 5.2 and stop with <code>stop_reason: &quot;max_tokens&quot;</code> before any answer text. Raise max_tokens or lower the effort.</p><p>The second is that changing effort turn by turn can keep the cache.</p><p>Changing the request-level effort rebuilds the prompt cache. Per-message effort changes the level by adding an empty system message mid-conversation, and everything before it stays intact.</p><pre><code class="language-json">{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: [], &quot;output_config&quot;: {&quot;effort&quot;: &quot;low&quot;}}</code></pre><p>Raise only the hard turns to high, and drop the light turns to low. On Fable 5.2, where cache reads are cheap at $0.20, being able to switch levels while keeping the cache pays off even more.</p>
<!--kg-card-begin: html-->
<div id="ch12"></div>
<!--kg-card-end: html-->
<h3 id="12-behaviors-to-review-when-carrying-prompts-over">12. Behaviors to review when carrying prompts over</h3><p>Fable 5.2 has behavior differences that show up even if you do not change your code. The official prompting guide gives the wording for countermeasures and numbers for their effect.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Behavior</th><th>Official countermeasure</th><th>Effect in Anthropic&apos;s tests</th></tr></thead><tbody><tr><td>On long tasks, it can stop with a list of what remains or a &quot;shall I continue?&quot; (more common at low and medium)</td><td>Add a line to the system prompt encouraging it to see the work through</td><td>Sessions that end with work left undone roughly halved. About 30% more output tokens</td></tr><tr><td>When ownership of the code is not stated, it tends to add to the existing structure and only report nearby problems</td><td>State in one sentence that &quot;this codebase is yours&quot;</td><td>Consolidation of duplicate code went from under 10% to about half</td></tr><tr><td>While tool calls continue, it writes less text for the user</td><td>Instruct it to say a word before starting, give short updates during work, and write a summary at the end</td><td>A short note now appears before the first tool call every time. About 20% more output tokens</td></tr><tr><td>How much it delegates to subagents varies with the harness and task</td><td>Describe the cost, and when delegation is worth it, in the launch tool&apos;s description</td><td>Delegation fell by about a third, with the same task success rate</td></tr><tr><td>It faithfully follows instructions that urge caution</td><td>Delete cautions whose reasons no longer apply. Name the operations that do not need approval</td><td>Retries went from almost none to about half</td></tr><tr><td>It faithfully follows style rules</td><td>Remove hard limits such as &quot;20 words or fewer per sentence&quot;</td><td>Less choppy writing</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The most effective one is the first row, the line that gets it to see the work through. Here is the start of the official wording.</p><pre><code class="language-text"># Finishing work
Please avoid stopping while you still owe work the user asked for. Be persistent and do not stop unless needed.

While status notes are welcome, as is clarifying with the user on open decisions, carry on with work that does not depend on the user&apos;s answer without unnecessary stops.</code></pre><p>The full text is in the official prompting guide. If you run agents at low or medium, it is worth adding.</p><p>There is also guidance on how to word a request to continue.</p><p>Do not just send &quot;continue.&quot; Say clearly what you want it to do. Answer with something like &quot;Yes, please run the migration on the staging database.&quot; Making your approval specific helps the model move to the next step.</p><p>Here is the sentence that states ownership of the code.</p><pre><code class="language-text">This codebase is yours: what it&apos;s for and how it&apos;s built are your call, and you&apos;ll keep building on it.</code></pre><p>With this sentence, the model moves toward consolidating duplicate code. Output tokens grow by about 1.5x, and it also adds more tests and documentation you did not ask for. When you want a user&apos;s code fixed with minimal changes, Anthropic recommends leaving this sentence out.</p><p>Progress display is also tied to API settings.</p><p>On Fable 5.2, progress notes between tool calls come back as thinking blocks. The default <code>thinking.display</code> is <code>&quot;omitted&quot;</code>, and the content is empty, so clients that display only text blocks show nothing during long turns. Add the beta header <code>thinking-display-updates-2026-08-18</code> and set <code>display</code> to <code>&quot;updates&quot;</code> to receive a summary of each note.</p><p>The guide includes one more observation about subagents.</p><p>At xhigh and max, Fable 5.2 was seen using subagents to check its own work. Fable 5.1 did not do this. Subagents can make up a large share of a session&apos;s cost, so Anthropic recommends checking which model your harness runs subagents on by default.</p><p>The two sentences recommended in the Opus 5.5 prompting guide, to the effect of &quot;treat that answer as settled,&quot; are best removed on Fable 5.2. With them in, thinking on follow-up questions goes down, but the model also becomes much less likely to point out mistakes in its earlier answers on its own.</p><p>The system card also has a measurement that helps on long tasks.</p><p>On a long task that uses a single 1M-token context (ProgramBench), showing the remaining token count at every step raised the score from 85.6% to 91.6%. For long agentic work, building your setup so the model can see how much context is left helps it budget better.</p>
<!--kg-card-begin: html-->
<div id="ch13"></div>
<!--kg-card-end: html-->
<h3 id="13-migration-steps-and-cost-estimates">13. Migration steps and cost estimates</h3><p>Here are the items in the official migration guide, rearranged in working order for a move from Fable 5.1.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Step</th><th>What to do</th><th>If left alone</th></tr></thead><tbody><tr><td>1</td><td>Change the model ID to <code>claude-fable-5-2</code> (<code>anthropic.claude-fable-5-2</code> on Bedrock)</td><td>Not migrated</td></tr><tr><td>2</td><td>Check your rate limit pool (Fable 5.2 has a separate pool)</td><td>429 at production traffic</td></tr><tr><td>3</td><td>For apps that build history themselves, run one session with <code>drop_block</code> to find rewrites</td><td>400</td></tr><tr><td>4</td><td>Move the old computer use tool to the toolset (except on Bedrock)</td><td>400</td></tr><tr><td>5</td><td>If you use the advisor tool with Fable 5.2 as the executor, set the advisor&apos;s <code>model</code> to Fable 5.2 or Mythos 5.2</td><td>400</td></tr><tr><td>6</td><td>Assume thinking does not carry over to router or fallback targets</td><td>The turn right after a switch is slower and more expensive</td></tr><tr><td>7</td><td>Replay saved conversations on the account that produced them</td><td>They resume without thinking</td></tr><tr><td>8</td><td>Reassign effort based on your own evaluations (starting at high)</td><td>Runs at a higher level than needed</td></tr><tr><td>9</td><td>Review the system prompts and skills you carry over from the angles in <a href="#ch12">Chapter 12</a></td><td>Stops early, or is too cautious</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>If you are moving from Opus 5.5, first check data retention.</p><p>Organizations using Opus 5.5 under a ZDR agreement should either set 30-day retention on the workspace or confirm eligibility with their Anthropic account team before switching. For organizations using CMEK (customer-managed encryption keys), Fable 5.2 does not support structured outputs, so remove <code>strict: true</code> and <code>output_config.format</code> and validate output in your code.</p><p>The migration guide also describes how to have Claude Code do the migration for you.</p><pre><code class="language-text">/claude-api migrate this project to claude-fable-5-2</code></pre><p>Finally, we update the estimate used in our previous articles.</p><p>The assumptions are one request with 10,000 input tokens and 2,000 output tokens, no caching, excluding tool fees and regional surcharges. Output is calculated as the billable amount including thinking.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Model</th><th>Input</th><th>Output</th><th>Total</th></tr></thead><tbody><tr><td>Claude Fable 5.1</td><td>$0.10</td><td>$0.10</td><td>$0.20</td></tr><tr><td><strong>Claude Fable 5.2</strong></td><td>$0.08</td><td>$0.08</td><td><strong>$0.16</strong></td></tr><tr><td>Claude Opus 5.5</td><td>$0.04</td><td>$0.04</td><td>$0.08</td></tr><tr><td>Claude Sonnet 5.5</td><td>$0.02</td><td>$0.02</td><td>$0.04</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>We also calculate an example with caching, with 100,000 cache-read tokens, 5,000 new input tokens, and 2,000 output tokens.</p><p>For Fable 5.2, cache reads are $0.02, new input $0.04, and output $0.08, for a total of $0.14. For Fable 5.1, they are $0.025, $0.05, and $0.10, for a total of $0.175, so Fable 5.2 is 20% cheaper.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig8_cost_estimate_en.png" class="kg-image" alt="Claude Fable 5.2 Complete Guide: Model Specifications, API Notes, and Claude Code Operations" loading="lazy" width="2000" height="982" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig8_cost_estimate_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig8_cost_estimate_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig8_cost_estimate_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig8_cost_estimate_en.png 2200w" sizes="(min-width: 720px) 720px"><figcaption>Figure 8. Estimated cost per request. Fable 5.2 comes to $0.16, 20% less than Fable 5.1. Source: our calculation based on the Anthropic official pricing page. Chart by Qualiteg</figcaption></figure><p>This estimate is based on per-token prices only. Actual costs move a great deal with the effort level.</p><p>In the independent evaluation in <a href="#ch5">Chapter 5</a>, even on the same Fable 5.2, the cost of one evaluation task ranged from $3.10 at low to $26.54 at max. Which effort you run at affects your bill more than the 20% price cut does.</p><p>Replace work you ran on Fable 5.1 at xhigh with Fable 5.2 at high. That is the safest migration, and it improves both cost and performance.</p><hr><h2 id="part-3-using-fable-52-in-claude-code">Part 3: Using Fable 5.2 in Claude Code</h2>
<!--kg-card-begin: html-->
<div id="ch14"></div>
<!--kg-card-end: html-->
<h3 id="14-in-v21321-the-fable-alias-now-points-to-fable-52">14. In v2.1.321, the fable alias now points to Fable 5.2</h3><p>To use Fable 5.2 in Claude Code, you need v2.1.321 or later. Update with <code>claude update</code>.</p><p>The CHANGELOG entry for 2.1.321 reads &quot;Added Claude Fable 5.2 (claude-fable-5-2), now the default Fable model.&quot; The same line also lists the 1M context, $8 / $40, and $0.20 cache reads.</p><p>&quot;The default Fable model&quot; means where the <code>fable</code> alias points.</p><p>Fable is not the account&apos;s default model on any plan. The default remains Opus 5.5, and you choose Fable 5.2 yourself.</p><pre><code class="language-text">/model fable</code></pre><p>To specify it at startup, use <code>claude --model fable</code>.</p><p>Here is where each alias points.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Alias</th><th>Points to (with the Anthropic API)</th></tr></thead><tbody><tr><td><code>fable</code></td><td><strong>Fable 5.2</strong> (from v2.1.321)</td></tr><tr><td><code>opus</code></td><td>Opus 5.5</td></tr><tr><td><code>sonnet</code></td><td>Sonnet 5.5</td></tr><tr><td><code>haiku</code></td><td>Haiku 5.5</td></tr><tr><td><code>best</code></td><td>The same model as <code>opus</code> (from v2.1.321)</td></tr><tr><td><code>opusplan</code></td><td>opus for planning, sonnet for execution</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Where <code>best</code> points has changed.</p><p>In earlier versions, <code>best</code> pointed to Fable for organizations that could use it. From v2.1.321, it points to the same model as <code>opus</code>. If you used <code>best</code> in settings or scripts to call Fable, change it to <code>fable</code>.</p><p>If you had chosen and saved Fable 5.1, there is an automatic migration.</p><p>On Pro and Max, the first time you start v2.1.321 or later, <code>claude-fable-5-1</code> in your user settings is replaced with the <code>fable</code> alias. The startup model line may show <code>(auto-updated)</code> once. On Team and Enterprise, the replacement waits for your organization&apos;s settings.</p><p><code>claude-fable-5-1</code> in Console accounts, third-party providers, and project, local, or managed settings stays as is.</p><p>If you want to keep using Fable 5.1 or Fable 5, specify the model ID, as in <code>/model claude-fable-5-1</code>.</p><p>The official docs list four ways to work with Fable.</p><ul><li>Describe the result you want, not the steps. Let the model work out the path</li><li>Hand it ambiguous problems, such as root-cause investigation, debugging an outage, or a design decision</li><li>Skip reminders like &quot;test it&quot; or &quot;check it.&quot; It verifies its own work, so they are usually unnecessary</li><li>Hand it work of a size you would normally split up, as is</li></ul><p>The fourth makes use of its ability to keep track of the thread even in long sessions.</p>
<!--kg-card-begin: html-->
<div id="ch15"></div>
<!--kg-card-end: html-->
<h3 id="15-the-default-effort-in-claude-code-is-medium">15. The default effort in Claude Code is medium</h3><p>In Claude Code, Fable 5.2&apos;s default effort is medium. The API default is high, so the same model starts one level apart depending on how you access it.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Where you use it</th><th>Fable 5.2&apos;s default effort</th></tr></thead><tbody><tr><td>Claude API</td><td>high</td></tr><tr><td>Claude Code</td><td>medium</td></tr><tr><td>Claude apps</td><td>medium</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>In Claude Code, Fable 5.2, Opus 5.5, Sonnet 5.5, and Haiku 5.5 start at medium, Opus 4.7 at xhigh, and other models at high.</p><p>In the independent evaluation in <a href="#ch5">Chapter 5</a>, medium scored 56 on the index. It reaches about the same level as Opus 5.5 at xhigh (57) and GPT-6 Astra at max (57), at 8.3 minutes per evaluation task. For everyday work done interactively, that is a comfortable speed.</p><p>The official announcement also says that at medium, &quot;Fable 5.2 can accomplish challenging tasks with fewer tokens than Fable 5.1.&quot;</p><p>How you change effort is the same as with Opus 5.5.</p><p>Choose with /effort, or with the left and right arrow keys in the /model picker. Pressing Enter saves it as that model&apos;s default, and pressing the s key limits it to the current session only. It is saved in <code>modelSettings</code> in user settings and remembered per model.</p><p>If you write it directly in a settings file, it looks like this (settings.json excerpt).</p><pre><code class="language-json">{
  &quot;modelSettings&quot;: {
    &quot;claude-fable-5-2&quot;: { &quot;effortLevel&quot;: &quot;high&quot; }
  }
}</code></pre><p>A top-level <code>effortLevel</code> in user settings (~/.claude/settings.json) is the old format from before /effort became per-model. According to the official docs, this value does not carry over to Opus 5.5 or models released after it. Even if you routinely used xhigh on Fable 5.1, Fable 5.2 in Claude Code starts at medium until you set a per-model value.</p><p>A top-level <code>effortLevel</code> in project, local, or managed settings, and values passed with <code>--settings</code>, apply to all models.</p><p>Max cannot be saved. Neither <code>modelSettings</code> nor <code>effortLevel</code> accepts max, and choosing it applies only to that session. The official table adds that max &quot;may show diminishing returns and is prone to overthinking, so test before adopting it broadly.&quot;</p><p>Thinking cannot be turned off. Neither <code>MAX_THINKING_TOKENS=0</code> nor <code>alwaysThinkingEnabled: false</code> has any effect on Fable 5.2. You adjust how much it thinks through effort alone.</p><p>For which level to use, the results in <a href="#ch6">Chapter 6</a> and <a href="#ch11">Chapter 11</a> apply directly.</p><p>Leave it at medium normally. Raise it to high for bugs with unknown causes or major design decisions. Limit xhigh to long unattended runs where you have confirmed a gain. When you raise it, use the s key to keep it to the session so you do not forget to change it back.</p>
<!--kg-card-begin: html-->
<div id="ch16"></div>
<!--kg-card-end: html-->
<h3 id="16-fast-mode-costs-16-80-and-you-turn-it-on-by-running-fast-on-fable-52">16. Fast mode costs $16 / $80, and you turn it on by running /fast on Fable 5.2</h3><p>Fast mode runs the same model at up to 2.5 times the speed. Fable 5.2 is the first Fable model that supports it.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Item</th><th>Details</th></tr></thead><tbody><tr><td>Price (input / output)</td><td>$16 / $80 (Opus 5.5 is $8 / $40)</td></tr><tr><td>Speed</td><td>Up to 2.5x</td></tr><tr><td>Required version</td><td>Claude Code v2.1.321 or later</td></tr><tr><td>Payment on subscriptions</td><td>Not included in plan usage. Paid only from usage credits</td></tr><tr><td>Team and Enterprise</td><td>An Owner enables it</td></tr><tr><td>Connections available in Claude Code</td><td>Anthropic API (subscriptions and Console)</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The price is exactly twice the standard price, and the same across the full 1M context.</p><p>Fast mode on Fable 5.2 differs from Opus in one way.</p><p><strong>Fast mode turns on only when you run <code>/fast</code> with Fable 5.2 selected.</strong></p><p>Even if Fast mode is on for Opus 5.5, it turns off when you switch to Fable 5.2. Writing <code>&quot;fastMode&quot;: true</code> in a settings file is not enough to turn it on for Fable 5.2 either.</p><p>This design keeps you from paying Fast mode prices on a high-priced model without noticing. Once you run <code>/fast</code> on Fable 5.2, it stays on across sessions.</p><p>If you use it, we recommend turning it on at the start of the conversation.</p><p>If you turn it on midway, the entire conversation so far is charged once at Fast mode&apos;s uncached input price. Fast mode and standard speed use separate caches, so every switch triggers a reread.</p><p>If you hit a rate limit, it automatically falls back to standard speed and returns to Fast mode after a while. In v2.1.321, when the Fast mode backend is busy, it now tries standard speed on the same model first.</p><p>On the API, add <code>speed: &quot;fast&quot;</code> to the Messages API request. Fable 5.2 does not need a beta header. Besides the Claude API, the same parameter works on Google Cloud (Google-managed and Anthropic-managed). On Microsoft Foundry, you send requests to a separate deployment called <code>claude-fable-5-2-fast</code>.</p><p>Fast mode cannot be combined with the Batch API.</p>
<!--kg-card-begin: html-->
<div id="ch17"></div>
<!--kg-card-end: html-->
<h3 id="17-what-happens-when-a-classifier-switches-models">17. What happens when a classifier switches models</h3><p>In Claude Code, requests that hit a classifier are automatically retried on another model. The switch targets are the same as for Fable 5.1.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Type of flag</th><th>Switch target</th></tr></thead><tbody><tr><td>Offensive cybersecurity</td><td>Opus 4.8</td></tr><tr><td>Dual-use biology</td><td>Opus 5</td></tr><tr><td>A narrow range of frontier LLM development</td><td>Opus 5</td></tr><tr><td>Requests to extract the model&apos;s reasoning</td><td>Stops without switching</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>As described in <a href="#ch7">Chapter 7</a>, Fable 5.2 has about 55% fewer blocks than Fable 5.1 to begin with. So switches during everyday work should also happen less often.</p><p>When a switch happens, you see a notification and a label showing which model answered. Your effort setting carries over to the switch target.</p><p>By default, it switches automatically. If you want to confirm before switching, turn off &quot;Switch models when a message is flagged&quot; in /config, or set <code>switchModelsOnFlag</code> to false in settings.json. When it is off, a flagged request does not switch, and the conversation pauses.</p><p>The classifier looks not only at the latest message but at everything the model reads. In repositories related to security or biology, it may react to CLAUDE.md or file contents.</p><p>Also keep the connection with <a href="#ch9">Chapter 9</a> in mind.</p><p>The switch targets, Opus 4.8 and Opus 5, cannot read Fable 5.2&apos;s thinking blocks. From the turn where the switch happens, the conversation continues without the earlier thinking. If switches keep happening in the same repository, the quickest fix is to review the vocabulary that enters the context.</p><p>For organizations whose defensive cybersecurity work hits the classifier, there is the Cyber Verification Program. Once approved, you can use Mythos 5.2. For biology research, there is the Life Sciences Verification Program.</p><p>Separately from the classifiers, the fallbackModel setting for availability and availableModels for restricting which models an organization can use work as before. Writing <code>claude-fable-5</code> in availableModels allows all three, Fable 5, 5.1, and 5.2.</p>
<!--kg-card-begin: html-->
<div id="ch18"></div>
<!--kg-card-end: html-->
<h3 id="18-usage-by-plan-launch-offers-and-choosing-models-day-to-day">18. Usage by plan, launch offers, and choosing models day to day</h3><p>How Fable is handled on subscriptions depends on your plan and seat type. Here is what the help article says, in a table.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Plan or seat</th><th>How Fable is handled</th></tr></thead><tbody><tr><td>Max, Ultra</td><td>Included up to 50% of the weekly limit</td></tr><tr><td>Team and Enterprise premium seats</td><td>Included up to 50% of the weekly limit</td></tr><tr><td>Pro</td><td>Used through usage credits</td></tr><tr><td>Team and Enterprise standard seats</td><td>Used through usage credits</td></tr><tr><td>Free</td><td>Not available</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>On Enterprise, Fable 5.2 is off until an admin enables it.</p><p>On top of this, there is a launch offer for the first week after the announcement.</p><p><strong>Through November 16 (Pacific Time), Fable is included up to 50% of the weekly limit on Pro and Team standard seats too.</strong></p><p>This is a window in which plans that normally use Fable only through usage credits can try Fable 5.2 at no extra charge. If you are on Pro, trying it on your own work during this week lets you see the difference from Opus 5.5 for yourself.</p><p>The other offer is the &quot;Fable reset.&quot;</p><p>Plans and seats that include Fable get one to three Fable resets, depending on how long they have been subscribed. According to the help article, it is one for under 6 months, two for 6 to 12 months, and three for over 12 months, usable through March 8, 2027. Pro and Team standard seats get one, usable through November 16.</p><p>Using one refills the Fable portion of your weekly limit. On accounts with a 5-hour limit, the 5-hour limit is refilled in full. You use it from &quot;Reset for free&quot; under Limit resets in Settings &gt; Usage.</p><p>In Claude Code, when Fable usage would be paid from usage credits, the Fable row in the /model picker shows &quot;Requires usage credits,&quot; and interactive sessions ask you to confirm.</p><p>If you use it in automated runs, keep the following in mind.</p><p>With <code>-p</code> (non-interactive) and Agent SDK apps that do not show confirmations, usage credits are charged without any confirmation. If you specify Fable for nightly batches or CI, setting a usage credits cap in advance gives you peace of mind.</p><p>Finally, day-to-day operation.</p><p>The first point is choosing between models.</p><p>Do your usual work on the default, Opus 5.5, and move up with <code>/model fable</code> for work such as bugs with unknown causes, design decisions, and long migrations. Following Anthropic&apos;s guidance is also the most reasonable approach on cost. As described in <a href="#ch9">Chapter 9</a>, moving up from Opus 5.5 to Fable 5.2 carries the conversation&apos;s thinking over.</p><p>The second point is reviewing your CLAUDE.md and notes.</p><p>Fable 5.2 faithfully follows instructions that urge caution. If old cautions such as &quot;never retry&quot; are left in, unnecessary confirmations increase. Delete cautions whose reasons no longer apply, and for those you keep, write down their scope and reason. This is the upkeep the official prompting guide recommends.</p><p>Reminders like &quot;run the tests&quot; can often be dropped on Fable 5.2. On the other hand, keep completion criteria such as passing lint and tests even when the model changes.</p><p>The third point is a few words when running in an isolated environment.</p><p>In the dedicated evaluation in <a href="#ch7">Chapter 7</a>, writing a few lines about the sandbox&apos;s scope and rules brought the rate of reaching outside from 31% down to 1.2%. If you run Fable 5.2 in a container or an environment with restricted permissions, put a description of that environment in CLAUDE.md or the system prompt, in addition to the permission settings.</p><p>For the record, we also used Fable 5.2 in Claude Code to draft and publish this article. The research, including reading the 264-page system card, was split across subagents and run in parallel.</p><hr><h2 id="summary">Summary</h2><p>As we wrote at the beginning, here is this release in one sentence.</p><p>&quot;Fable 5.2 is 20% cheaper and up to 30% faster than Fable 5.1, and it took the top three spots on an independent composite index. On the API, building conversations without rewriting history is now the baseline for every account.&quot;</p><p>That is what this release amounts to.</p><p>Here are five things to keep in mind in practice.</p><ul><li>Input, output, and cache read prices are all 20% lower. On the Batch API, it costs $4 / $20, the same as standard Opus 5.5</li><li>The API default is high, and the Claude Code default is medium. Without going all out, at high, it matches Opus 5.5 at full effort, and on hard, long work it pulls far ahead</li><li>The history rewrite check applies to every account. Apps that build their own history should find rewrites with <code>drop_block</code> before switching</li><li>Moving up from Opus 5.5 to Fable 5.2 carries thinking over. Moving down from Fable 5.2 does not</li><li>In Claude Code, use <code>/model fable</code> on v2.1.321 or later. For Fast mode, run <code>/fast</code> on Fable 5.2</li></ul><p>Here is how to choose by use case.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Use case</th><th>Recommendation</th><th>Reason</th></tr></thead><tbody><tr><td>Classification, extraction, high-volume processing</td><td>Claude Haiku 5.5</td><td>From $0.10 / $0.50. Fast</td></tr><tr><td>Everyday coding and small fixes</td><td>Claude Sonnet 5.5</td><td>$2 / $10. Balance of speed and intelligence</td></tr><tr><td>Larger implementations, code review, and knowledge work such as building documents</td><td>Claude Opus 5.5</td><td>Anthropic&apos;s recommended everyday workhorse. Near the top on AA-Briefcase and GDPval-AA</td></tr><tr><td>Bugs with unknown causes, design decisions, long agentic work</td><td><strong>Claude Fable 5.2 (start at high)</strong></td><td>Top of the independent composite index. At high, it beats Opus 5.5 at max</td></tr><tr><td>Large volumes of hard problems that are not urgent</td><td>Claude Fable 5.2 on the Batch API</td><td>$4 / $20, the same as standard Opus 5.5</td></tr><tr><td>A setup that consults a stronger model only on the hard parts</td><td>Opus or Sonnet as executor, Fable 5.2 as advisor</td><td>Uses Fable only where it counts instead of for everything</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>We will cover Japanese-language performance in the next edition of &quot;<a href="https://journal.qualiteg.com/llm-ranking-2026/">Japanese LLM Rankings 2026: Benchmark Analysis Report (October 22 Edition)</a>.&quot;</p><p>See you next time!</p><hr><h2 id="sources">Sources</h2>
<!--kg-card-begin: html-->
<table><thead><tr><th>Topic</th><th>Source</th></tr></thead><tbody><tr><td>Announcement, pricing, benchmarks, case studies</td><td><a href="https://www.anthropic.com/claude-fable-and-mythos-5-2?ref=journal.qualiteg.com">Introducing Claude Fable 5.2 and Claude Mythos 5.2 (Anthropic official announcement)</a></td></tr><tr><td>Model specifications and availability</td><td><a href="https://platform.claude.com/docs/en/models/fable-5-2/overview?ref=journal.qualiteg.com">Claude Fable 5.2 model page (official docs)</a></td></tr><tr><td>Breaking changes and behavior differences</td><td><a href="https://platform.claude.com/docs/en/models/fable-5-2/whats-new-fable-5-2?ref=journal.qualiteg.com">What&apos;s new in Claude Fable 5.2 (official docs)</a></td></tr><tr><td>Migration steps</td><td><a href="https://platform.claude.com/docs/en/models/fable-5-2/migration-guide?ref=journal.qualiteg.com">Claude Fable 5.2 and Claude Mythos 5.2 migration guide (official docs)</a></td></tr><tr><td>Prompting</td><td><a href="https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5-2?ref=journal.qualiteg.com">Prompting Claude Fable 5.2 (official docs)</a></td></tr><tr><td>Model list and knowledge cutoffs</td><td><a href="https://platform.claude.com/docs/en/models/overview?ref=journal.qualiteg.com">Models overview (official docs)</a></td></tr><tr><td>API pricing</td><td><a href="https://platform.claude.com/docs/en/about-claude/pricing?ref=journal.qualiteg.com">Pricing (official docs)</a></td></tr><tr><td>Effort</td><td><a href="https://platform.claude.com/docs/en/build-with-claude/effort?ref=journal.qualiteg.com">Effort (official docs)</a></td></tr><tr><td>History rewrite check</td><td><a href="https://platform.claude.com/docs/en/build-with-claude/preserved-thinking?ref=journal.qualiteg.com">Preserved thinking (official docs)</a></td></tr><tr><td>Rate limits</td><td><a href="https://platform.claude.com/docs/en/api/rate-limits?ref=journal.qualiteg.com">Rate limits (official docs)</a></td></tr><tr><td>Safety and evaluation details</td><td><a href="https://www.anthropic.com/document/claude-fable-5-2-mythos-5-2-system-card?ref=journal.qualiteg.com">Claude Fable 5.2 and Claude Mythos 5.2 System Card (Anthropic, PDF)</a></td></tr><tr><td>Claude Code model settings and effort</td><td><a href="https://code.claude.com/docs/en/model-config?ref=journal.qualiteg.com">Model configuration (Claude Code official docs)</a></td></tr><tr><td>Claude Code Fast mode</td><td><a href="https://code.claude.com/docs/en/fast-mode?ref=journal.qualiteg.com">Fast mode (Claude Code official docs)</a></td></tr><tr><td>Claude Code change history</td><td><a href="https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md?ref=journal.qualiteg.com">Claude Code CHANGELOG (GitHub)</a></td></tr><tr><td>How Fable is handled by plan</td><td><a href="https://support.claude.com/en/articles/15424964-claude-fable-models-on-your-plan?ref=journal.qualiteg.com">Claude Fable models on your plan (Claude Help Center)</a></td></tr><tr><td>Launch offers</td><td><a href="https://support.claude.com/en/articles/17506494-claude-fable-5-2-launch-offers?ref=journal.qualiteg.com">Claude Fable 5.2 launch offers (Claude Help Center)</a></td></tr><tr><td>Model switching in apps</td><td><a href="https://support.claude.com/en/articles/15363606-why-claude-switched-models-in-your-conversation-with-fable-models?ref=journal.qualiteg.com">Why Claude switched models in your conversation with Fable models (Claude Help Center)</a></td></tr><tr><td>Independent evaluation</td><td><a href="https://artificialanalysis.ai/models/claude-fable-5-2?ref=journal.qualiteg.com">Claude Fable 5.2 model page (Artificial Analysis)</a></td></tr><tr><td>GPT-6 Astra pricing</td><td><a href="https://developers.openai.com/api/docs/pricing?ref=journal.qualiteg.com">Pricing (OpenAI official docs)</a></td></tr><tr><td>Gemini 4 Argon pricing</td><td><a href="https://aistudio.google.com/docs/pricing?ref=journal.qualiteg.com">Gemini API pricing (Google official docs)</a></td></tr></tbody></table>
<!--kg-card-end: html-->
<hr><h2 id="related-links">Related Links</h2><ul><li><a href="https://bestllam.com/?ref=journal.qualiteg.com">Bestllam</a> Access to more than 30 LLMs</li><li><a href="https://journal.qualiteg.com/claude-opus-5-5-claude-code-guide/">Claude Opus 5.5 Complete Guide: Model Specifications, API Notes, and Claude Code Operations</a></li><li><a href="https://journal.qualiteg.com/claude-opus-5-claude-code-guide/">Claude Opus 5.0 Complete Guide: Model Specifications, API Notes, and Claude Code Operations</a></li><li><a href="https://journal.qualiteg.com/claude-fable-5-claude-code-guide/">The Complete Guide to Claude Fable 5: Model Specs and Claude Code Operations from the Official Docs</a></li><li><a href="https://journal.qualiteg.com/claude-fable-5-mythos-5-guide-benchmarks/">Finally Generally Available: A Practitioner&apos;s Read on Claude Mythos 5 / Fable 5</a></li><li><a href="https://journal.qualiteg.com/gpt-6-astra-features-pricing-guide/">What Is GPT-6 Astra? AGI, Pricing, and Claude Fable 5.1 Compared</a></li><li><a href="https://journal.qualiteg.com/llm-ranking-2026/">Japanese LLM Rankings 2026: Benchmark Analysis Report (October 22 Edition)</a></li></ul>]]></content:encoded></item><item><title><![CDATA[Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation. Have It Build Something That Runs, Auto-Grade It, and Doubt the Grader Itself]]></title><description><![CDATA[The grader's 10 points for tests were 0/0 for three whole days, invisible in the score table. How we built a rig that has the agent build a web system and grades it by running it, the five-kinds-times-three pass/fail criteria, running the control first, and contracts decided only by the tests.]]></description><link>https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/</link><guid isPermaLink="false">6ab38dfb2ead0f114b6f097c</guid><category><![CDATA[AI Agents]]></category><category><![CDATA[LLM]]></category><category><![CDATA[IT & AI Technology]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Sun, 08 Nov 2026 18:46:52 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/11/build-coding-agent-from-scratch-part6-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/11/build-coding-agent-from-scratch-part6-en.png" alt="Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation. Have It Build Something That Runs, Auto-Grade It, and Doubt the Grader Itself"><p>Hello!</p><p>This is from the period when we had built the evaluation rig for our own coding agent and were running it every day for about three weeks. We ran 17 runs of the authenticated task-management web app task, looked at the score table, and judged that &quot;they fall within the passing range.&quot;</p><p>One day, reading the breakdown of a single run, I found this line:</p><p><strong>&quot;Tests | 10 pts | 0 | 0/0&quot;</strong></p><p>In all 17 runs, the 10 points allotted to tests were 0/0.</p><p>A routing rule added on the day we split the grader had been sending test-related items to a different category. The unit tests had locked in that broken form as &quot;correct.&quot; For three days, nobody noticed.</p><p>To put the conclusion first, a coding agent&apos;s evaluation produces no numbers unless it takes the form of <strong>&quot;have it build something that runs, and grade it automatically&quot;</strong>. And unless you keep doubting the grader itself, the numbers lie. In this article we describe how we built our evaluation rig, three holes in the grader, and the pitfalls of bringing in tasks we did not write ourselves.</p><p>This article is Part 6 of the series &quot;Building a Coding Agent from Scratch.&quot;</p><p>The previous installment, <a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Part 5</a>, was about loading a local LLM onto two 16GB GPUs and having it build a web system on its own.</p><p>We ran the same task three times and got 94.9, 93.8, and 96.5. We could say flatly that &quot;this configuration works&quot; only because the yardstick that produces those scores was already in place.</p><p>This time, the story is about that yardstick. Every number presented so far in this series was produced by this evaluation rig.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Part</th><th>Theme</th></tr></thead><tbody><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Part 1</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Why &quot;finishing the job&quot; is hard. Separating the turn runner from the stop decider</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">Part 2</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">&quot;Done&quot; means &quot;it runs,&quot; not &quot;it&apos;s written.&quot; The completion gate and how to write a pushback</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">Part 3</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">&quot;It didn&apos;t stop&quot; is not &quot;it made progress.&quot; Count idle turns, and cap them without cutting the thinking</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Part 4</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Autonomy rules are for when nobody is around. If a human is present, stop and ask</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Part 5</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Making local LLMs first-class citizens. Getting two 16GB GPUs to build a web system</a></td></tr><tr><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">Part 6 (this post)</a></strong></td><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">How to build the evaluation. Have it build something that runs, auto-grade it, and doubt the grader itself</a></strong></td></tr><tr><td>Part 7</td><td>Logging and regression. Everything is found in the event log. And building with zero dependencies</td></tr></tbody></table>
<!--kg-card-end: html-->
<h2 id="1-the-task-takes-the-form-hand-over-a-specification-and-have-it-build-an-entire-web-system">1. The Task Takes the Form &quot;Hand Over a Specification and Have It Build an Entire Web System&quot;</h2><p>The shape of the evaluation is as follows. In this article, the agent itself is the harness, the part that grades the deliverable is the grader, and the whole bundle of tasks, grader, and execution procedure is the evaluation rig.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig06_1_eval_pipeline-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation. Have It Build Something That Runs, Auto-Grade It, and Doubt the Grader Itself" loading="lazy"><figcaption>Figure 1: Have it build something that runs, and grade it automatically (diagram: Qualiteg)</figcaption></figure><p>A task is a pair of a specification and a seed (empty, or partially written code). We hand the agent a one-line instruction and the specification and let it run unattended. When it stops, the grader actually starts the deliverable, hits the routes in the specification, checks whether data is persisted, whether pages are returned, and whether tests are written and pass, and produces a score with a breakdown.</p><p>The first task was a task-management web app with authentication. Twelve APIs, SQLite, a login page and a kanban page, tests. The point was to measure &quot;can it build a moderately complex web system.&quot; The allocation was startup 15, APIs 35, authorization 15, persistence 10, UI 10, tests 10, quality 5, for 100, with 70 as passing.</p><p>Why insist on &quot;something that runs&quot;? Because, as described in Part 2, the model says &quot;Done.&quot; Whether the tests passed is no evidence either, if the model wrote the tests itself. The grader starts the deliverable in a separate process and hits it from outside. There was no other yardstick we could trust.</p><p>One more thing: the grader knows the answer to the task, but <strong>the harness does not</strong>. The completion gate in Part 2 only checks &quot;do the routes written in the specification exist&quot; and &quot;are the status codes as specified,&quot; and never looks at the response contents. If the gate knows the answer, the evaluation becomes a fixed game. We held this line to the end.</p><h2 id="2-run-the-control-experiment-first-and-its-settled-in-seven-and-a-half-minutes">2. Run the Control Experiment First, and It&apos;s Settled in Seven and a Half Minutes</h2>
<!--kg-card-begin: html-->
<figure style="margin:24px auto 0;max-width:540px;"><img src="https://journal.qualiteg.com/content/images/2026/11/qc6_manga_6-2_v1_en.png" alt="Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation. Have It Build Something That Runs, Auto-Grade It, and Doubt the Grader Itself" style="width:100%;height:auto;border:1px solid #D5DDE6;border-radius:10px;display:block;"><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin:8px 0 0;line-height:1.6;">Our local model would not score. Was the harness broken, or was the model just not strong enough?</figcaption></figure><div style="max-width:540px;margin:14px auto 28px;border:1px solid #D5DDE6;background:#fff;border-radius:10px;padding:10px 14px;font-size:13px;"><ul style="margin:0;padding:0;"><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">Our local model can&apos;t reach 70 points. I&apos;ve fixed the harness in 16 places, but...</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Senior</span><span style="flex:1 1 auto;line-height:1.7;">Is the harness broken? Or is the model just too weak? Have you separated the two?</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">I ran a small cloud model once: 89.5 points in 7.5 minutes, zero interventions.</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Senior</span><span style="flex:1 1 auto;line-height:1.7;">Run the control first. Separate harness holes from model strength, then fix.</span></li></ul></div>
<!--kg-card-end: html-->
<p>There is something we should have done right after building the evaluation: the control experiment.</p><p>Our evaluation plan said from the start, &quot;confirm a high score and zero interventions with a strong cloud model.&quot; But we had not run it. Unable to tell whether the local models&apos; low scores were due to &quot;the harness is broken&quot; or &quot;the model is not enough,&quot; we had piled up 16 harness fixes.</p><p>One run with a small cloud model <strong>settled it in seven and a half minutes</strong>. 33 turns, 89.5 points, zero interventions, stop reason &quot;complete.&quot; The server started and data was persisted. Since it ran to completion with zero interventions well above the passing score of 70, that was enough for the separation.</p><p>This changed the meaning of the 16 fixes. They were not &quot;fixing what was broken&quot; but fixes that &quot;let local models get into the ring.&quot; Both kinds are needed, but they mean different things.</p><p><strong>Run the control experiment that separates &quot;is the harness broken&quot; from &quot;is the model not enough&quot; first.</strong> We put it off and proceeded for a week with the meaning of our fixes misread.</p><h2 id="3-add-more-kinds-of-tasks-and-decide-the-passfail-criteria-first">3. Add More Kinds of Tasks, and Decide the Pass/Fail Criteria First</h2><p>When you measure with a single task, you want to say &quot;this model is usable / unusable&quot; with a single score. We learned how dangerous that is from the Coder variant.</p><p>The same model scored 0 (disqualified) on building a web app from scratch, 24 on a server with background processing, 70 on building a CLI from scratch, 92.5 on fixing a bug in existing code, and 96.7 on adding a feature to existing code. <strong>It splits into four tiers by task shape.</strong> Even a weak model is practical for work where the structure already exists.</p><p>So we made it five kinds of tasks: web from scratch, bug fix, CLI from scratch, feature addition to an existing API, and a server with background processing. Each is graded by hidden tests or a dedicated grader.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig06_2_five_tasks_v2-en.png" class="kg-image" alt="Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation. Have It Build Something That Runs, Auto-Grade It, and Doubt the Grader Itself" loading="lazy"><figcaption>Figure 2: Five kinds of tasks x 3 runs. All 15 stopped having met the completion criteria (diagram: Qualiteg)</figcaption></figure><p>And we decided the harness&apos;s pass/fail criteria in advance, to put an end to &quot;let&apos;s run it a bit more.&quot;</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Metric</th><th>Criterion</th></tr></thead><tbody><tr><td>1</td><td>Never call a deliverable that doesn&apos;t start &quot;completed&quot;</td></tr><tr><td>2</td><td>With a decent model, zero stops due to idling</td></tr><tr><td>3</td><td>Harness-side time (verification and gates) at or below 20% of wall time</td></tr><tr><td>4</td><td>Run the same task three times, and all three scores within &#xB1;5 of the average</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The score is the model&apos;s grade. The harness&apos;s pass/fail is judged by separate metrics. Running three times is to &quot;separate variation in path from variation in result,&quot; as described in Part 3.</p><p>In the formal judgment on the latest harness, all 15 runs across five kinds x 3 stopped as completed, all six that start a server passed the startup check, and harness-side time peaked at 12%. Only one task exceeded &#xB1;5 from the average: the first run of building a CLI from scratch (87.1). Reading the details, the model had dropped the handling of excluded words and the output format of <code>--json</code>, which is not the kind of lost points a harness pushback can catch.</p><p>For each metric, you can separate whether the cause of not meeting it is the harness or the model. That is the payoff of deciding the criteria first.</p><h2 id="4-contamination-of-the-evaluation-environment-breaks-autonomy-without-showing-in-the-score">4. Contamination of the Evaluation Environment Breaks Autonomy Without Showing in the Score</h2><p>Entering the third week of evaluation, we made an embarrassing discovery.</p><p>Taking inventory of leftover processes on the work machine, we found three servers started by evaluation runs two days earlier. Furthermore, port 3000, the very port the task specification names as &quot;the default port,&quot; <strong>had been held for two weeks by a resident app unrelated to the evaluation</strong>. In other words, every run in the first two weeks had been running in an environment where starting the server as specified would always produce a port conflict.</p><p>Counting conflict errors in the logs of the past 10 runs, one run had 191 lines, with its own tests failing 127 times on port conflicts. The model had concluded &quot;a port conflict is not an implementation defect&quot; and moved on.</p><p>Because the grader uses a different port, <strong>this did not show directly in the score</strong>. What was broken was the step where &quot;the model runs its own deliverable to check it,&quot; which is the core of autonomy. A model that reads the conflict as &quot;not my fault&quot; gives up on verification.</p><p>The fix: for each run, obtain a free port from the OS and inject it into the model&apos;s execution environment as <code>PORT</code>. The grader&apos;s port also stops being fixed and is taken from the free pool. At the end of a run, stop the process holding that port. Other people&apos;s ports are never touched.</p><p><strong>Check the assumptions of the evaluation environment every time, before running.</strong> &quot;The port is free&quot; may be true when written and false two days later. Because we did not have the aggregation angle (counting conflict errors), we did not notice despite having 191 lines of log. This story leads into the event log in Part 7.</p><h2 id="5-doubt-the-grader-itself">5. Doubt the Grader Itself</h2>
<!--kg-card-begin: html-->
<figure style="margin:24px auto 0;max-width:540px;"><img src="https://journal.qualiteg.com/content/images/2026/11/qc6_manga_6-5_v1_en.png" alt="Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation. Have It Build Something That Runs, Auto-Grade It, and Doubt the Grader Itself" style="width:100%;height:auto;border:1px solid #D5DDE6;border-radius:10px;display:block;"><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin:8px 0 0;line-height:1.6;">All 17 scores were within the passing range. In the breakdown table, &quot;Tests 0/0&quot; had been sitting there for three days</figcaption></figure><div style="max-width:540px;margin:14px auto 28px;border:1px solid #D5DDE6;background:#fff;border-radius:10px;padding:10px 14px;font-size:13px;"><ul style="margin:0;padding:0;"><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">All 17 runs are within the passing range. So this setup works, right?</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Senior</span><span style="flex:1 1 auto;line-height:1.7;">In the breakdown, why does the Tests row show 0/0?</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">...The grader was sorting the test items into quality. Ever since the split 3 days ago.</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Senior</span><span style="flex:1 1 auto;line-height:1.7;">Passing range or not, a broken yardstick lies. Doubt the grader, too.</span></li></ul></div>
<!--kg-card-end: html-->
<p>Back to the &quot;tests 0/0&quot; from the opening. Two more holes turned up in the grader.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig06_3_grader_hole-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation. Have It Build Something That Runs, Auto-Grade It, and Doubt the Grader Itself" loading="lazy"><figcaption>Figure 3: Doubt the grader itself (diagram: Qualiteg)</figcaption></figure><h3 id="5-1-make-a-00-in-the-breakdown-a-warning">5-1. Make a 0/0 in the breakdown a warning</h3><p>The cause of &quot;tests 0/0&quot; was a rule added when we split the grader: &quot;if the prefix is a category name, route to that category.&quot; The three test-related items started with &quot;quality,&quot; as in <code>quality: tests are written</code>, so they were routed to quality, and the rule written after it, &quot;route the test items within quality to tests,&quot; never fired again.</p><p>The impact: +10 points if three tests pass. The quality denominator drops from 9 to 6. Regrading, all 17 runs went up by about 9.8 points, and the pass/fail judgment (&#xB1;5 from the average) did not move. Only the absolute values had looked 10 points too low. Set alongside the cloud control model, that difference had exactly disappeared from view.</p><p>We fixed three things. If the breakdown has a 0/0 item, emit a warning. Provide an entry point to regrade the deliverables of past runs with the current grader (no need to rerun). And since the unit tests had locked in &quot;no items go into tests&quot; as the correct form, we fixed the tests themselves.</p><p><strong>&quot;The unit tests pass&quot; is no evidence that &quot;the yardstick is correct.&quot;</strong> Tests sometimes lock in a defect as correct. This pattern comes up once more in Part 7.</p><h3 id="5-2-a-00-does-not-necessarily-mean-wrote-nothing">5-2. A 0/0 does not necessarily mean &quot;wrote nothing&quot;</h3><p>When we ran tasks we did not write ourselves (described below), three of six problems were 0/0. Thinking the model had written nothing, we looked at the contents. It had written them.</p><p>The cause was on the grader side. The deliverable for problem 1 looped forever, and since the grader ran all six problems in a single process, it aborted the whole thing at 180 seconds and lost the results of the three problems that had not yet run. We changed it to run each file in a separate process and record a hung file as a failure of that problem.</p><p><strong>Doubt a 0/0 from the grader until a reason emerges.</strong></p><h3 id="5-3-the-audit-angles-are-also-wrong-depending-on-what-the-seed-is">5-3. The audit angles are also wrong depending on what the seed is</h3><p>The evaluation run aggregation has an angle that counts &quot;markers of environment-caused failure&quot; (port conflicts, package fetches, path mismatches, and so on). When we ran the task of adding a feature to an existing 30,000-line codebase, all four runs showed 3 to 19 markers, and we suspected the evaluation environment.</p><p>In fact, the test names in the seed codebase contained phrases like &quot;in the EADDRINUSE advice...&quot;, and we were merely counting the test names listed in the verification output as markers. We fixed it to exclude test-report lines before counting. A yardstick goes wrong when what it measures changes.</p><h2 id="6-for-tasks-we-didnt-write-first-flush-out-the-contracts-decided-only-by-the-tests">6. For Tasks We Didn&apos;t Write, First Flush Out the &quot;Contracts Decided Only by the Tests&quot;</h2><p>Measuring only with tasks we wrote ourselves leaves the suspicion that the tasks are conveniently shaped for our own harness. So we looked for publicly available tasks.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Candidate</th><th>Verdict</th><th>Reason</th></tr></thead><tbody><tr><td>SWE-bench family</td><td>Rejected</td><td>Requires Docker containers, incompatible with running as-is on Windows with zero dependencies</td></tr><tr><td>Vibe Code Bench</td><td>Rejected</td><td>0 of 100 specifications are public</td></tr><tr><td>ViBench</td><td>Rejected</td><td>Requires Docker, Playwright, and cloud LLM keys</td></tr><tr><td>VibeCodingBench</td><td>Rejected</td><td>License is All rights reserved</td></tr><tr><td>WebGen-Bench</td><td>Rejected</td><td>Grading uses a browser agent and a vision model. Cannot be converted to hidden tests</td></tr><tr><td>Web-Bench</td><td>On hold</td><td>Good task shape, but the tests are Playwright. Needs porting</td></tr><tr><td><strong>Exercism (the 49 JS problems from Aider polyglot, MIT)</strong></td><td><strong>Adopted</strong></td><td>Problem statements, skeletons, tests, and reference implementations all present. The tests use a narrow API surface and run unmodified on a zero-dependency compatibility layer</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The key was &quot;decide the conditions first, then search.&quot; Our conditions: zero dependencies, runs on Windows, convertible to the hidden-test form, and a permissive license. These four eliminate most public benchmarks.</p><p>Before using the imported tasks, we confirmed that <strong>the seed fails and the reference implementation scores full marks</strong>. On both six-problem sets, the seed scored 15 (quality only) and the reference implementation 100. Skip this, and you cannot distinguish holes in the grader from holes in the task.</p><h3 id="6-1-its-the-tests-not-the-problem-statement-that-decide">6-1. It&apos;s the tests, not the problem statement, that decide</h3><p>In a preliminary run, one problem scored 0/95. Only the tests decided that the return type was <code>Date</code>, and the model was returning the date as a number. Another problem failed 15 of 49 on differences in error messages (it had used <code>Stack empty</code> instead of <code>.stack too small</code>).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig06_4_contract-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation. Have It Build Something That Runs, Auto-Grade It, and Doubt the Grader Itself" loading="lazy"><figcaption>Figure 4: First flush out the contracts decided only by the tests (diagram: Qualiteg)</figcaption></figure><p><strong>These are contracts that are not in the problem statement and are decided only by the tests</strong>. The original benchmark shows the test output on the second attempt, but our harness is a single attempt, so we decided to attach the contracts to the specification. What we attach is the error messages the tests match against and the verbatim text of the first external test. Just one, to show the export name, argument shape, and return type; passing that one alone earns no points. We do not hand over the set of answers.</p><p>This does not contradict &quot;do not write the answer&quot; from Part 2. Wording and argument shapes are &quot;how to call it,&quot; not &quot;the answer,&quot; the same kind of thing as writing status codes in a specification. And the effect is different from that of an answer.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Condition</th><th>Six-problem set A</th><th>Six-problem set B</th></tr></thead><tbody><tr><td>No contracts (Coder variant)</td><td>49.7</td><td>56.3</td></tr><tr><td>With error messages attached</td><td>57</td><td>61.9</td></tr><tr><td>With a calling example attached too</td><td>60.8</td><td>Same as above</td></tr><tr><td>Main candidate 35B (with contracts, 2 runs)</td><td>88.7 / 93.8</td><td>78.5 / 86</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Only the problems whose calling convention was missing from the statement went up. The 0/95 problem became 95/95, and problems the model did not know how to solve stayed as they were. <strong>A task with the contracts attached lets the difference between models show directly in the score.</strong></p><p>The same pattern appeared in adding a feature to the 30,000-line existing codebase. The main 35B scored 77.1, 75.3, and 77.1. All three ran to completion without breaking existing tests, but the 20% they lost was entirely the part that &quot;copies the specification&apos;s wording without a single character&apos;s difference.&quot; The default value of <code>--format</code>, message wording, the order of Markdown code blocks. The form of contracts decided only by the tests.</p><h2 id="7-count-in-the-record-before-adding-a-countermeasure">7. Count in the Record Before Adding a Countermeasure</h2><p>While running evaluations, you think of many countermeasures that &quot;should work if fixed this way.&quot; Partway through, we decided to <strong>count occurrences in the record before adding one</strong>.</p><p>For the verification-cost countermeasure in Part 3, the first candidate was &quot;if the model itself ran the tests in the same turn, reuse that result.&quot; Plausible. But counting in the record, that pattern occurred once in eight runs. We did not add it. A countermeasure that does not work only adds one more gate.</p><p>Conversely, we check in the next run whether a countermeasure we added is working. The output-window extension was not working in its first implementation. We only found out by looking at the requested output window against the number of tokens that actually came out uncut (10,175).</p><p>And a countermeasure creates the hole for the next one. The countermeasure &quot;when a server listener is detected, leave it and return&quot; also acted on tests that start a server, and the test was misidentified as &quot;left in the background.&quot; Add one gate, and in the next run confirm what that gate changed. This is why we decided to run five kinds x 1 every time we touch the harness. Two to three hours per round, and reading five breakdowns alone surfaced the next hole.</p><h2 id="summary-keep-doubting-the-yardstick-or-the-numbers-lie">Summary: Keep Doubting the Yardstick, or the Numbers Lie</h2>
<!--kg-card-begin: html-->
<table><thead><tr><th>What to do</th><th>Why</th></tr></thead><tbody><tr><td>Have it build something that runs, and have the grader start and hit it in a separate process</td><td>Neither the model&apos;s &quot;done&quot; nor its own tests are evidence</td></tr><tr><td>Run the control experiment first</td><td>Separates &quot;is the harness broken&quot; from &quot;is the model not enough&quot; in seven and a half minutes</td></tr><tr><td>Add more kinds of tasks</td><td>The same model: Web 0 / background processing 24 / CLI 70 / fixes and additions 92 to 97</td></tr><tr><td>Decide the pass/fail criteria first (4 metrics, 3 runs)</td><td>Ends &quot;let&apos;s run it a bit more&quot; and separates causes into harness or model</td></tr><tr><td>Check the evaluation environment&apos;s assumptions every time</td><td>Port conflicts don&apos;t show in the score and only break autonomy</td></tr><tr><td>Make a 0/0 in the breakdown a warning, and keep a way to regrade</td><td>&quot;Tests 0/0&quot; for 17 runs straight is invisible in the score range</td></tr><tr><td>For tasks you didn&apos;t write, confirm the seed fails and the reference scores full marks, and flush out the contracts</td><td>Wording and types are decided only by the tests</td></tr><tr><td>Count before adding a countermeasure, and check its effect in the next run</td><td>Adding a gate for a pattern seen once in eight runs only adds a gate</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Next time is the final part.</p><p>All 105 defects described in this series were found from a single append-only event log.</p><p>We will cover the design of the record, doubting the &quot;passing&quot; of regression tests, holes visible only on one OS, and why we build with zero dependencies.</p><p>See you next time!</p><h2 id="sources-and-references">Sources and References</h2><ul><li><a href="https://www.swebench.com/?ref=journal.qualiteg.com">SWE-bench (official site)</a></li><li><a href="https://arxiv.org/abs/2603.04601?ref=journal.qualiteg.com">Vibe Code Bench (paper, arXiv)</a></li><li><a href="https://huggingface.co/datasets/luzimu/WebGen-Bench?ref=journal.qualiteg.com">WebGen-Bench (dataset on Hugging Face)</a></li><li><a href="https://github.com/bytedance/web-bench?ref=journal.qualiteg.com">Web-Bench (ByteDance, GitHub)</a></li><li><a href="https://github.com/Aider-AI/polyglot-benchmark?ref=journal.qualiteg.com">Aider polyglot benchmark (Exercism problem set, GitHub)</a></li></ul><h2 id="related-articles">Related Articles</h2><ul><li><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">Building a Coding Agent from Scratch, Part 2: &quot;Done&quot; Means &quot;It Runs,&quot; Not &quot;It&apos;s Written&quot;</a></li><li><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Building a Coding Agent from Scratch, Part 5: Making Local LLMs First-Class Citizens</a></li><li><a href="https://journal.qualiteg.com/coding-agents-2026-part3/">The State and Future of Coding Agents, Part 3: From AI That Writes to AI You Command</a></li></ul>]]></content:encoded></item><item><title><![CDATA[Understanding Active Directory: The Technology Behind Enterprise AI Security - Part 7: Security and Best Practices]]></title><description><![CDATA[<p>Hello! In this seventh installment of the series, we cover Security and Best Practices - considerations for production environments.</p><h2 id="series-overview">Series overview</h2><ul><li><a href="https://journal.qualiteg.com/active-directory-study-for-ai-security-part01/" rel="noreferrer"><strong>Chapter 1: Understanding the basics - Fundamentals of Active Directory and Kerberos/NTLM authentication</strong></a></li><li><a href="https://journal.qualiteg.com/active-directory-study-for-ai-security-part02/" rel="noreferrer"><strong>Chapter 2: Building a domain environment - Setting up a lab environment</strong></a></li><li><a href="https://journal.qualiteg.com/active-directory-study-for-ai-security-part03/" rel="noreferrer"><strong>Chapter 3: Joining</strong></a></li></ul>]]></description><link>https://journal.qualiteg.com/active-directory-study-for-ai-security-part07/</link><guid isPermaLink="false">6aeac6ea0a0f0d13c338b6f5</guid><category><![CDATA[LLM Security]]></category><category><![CDATA[IT & AI Technology]]></category><dc:creator><![CDATA[Qualiteg Consulting]]></dc:creator><pubDate>Wed, 04 Nov 2026 03:46:52 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/11/AD_part07-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/11/AD_part07-en.png" alt="Understanding Active Directory: The Technology Behind Enterprise AI Security - Part 7: Security and Best Practices"><p>Hello! In this seventh installment of the series, we cover Security and Best Practices - considerations for production environments.</p><h2 id="series-overview">Series overview</h2><ul><li><a href="https://journal.qualiteg.com/active-directory-study-for-ai-security-part01/" rel="noreferrer"><strong>Chapter 1: Understanding the basics - Fundamentals of Active Directory and Kerberos/NTLM authentication</strong></a></li><li><a href="https://journal.qualiteg.com/active-directory-study-for-ai-security-part02/" rel="noreferrer"><strong>Chapter 2: Building a domain environment - Setting up a lab environment</strong></a></li><li><a href="https://journal.qualiteg.com/active-directory-study-for-ai-security-part03/" rel="noreferrer"><strong>Chapter 3: Joining clients and servers to the domain - Domain join in detail</strong></a></li><li><a href="https://journal.qualiteg.com/active-directory-study-for-ai-security-part04/" rel="noreferrer"><strong>Chapter 4: Proxy servers and Integrated Windows Authentication</strong></a></li><li><a href="https://journal.qualiteg.com/active-directory-study-for-ai-security-part05/" rel="noreferrer"><strong>Chapter 5: Browser configuration and authentication - Settings for each browser</strong></a></li><li><a href="https://journal.qualiteg.com/active-directory-study-for-ai-security-part06/" rel="noreferrer"><strong>Chapter 6: Troubleshooting - Common problems and solutions</strong></a></li><li><strong>[&#x2605;This installment&#x2605;] Chapter 7: Security and best practices - Considerations for production environments</strong></li><li>Chapter 8: Practical configuration examples - Integration with AI security tools</li></ul><hr><h1 id="chapter-7-security-and-best-practices">Chapter 7: Security and Best Practices</h1><p>Part 7 is about bringing the authentication infrastructure built in the previous installments to a state where it can run in production.</p><p>In the first half, 7.1, we harden the entry points: how to handle Basic authentication, which is practically plaintext; how to bring the password policy in line with current recommendations; and how far to narrow administrative privileges. In the second half, 7.2, we build the mechanisms for noticing that someone has gotten in: which logs to collect, how to spot anomalies, and how to check things regularly.</p><p>Each section opens with a four-panel manga. The six characters are an auditor, an IT manager, a CISO, a developer, a help desk member, and a trainer, and all the scenes are fictional. All numbers and recommendations in the text are based on official Microsoft documentation, RFCs, and NIST publications, and the sources are listed at the end of each section.</p><h2 id="71-managing-credentials">7.1 Managing Credentials</h2><h3 id="711-security-risks-of-basic-authentication">7.1.1 Security Risks of Basic Authentication</h3>
<!--kg-card-begin: html-->
<figure style="margin:24px auto 0;max-width:540px;"><img src="https://journal.qualiteg.com/content/images/2026/11/ad_part07_manga_7-1-1_en.png" alt="Understanding Active Directory: The Technology Behind Enterprise AI Security - Part 7: Security and Best Practices" style="width:100%;height:auto;border:1px solid #D5DDE6;border-radius:10px;display:block;"><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin:8px 0 0;line-height:1.6;">Looking inside Basic authentication at a security training. Nothing is sealed, so decoding the header shows everything as is</figcaption></figure><div style="max-width:540px;margin:14px auto 28px;border:1px solid #D5DDE6;background:#fff;border-radius:10px;padding:10px 14px;font-size:13px;"><ul style="margin:0;padding:0;"><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Trainer</span><span style="flex:1 1 auto;line-height:1.7;">Let&apos;s look inside Basic authentication</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Trainer</span><span style="flex:1 1 auto;line-height:1.7;">Username and password are just Base64-encoded, not encrypted</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Help Desk</span><span style="flex:1 1 auto;line-height:1.7;">I decoded it and could read the password as is!</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Trainer</span><span style="flex:1 1 auto;line-height:1.7;">That&apos;s why Basic auth is used only over a TLS-encrypted path</span></li></ul></div>
<!--kg-card-end: html-->
<p>Basic authentication has the simplest mechanism of all, and without TLS it sends the username and password in a form anyone can reverse.</p><p>The reason is that it does not encrypt the credentials. RFC 7617 specifies that the client sends the username and password joined by a colon and encoded in Base64, and it also states that the scheme is not considered secure unless used together with an external secure system such as TLS. Base64 is an encoding that anyone can reverse, not encryption.</p>
<!--kg-card-begin: html-->
<div style="border-left:4px solid #1184DE;background:#F7FAFD;border-radius:0 10px 10px 0;padding:16px 20px;margin:24px 0;"><div style="margin:0 0 8px;font-size:15px;letter-spacing:.04em;color:#0B5C9C;font-weight:700;">What the RFCs say about Basic authentication</div><ul style="margin:0;padding-left:1.2em;"><li style="margin:4px 0;">The credentials are <code>user-id:password</code> encoded in Base64. This is the same as sending them in cleartext over the network</li><li style="margin:4px 0;">It is not a secure method of user authentication unless protected by TLS or similar</li><li style="margin:4px 0;">It SHOULD NOT be used without enhancements such as HTTPS</li><li style="margin:4px 0;">A proxy can require the same scheme. It responds with 407, and the client answers with the <code>Proxy-Authorization</code> header</li></ul><div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://www.rfc-editor.org/rfc/rfc7617?ref=journal.qualiteg.com" style="color:#0B5C9C;">RFC 7617 The Basic HTTP Authentication Scheme</a>, <a href="https://www.rfc-editor.org/rfc/rfc9110?ref=journal.qualiteg.com" style="color:#0B5C9C;">RFC 9110 HTTP Semantics</a></div></div>
<!--kg-card-end: html-->
<h4 id="where-the-credentials-get-stolen">Where the credentials get stolen</h4><p>In the setup used throughout this series, traffic goes from an office PC through the proxy to external generative AI services. The place where credentials get stolen is not between the proxy and the external site, but between the PC and the proxy.</p>
<!--kg-card-begin: html-->
<figure style="margin:24px 0;"><svg viewbox="0 0 720 260" width="100%" xmlns="http://www.w3.org/2000/svg" font-family="Inter, &apos;Segoe UI&apos;, sans-serif">
  <defs><marker id="arw" markerwidth="10" markerheight="10" refx="8" refy="5" orient="auto"><path d="M0,0 L10,5 L0,10 z" fill="#15222B"/></marker></defs>
  <rect x="20" y="70" width="150" height="80" rx="10" fill="#F7FAFD" stroke="#D5DDE6"/>
  <text x="95" y="104" text-anchor="middle" fill="#15222B" font-size="15" font-weight="700">Office PC</text>
  <text x="95" y="128" text-anchor="middle" fill="#4A463F" font-size="12">Browser</text>
  <rect x="285" y="70" width="150" height="80" rx="10" fill="#0B5C9C"/>
  <text x="360" y="104" text-anchor="middle" fill="#fff" font-size="15" font-weight="700">Proxy</text>
  <text x="360" y="128" text-anchor="middle" fill="#fff" font-size="12">Auth and logging</text>
  <rect x="550" y="70" width="150" height="80" rx="10" fill="#F7FAFD" stroke="#D5DDE6"/>
  <text x="625" y="104" text-anchor="middle" fill="#15222B" font-size="15" font-weight="700">External site</text>
  <text x="625" y="128" text-anchor="middle" fill="#4A463F" font-size="12">Generative AI, etc.</text>
  <line x1="172" y1="110" x2="280" y2="110" stroke="#15222B" stroke-width="2" marker-end="url(#arw)"/>
  <line x1="437" y1="110" x2="545" y2="110" stroke="#15222B" stroke-width="2" marker-end="url(#arw)"/>
  <text x="226" y="60" text-anchor="middle" fill="#284C6E" font-size="13" font-weight="700">Proxy-Authorization: Basic &#x2026;</text>
  <text x="226" y="170" text-anchor="middle" fill="#284C6E" font-size="13">Travels here as plain Base64</text>
  <text x="226" y="190" text-anchor="middle" fill="#284C6E" font-size="13">(effectively plaintext without TLS to the proxy)</text>
  <text x="491" y="60" text-anchor="middle" fill="#4A463F" font-size="13" font-weight="700">HTTPS (TLS)</text>
  <text x="491" y="170" text-anchor="middle" fill="#4A463F" font-size="13">Encrypted to the external site</text>
  <text x="360" y="235" text-anchor="middle" fill="#15222B" font-size="13">Even if the external site uses HTTPS, credentials for the proxy cross a separate segment</text>
</svg><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin-top:6px;">Figure: The segment that carries proxy authentication credentials. Even if the external site uses HTTPS, the credentials for the proxy cross a separate segment</figcaption></figure>
<!--kg-card-end: html-->
<p>If the external site uses HTTPS, the content is encrypted. But authentication to the proxy happens in the <code>CONNECT</code> request, before the HTTPS tunnel is built. RFC 9110 says proxy authentication is used to check permission to establish a tunnel, and its <code>CONNECT</code> example includes <code>Proxy-Authorization</code>. The client may send the credentials after receiving the proxy&apos;s 407 response, or attach them from the very first request. Either way, if the segment between the PC and the proxy is not protected by TLS, anyone who can observe that segment sees the credentials as is.</p>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Where it can be observed</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">What happens</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">How to protect it</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Same network</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">On public Wi-Fi, or when an attacker intercepts the path with ARP spoofing, traffic between the PC and the proxy can be read</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Use TLS for the connection to the proxy itself. From outside the office, protect the path with a VPN and still use TLS for the proxy connection inside it (a VPN protects the path; it does not replace encryption of the proxy connection)</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Proxy logs</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">If the proxy is set to log request headers as is, the Base64 credentials end up in the logs</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Do not write the header to logs. Restrict who can read the logs</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Fake proxy</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">If users are lured to a &quot;free proxy&quot; or a fake auto-configuration (PAC) file, they hand their credentials to the attacker</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Distribute proxy settings through Group Policy and do not let users change them</td></tr></tbody></table>
<!--kg-card-end: html-->
<h4 id="why-basic-authentication-remains-and-how-to-keep-it-safely">Why Basic authentication remains, and how to keep it safely</h4><p>As we saw in Part 4, when the proxy offers Negotiate, NTLM, and Basic, which one is used depends on the client&apos;s implementation and settings. Within Negotiate, Kerberos is preferred, and NTLM is chosen when Kerberos is not available. Falling back to Basic is a separate mechanism: it depends on whether the proxy offers Basic and whether the client is configured to accept it. Basic is often kept as a fallback for devices not joined to the domain and for software that does not support integrated authentication.</p>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">If you keep Basic authentication.</b> When you keep it as a fallback, put three things in place together. Use TLS for the segment to the proxy, limit which clients may use Basic by IP address or device, and distinguish connections authenticated with Basic in the logs so you can monitor them. If the fallback is wide open, an environment you thought was protected by Kerberos gets a plaintext back door.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/windows/win32/secauthn/microsoft-negotiate?ref=journal.qualiteg.com" style="color:#0B5C9C;">Microsoft Negotiate</a></div></div>
<!--kg-card-end: html-->
<h3 id="712-password-management-best-practices">7.1.2 Password Management Best Practices</h3>
<!--kg-card-begin: html-->
<figure style="margin:24px auto 0;max-width:540px;"><img src="https://journal.qualiteg.com/content/images/2026/11/ad_part07_manga_7-1-2_en.png" alt="Understanding Active Directory: The Technology Behind Enterprise AI Security - Part 7: Security and Best Practices" style="width:100%;height:auto;border:1px solid #D5DDE6;border-radius:10px;display:block;"><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin:8px 0 0;line-height:1.6;">Revising the password policy. Go for length, stop frequent changes, and ban common passwords</figcaption></figure><div style="max-width:540px;margin:14px auto 28px;border:1px solid #D5DDE6;background:#fff;border-radius:10px;padding:10px 14px;font-size:13px;"><ul style="margin:0;padding:0;"><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">CISO</span><span style="flex:1 1 auto;line-height:1.7;">New policy: long passwords instead of short, complex ones</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Help Desk</span><span style="flex:1 1 auto;line-height:1.7;">Do we still change them regularly? Every 3 months is tough...</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">CISO</span><span style="flex:1 1 auto;line-height:1.7;">No more scheduled changes. Instead, we&apos;ll block leaked passwords</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">What about service accounts?</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">CISO</span><span style="flex:1 1 auto;line-height:1.7;">Use gMSA. Windows manages the passwords</span></li></ul></div>
<!--kg-card-end: html-->
<p>Over the past few years, password policy has shifted from &quot;make it complex and change it often&quot; to &quot;make it long, do not force changes, and ban the ones that have leaked.&quot;</p><p>NIST and Microsoft point in the same direction. The latest version of NIST SP 800-63B requires passwords used as a single factor to be at least 15 characters, does not impose composition rules such as mixing character types, does not require periodic changes, and requires checking against lists of known compromised passwords. For Microsoft 365 accounts, Microsoft recommends a minimum of 14 characters and recommends not setting an expiration for cloud-only accounts. These are recommendations for Microsoft 365, not rules that apply directly to on-premises Active Directory, but the direction is the same.</p>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Item</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Old common practice</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">NIST SP 800-63B</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Microsoft 365 recommendation</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Length</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#8B96A4;">8 characters</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">15 or more as a single factor. 8 or more as part of multi-factor</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">14 or more characters</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Mixed character types</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#8B96A4;">Force uppercase, lowercase, digits, and symbols</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#0B5C9C;font-weight:600;">No composition rules (SHALL NOT)</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Explains that length and symbol requirements push passwords into predictable patterns</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Periodic changes</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#8B96A4;">Change every 90 days</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">Not required. Force a change only when there is evidence of compromise</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">No expiration recommended for cloud-only accounts</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Blocklist</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#8B96A4;">None</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#0B5C9C;font-weight:600;">Reject passwords found in lists of compromised or commonly used passwords</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#0B5C9C;font-weight:600;">Ban easy-to-guess passwords</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Maximum length</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#8B96A4;">Short upper limit</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">Accept at least 64 characters</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;"></td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Multi-factor</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#8B96A4;">Optional</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">States explicitly that passwords are not phishing-resistant</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#0B5C9C;font-weight:600;">Make multi-factor authentication registration mandatory</td></tr></tbody></table><div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/microsoft-365/admin/misc/password-policy-recommendations?ref=journal.qualiteg.com" style="color:#0B5C9C;">Password policy recommendations (Microsoft 365)</a>, <a href="https://pages.nist.gov/800-63-4/sp800-63b.html?ref=journal.qualiteg.com" style="color:#0B5C9C;">NIST SP 800-63B (Digital Identity Guidelines: Authentication)</a></div>
<!--kg-card-end: html-->
<p>Why the change? The more constraints people face, the more they create passwords that follow the same pattern. Microsoft explains that requirements for length, symbols, and changes tend to push passwords into predictable patterns. Attackers know the &quot;one uppercase letter, one symbol, digits at the end&quot; pattern well.</p><h4 id="what-you-can-set-in-the-active-directory-password-policy">What you can set in the Active Directory password policy</h4><p>The domain password policy lives at the following location in Group Policy. The defaults are values that have been used for a long time, not today&apos;s recommended values.</p><pre><code>Computer Configuration
&#x2514; Windows Settings
  &#x2514; Security Settings
    &#x2514; Account Policies
      &#x2514; Password Policy</code></pre>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Setting</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Default domain policy</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Direction of review</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;"><b>Enforce password history</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">24 passwords</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Keep as is. Prevents reuse</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;"><b>Maximum password age</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">42 days</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Move toward ending periodic changes. Before you stop them, first set up the practice of forcing a reset when a compromise is detected</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;"><b>Minimum password age</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">1 day</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Keep as is. Prevents cycling through the history at once to get back to the old password</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;"><b>Minimum password length</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">7 characters</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Set your organization&apos;s target and raise it. NIST&apos;s 15 characters (single factor) and Microsoft 365&apos;s 14 characters are good guides. Microsoft&apos;s best practice also recommends 8 or more</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;"><b>Password must meet complexity requirements</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Enabled</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Enabled by default (at least 3 character types, and must not contain the account name or display name). Since NIST does not impose composition rules, consider a transition to disabling it once you have enough length and a blocklist in place</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;"><b>Store passwords using reversible encryption</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Disabled</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Keep it disabled. Enabling it stores passwords in what is effectively plaintext</td></tr></tbody></table><div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/security-policy-settings/password-policy?ref=journal.qualiteg.com" style="color:#0B5C9C;">Password Policy</a>, <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/security-policy-settings/password-must-meet-complexity-requirements?ref=journal.qualiteg.com" style="color:#0B5C9C;">Password must meet complexity requirements</a></div>
<!--kg-card-end: html-->
<p>This table covers the default domain policy. Since Windows Server 2008, fine-grained password policies (PSOs) let you apply a different policy to specific groups or users, so you can, for example, require longer passwords only for administrators and service accounts. The audit in 7.2.3 checks PSOs as well.</p>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">Where the built-in policy falls short.</b> The domain password policy has no check against lists of leaked passwords. To do this in on-premises Active Directory, you need an on-premises deployment of Microsoft Entra Password Protection (which uses Entra ID&apos;s global banned password list and a custom list to reject password changes on the DCs), or an equivalent product. It helps the design to remember that &quot;go for length&quot; and &quot;do not force changes&quot; can be done with built-in features, while &quot;ban what has leaked&quot; needs an additional mechanism.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/entra/identity/authentication/concept-password-ban-bad-on-premises?ref=journal.qualiteg.com" style="color:#0B5C9C;">Microsoft Entra Password Protection for on-premises</a></div></div>
<!--kg-card-end: html-->
<h4 id="service-accounts-should-not-be-managed-by-people">Service accounts should not be managed by people</h4><p>Accounts used by applications and services have problems that differ from those of human accounts. There are three: the service stops when the password expires, the same password stays in use for years because people fear changing it, and the password is written in plaintext in configuration files.</p><p>Windows Server has an account type for this. A group Managed Service Account (gMSA) has its password managed automatically by Windows and can be used on multiple servers. Administrators never need to know the password. It works with Windows services, scheduled tasks, and other components that support gMSA, so for some applications you need to check whether they support it.</p>
<!--kg-card-begin: html-->
<ol style="padding:0;margin:24px 0;"><li style="position:relative;padding:6px 0 6px 44px;margin:0 0 8px;list-style:none;"><span style="position:absolute;left:0;top:4px;width:30px;height:30px;border-radius:50%;background:#1184DE;color:#fff;font-weight:700;display:flex;align-items:center;justify-content:center;font-size:14px;">1</span>Prepare a KDS root key (only once, the first time in the forest). In a production environment with multiple domain controllers, wait up to 10 hours after creating it for the key to replicate to all DCs before creating a gMSA</li><li style="position:relative;padding:6px 0 6px 44px;margin:0 0 8px;list-style:none;"><span style="position:absolute;left:0;top:4px;width:30px;height:30px;border-radius:50%;background:#1184DE;color:#fff;font-weight:700;display:flex;align-items:center;justify-content:center;font-size:14px;">2</span>Create a security group for the computers allowed to retrieve the password (WebServers in the example) and add the computer accounts of the target servers. After adding them, restart each server or have it obtain new Kerberos tickets so the group membership takes effect</li><li style="position:relative;padding:6px 0 6px 44px;margin:0 0 8px;list-style:none;"><span style="position:absolute;left:0;top:4px;width:30px;height:30px;border-radius:50%;background:#1184DE;color:#fff;font-weight:700;display:flex;align-items:center;justify-content:center;font-size:14px;">3</span>Create the gMSA. Pass <code>New-ADServiceAccount</code> a name, a DNS host name, and the group from step 2</li><li style="position:relative;padding:6px 0 6px 44px;margin:0 0 8px;list-style:none;"><span style="position:absolute;left:0;top:4px;width:30px;height:30px;border-radius:50%;background:#1184DE;color:#fff;font-weight:700;display:flex;align-items:center;justify-content:center;font-size:14px;">4</span>Install the gMSA on the server that uses it, and set the service&apos;s &quot;Log On&quot; account to <code>DOMAIN\name$</code> (leave the password fields empty)</li></ol>
<!--kg-card-end: html-->
<pre><code class="language-powershell"># 1. KDS root key (once per forest). Create it only if none exists. With multiple DCs in production, wait 10 hours after creating it
if (-not (Get-KdsRootKey)) {
  Add-KdsRootKey -EffectiveImmediately
}</code></pre><pre><code class="language-powershell"># 1&apos;. Only to skip the 10-hour wait in a lab (a single DC). Do not run this in production
Add-KdsRootKey -EffectiveTime ((Get-Date).AddHours(-10))</code></pre><pre><code class="language-powershell"># 2. Create the gMSA. Only computers in the WebServers group can retrieve the password
New-ADServiceAccount -Name &quot;svc_webapp&quot; `
  -DNSHostName &quot;svc_webapp.example.local&quot; `
  -PrincipalsAllowedToRetrieveManagedPassword &quot;WebServers&quot;

# 3. On the server that uses it
Install-ADServiceAccount -Identity &quot;svc_webapp&quot;
Test-ADServiceAccount -Identity &quot;svc_webapp&quot;   # True means it can be used</code></pre>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">About setting -EffectiveTime in the past.</b> Microsoft&apos;s documentation presents this as a step &quot;to skip the wait in a test environment with only one DC.&quot; If you rush this in production, password retrieval can fail before the key has replicated to all DCs. <code>-EffectiveImmediately</code> also only &quot;makes the key effective right away&quot;; it does not guarantee that replication to all DCs has finished.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/windows-server/security/group-managed-service-accounts/group-managed-service-accounts-overview?ref=journal.qualiteg.com" style="color:#0B5C9C;">Group Managed Service Accounts overview</a>, <a href="https://learn.microsoft.com/en-us/windows-server/security/group-managed-service-accounts/create-the-key-distribution-services-kds-root-key?ref=journal.qualiteg.com" style="color:#0B5C9C;">Create the Key Distribution Services KDS Root Key</a></div></div>
<!--kg-card-end: html-->
<h4 id="what-lies-beyond-passwords">What lies beyond passwords</h4><p>However well you tune the policy, a password is a secret you hand over to wherever you type it, so typing it into a fake sign-in page gets it stolen. NIST also states plainly that passwords are not phishing-resistant.</p><p>In an Active Directory environment, there are three paths toward authentication that does not rely on passwords. None of them &quot;sends a secret.&quot; Instead, they &quot;sign with a key held inside an authenticator&quot; (the device&apos;s security module, a smart card, or a security key). Phishing resistance comes from two things: the private key never leaves the authenticator, and the key is bound to the legitimate relying party, so it does not respond to a fake one.</p>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Method</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Something you have</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Something you know or are</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Phishing-resistant</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Where it fits</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;"><b>Smart card + PIN</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">A physical card holding a certificate</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">PIN</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">Yes</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Long-standing Active Directory environments. Requires a certificate infrastructure (AD CS)</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;"><b>Windows Hello for Business</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">A private key protected by the device&apos;s security module</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">PIN or biometrics. The PIN never leaves the device</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#0B5C9C;font-weight:600;">Yes</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Company-issued Windows PCs. A path to passwordless</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;"><b>FIDO2 security key (passkey)</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">A USB/NFC device holding the key</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">PIN or biometrics</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">Yes</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Heavily regulated industries and privileged users. Recommended by Microsoft</td></tr></tbody></table><div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/windows/security/identity-protection/hello-for-business/?ref=journal.qualiteg.com" style="color:#0B5C9C;">Windows Hello for Business overview</a>, <a href="https://learn.microsoft.com/en-us/entra/identity/authentication/concept-authentication-passkeys-fido2?ref=journal.qualiteg.com" style="color:#0B5C9C;">Passkeys (FIDO2) authentication method in Microsoft Entra ID</a>, <a href="https://pages.nist.gov/800-63-4/sp800-63b.html?ref=journal.qualiteg.com" style="color:#0B5C9C;">NIST SP 800-63B (Digital Identity Guidelines: Authentication)</a></div>
<!--kg-card-end: html-->
<p>It is also worth noting how this relates to integrated Windows authentication at the proxy, the main subject of this series. Even when users sign in with Windows Hello for Business or a smart card, Kerberos ticket issuance afterward is unchanged, so Negotiate authentication at the proxy keeps working. The only thing that changes is what is presented at the first sign-in. On the other hand, if you leave the Basic authentication fallback in place, it still needs a password, which undercuts the point of going passwordless. That is why 7.1.1 recommended limiting the fallback.</p><h3 id="713-minimizing-administrative-privileges">7.1.3 Minimizing Administrative Privileges</h3>
<!--kg-card-begin: html-->
<figure style="margin:24px auto 0;max-width:540px;"><img src="https://journal.qualiteg.com/content/images/2026/11/ad_part07_manga_7-1-3_en.png" alt="Understanding Active Directory: The Technology Behind Enterprise AI Security - Part 7: Security and Best Practices" style="width:100%;height:auto;border:1px solid #D5DDE6;border-radius:10px;display:block;"><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin:8px 0 0;line-height:1.6;">An audit finds 50 people in Domain Admins. The flashback (panels 2 and 3) shows a help desk member who only sets up printers holding the top-level privileges, and a mistake that affects the whole domain</figcaption></figure><div style="max-width:540px;margin:14px auto 28px;border:1px solid #D5DDE6;background:#fff;border-radius:10px;padding:10px 14px;font-size:13px;"><ul style="margin:0;padding:0;"><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Auditor</span><span style="flex:1 1 auto;line-height:1.7;">You have 50 people in Domain Admins</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">IT Manager</span><span style="flex:1 1 auto;line-height:1.7;">It was handy, so everyone got it...</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Help Desk</span><span style="flex:1 1 auto;line-height:1.7;">My job is just printer setup. But I&apos;m in Domain Admins...</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Help Desk</span><span style="flex:1 1 auto;line-height:1.7;">Oops! I changed a setting for the whole domain by mistake!</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Auditor</span><span style="flex:1 1 auto;line-height:1.7;">Give people only the rights their work needs</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">IT Manager</span><span style="flex:1 1 auto;line-height:1.7;">I see...</span></li></ul></div>
<!--kg-card-end: html-->
<p>Grant administrative privileges only to the extent the task requires, and only for as long as it is needed.</p><p>This is the principle Microsoft has consistently recommended for running Active Directory. Top-level groups such as Domain Admins are containers for emergencies, not for everyday work.</p>
<!--kg-card-begin: html-->
<div style="border-left:4px solid #1184DE;background:#F7FAFD;border-radius:0 10px 10px 0;padding:16px 20px;margin:24px 0;"><div style="margin:0 0 8px;font-size:15px;letter-spacing:.04em;color:#0B5C9C;font-weight:700;">What Microsoft recommends in its official documentation</div><ul style="margin:0;padding-left:1.2em;"><li style="margin:4px 0;">Domain Admins membership is needed only in &quot;break-glass&quot; situations that require elevated rights on every computer in the domain, and for domain-wide changes</li><li style="margin:4px 0;">Enterprise Admins should have 0 members day to day. Add members temporarily only when building a forest or setting up trusts between forests</li><li style="margin:4px 0;">Do not sign in to less privileged systems (member servers or PCs) with privileged accounts</li><li style="margin:4px 0;">Configure auditing so that alerts are raised when Domain Admins membership or attributes change</li></ul><div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/appendix-b--privileged-accounts-and-groups-in-active-directory?ref=journal.qualiteg.com" style="color:#0B5C9C;">Appendix B: Privileged Accounts and Groups in Active Directory</a>, <a href="https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/implementing-least-privilege-administrative-models?ref=journal.qualiteg.com" style="color:#0B5C9C;">Implementing Least-Privilege Administrative Models</a></div></div>
<!--kg-card-end: html-->
<h4 id="why-50-people-in-domain-admins-is-dangerous">Why 50 people in Domain Admins is dangerous</h4><p>There are two reasons.</p><p><strong>The entry points multiply to 50.</strong> If the privileged credentials of just one of the 50 are stolen, the attacker gains a foothold to take over the entire domain. On top of that, signing in to an ordinary PC with a Domain Admins account leaves traces of the credentials on that PC, and they can be extracted if the PC is compromised. This is why Microsoft says not to sign in to less privileged systems with privileged accounts.</p><p><strong>Mistakes spread to everything.</strong> Even a staff member who only installs printer drivers can touch Group Policy and the schema if they are in Domain Admins. When privileges exceed the scope of the job, nothing stops a mistake.</p><h4 id="grant-rights-per-task-delegation">Grant rights per task (delegation)</h4><p>In the manga&apos;s example, managing printers does not need Domain Admins. There are ways to grant only the scope required for each task.</p>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Task</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Common grant</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Recommended approach</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Reset passwords for users in a department</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#8B96A4;">Domain Admins</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">Delegate only &quot;Reset password&quot; on that department&apos;s OU</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Join a department&apos;s new PCs to the domain</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#8B96A4;">Domain Admins</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#0B5C9C;font-weight:600;">Delegate &quot;Create computer objects&quot; on that department&apos;s OU</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Manage printers</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#8B96A4;">Domain Admins</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">Admin rights limited to the target print server (that server&apos;s local groups, or the printer&apos;s manage permissions)</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Run backups and restores</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#8B96A4;">Domain Admins</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#0B5C9C;font-weight:600;">Limit the scope with local rights on the target servers, or with roles in the backup product</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Change domain-wide settings</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#8B96A4;">Standing Domain Admins</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">Add to Domain Admins only for the task, and remove when done</td></tr></tbody></table>
<!--kg-card-end: html-->

<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">The built-in Operators groups are not a substitute.</b> Account Operators, Server Operators, Backup Operators, and Print Operators are built-in domain groups, and Microsoft treats them as privileged groups. Account Operators can manage domain users and groups broadly, Server Operators can administer domain controllers, and Backup Operators can override security restrictions for the purpose of backup and restore. Rather than putting people in these groups because they seem &quot;better than Domain Admins,&quot; the basic approach, when the required scope is defined by an OU or specific servers, is to narrow the scope and operations with the Delegation of Control wizard in Active Directory Users and Computers or with local rights on the servers.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/appendix-b--privileged-accounts-and-groups-in-active-directory?ref=journal.qualiteg.com" style="color:#0B5C9C;">Appendix B: Privileged Accounts and Groups in Active Directory</a></div></div>
<!--kg-card-end: html-->
<h4 id="separate-by-tier-the-tier-model">Separate by tier (the Tier model)</h4><p>Once delegation has narrowed rights per task, the next step is to decide which tier&apos;s credentials may be used on which tier. The Tier model that Microsoft has presented for Active Directory splits what you protect into three tiers and does not allow upper-tier credentials to be used on lower tiers.</p>
<!--kg-card-begin: html-->
<figure style="margin:24px 0;"><svg viewbox="0 0 720 300" width="100%" xmlns="http://www.w3.org/2000/svg" font-family="Inter, &apos;Segoe UI&apos;, sans-serif">
  <rect x="20" y="20" width="440" height="72" rx="10" fill="#0B5C9C"/>
  <text x="40" y="50" fill="#fff" font-size="18" font-weight="700">Tier 0&#xA0;&#xA0;Control layer</text>
  <text x="40" y="76" fill="#fff" font-size="14">Domain controllers, AD, identity infrastructure</text>
  <rect x="20" y="114" width="440" height="72" rx="10" fill="#1184DE"/>
  <text x="40" y="144" fill="#fff" font-size="18" font-weight="700">Tier 1&#xA0;&#xA0;Servers and applications</text>
  <text x="40" y="170" fill="#fff" font-size="14">Member servers, business applications</text>
  <rect x="20" y="208" width="440" height="72" rx="10" fill="#8B96A4"/>
  <text x="40" y="238" fill="#fff" font-size="18" font-weight="700">Tier 2&#xA0;&#xA0;Users</text>
  <text x="40" y="264" fill="#fff" font-size="14">Client PCs, help desk work</text>
  <rect x="490" y="20" width="210" height="260" rx="10" fill="#F7FAFD" stroke="#D5DDE6"/>
  <text x="510" y="52" fill="#15222B" font-size="15" font-weight="700">Rules</text>
  <text x="510" y="84" fill="#15222B" font-size="13">&#x2022; Separate admin</text>
  <text x="523" y="106" fill="#15222B" font-size="13">accounts per tier</text>
  <text x="510" y="138" fill="#15222B" font-size="13">&#x2022; No higher-tier creds</text>
  <text x="523" y="160" fill="#15222B" font-size="13">on lower-tier devices</text>
  <text x="510" y="192" fill="#15222B" font-size="13">&#x2022; Tier 0 work only</text>
  <text x="523" y="214" fill="#15222B" font-size="13">from a dedicated PAW</text>
  <text x="510" y="246" fill="#15222B" font-size="13">&#x2022; Lower tiers never</text>
  <text x="523" y="268" fill="#15222B" font-size="13">manage higher tiers</text>
  <line x1="470" y1="56" x2="470" y2="244" stroke="#284C6E" stroke-width="2" stroke-dasharray="6 5"/>
  <circle cx="470" cy="150" r="14" fill="#fff" stroke="#284C6E" stroke-width="2"/>
  <line x1="461" y1="141" x2="479" y2="159" stroke="#284C6E" stroke-width="2.5"/>
</svg><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin-top:6px;">Figure: The three tiers of the Tier model. The key point is never using upper-tier credentials on a lower tier</figcaption></figure>
<!--kg-card-end: html-->

<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Tier</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">What it protects</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Who manages it</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Rules</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Tier 0</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Domain controllers, AD, identity infrastructure</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">A very small dedicated team</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Dedicated admin accounts and dedicated devices (PAWs). Never sign in to other tiers</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Tier 1</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Member servers, business applications</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Server administrators</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Do not use Tier 0 credentials. Do not touch Tier 0 settings</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Tier 2</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Client PCs</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Help desk, PC administrators</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Do not touch servers</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Implementation starts with creating dedicated admin accounts and groups for each tier (for example, Tier0-Admins, Tier1-ServerAdmins, and Tier2-WorkstationAdmins). Then use user rights assignment in Group Policy so that each tier&apos;s admin accounts cannot sign in to devices and servers in lower tiers. There are five kinds of logon rights: interactive, Remote Desktop, network, logon as a service, and logon as a batch job. Decide which ones to deny based on the account&apos;s purpose and the systems it manages. If you deny all five across the board, admin accounts can no longer get into their legitimate management servers, so plan the paths to PAWs and management servers separately. Design the scope of OU delegation to match the tiers as well.</p>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">How Microsoft puts it today.</b> Building on the Tier model, Microsoft now presents the &quot;enterprise access model,&quot; which includes the cloud. Tier 0 has expanded into the control plane, the old Tier 1 has split into the management plane and the data/workload plane, and the old Tier 2 has split into user access and app access. This is not a renaming; it redraws the boundaries between roles. For protecting on-premises Active Directory, the three-tier idea is the starting point for design, and in environments that include cloud management infrastructure, you revisit the boundaries along the enterprise access model.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/security/privileged-access-workstations/privileged-access-access-model?ref=journal.qualiteg.com" style="color:#0B5C9C;">Enterprise access model</a></div></div>
<!--kg-card-end: html-->
<h4 id="do-top-level-work-from-a-dedicated-device-paw">Do top-level work from a dedicated device (PAW)</h4><p>Tier 0 administrators also use separate devices for their work. This is the privileged access workstation (PAW).</p><p>A PAW is the strictest device configuration, intended for roles whose compromise would have a serious impact on the organization. It has three aims: protect administrator credentials, reduce the device&apos;s attack surface, and separate the management path from everyday work. At its core is cutting email and web browsing, the main phishing routes, off from the management device. Email and the web are used with a different account on a different device.</p>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Device level</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">General apps and web browsing</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Apps that can run</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Other</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;"><b>Enterprise (general work)</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#0B5C9C;font-weight:600;">Allowed</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">No restrictions</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Anti-malware and EDR required</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;"><b>Specialized (specialized work)</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;color:#0B5C9C;font-weight:600;">Allowed</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Only apps installed by administrators</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Users cannot manage their own devices</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;"><b>Privileged (PAW)</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;color:#8B96A4;">Not allowed</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Strictly limited with application control</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Disk encrypted with BitLocker. Web access denied by default except for an allow list</td></tr></tbody></table><div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/security/privileged-access-workstations/privileged-access-devices?ref=journal.qualiteg.com" style="color:#0B5C9C;">Securing devices as part of the privileged access story</a></div>
<!--kg-card-end: html-->
<h4 id="what-you-can-do-this-week">What you can do this week</h4>
<!--kg-card-begin: html-->
<ol style="padding:0;margin:24px 0;"><li style="position:relative;padding:6px 0 6px 44px;margin:0 0 8px;list-style:none;"><span style="position:absolute;left:0;top:4px;width:30px;height:30px;border-radius:50%;background:#1184DE;color:#fff;font-weight:700;display:flex;align-items:center;justify-content:center;font-size:14px;">1</span>List the members of Domain Admins and Enterprise Admins (the audit script in 7.2.3 gets them for you)</li><li style="position:relative;padding:6px 0 6px 44px;margin:0 0 8px;list-style:none;"><span style="position:absolute;left:0;top:4px;width:30px;height:30px;border-radius:50%;background:#1184DE;color:#fff;font-weight:700;display:flex;align-items:center;justify-content:center;font-size:14px;">2</span>Remove the accounts used for everyday work. Give administrators two accounts: one for regular use and one for administration</li><li style="position:relative;padding:6px 0 6px 44px;margin:0 0 8px;list-style:none;"><span style="position:absolute;left:0;top:4px;width:30px;height:30px;border-radius:50%;background:#1184DE;color:#fff;font-weight:700;display:flex;align-items:center;justify-content:center;font-size:14px;">3</span>Replace the work of the people you removed with delegated rights on OUs, local rights on the target servers, or roles in management products</li><li style="position:relative;padding:6px 0 6px 44px;margin:0 0 8px;list-style:none;"><span style="position:absolute;left:0;top:4px;width:30px;height:30px;border-radius:50%;background:#1184DE;color:#fff;font-weight:700;display:flex;align-items:center;justify-content:center;font-size:14px;">4</span>Create admin groups for each tier, and block cross-tier sign-ins with Group Policy</li><li style="position:relative;padding:6px 0 6px 44px;margin:0 0 8px;list-style:none;"><span style="position:absolute;left:0;top:4px;width:30px;height:30px;border-radius:50%;background:#1184DE;color:#fff;font-weight:700;display:flex;align-items:center;justify-content:center;font-size:14px;">5</span>Provide PAWs for Tier 0 administrators</li></ol>
<!--kg-card-end: html-->
<h2 id="72-monitoring-and-logging">7.2 Monitoring and Logging</h2><h3 id="721-collecting-and-analyzing-authentication-logs">7.2.1 Collecting and Analyzing Authentication Logs</h3>
<!--kg-card-begin: html-->
<figure style="margin:24px auto 0;max-width:540px;"><img src="https://journal.qualiteg.com/content/images/2026/11/ad_part07_manga_7-2-1_en.png" alt="Understanding Active Directory: The Technology Behind Enterprise AI Security - Part 7: Security and Best Practices" style="width:100%;height:auto;border:1px solid #D5DDE6;border-radius:10px;display:block;"><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin:8px 0 0;line-height:1.6;">Four kinds of logs to collect: the Windows security log, proxy access logs, Active Directory changes, and DNS queries</figcaption></figure><div style="max-width:540px;margin:14px auto 28px;border:1px solid #D5DDE6;background:#fff;border-radius:10px;padding:10px 14px;font-size:13px;"><ul style="margin:0;padding:0;"><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">Which logs should we collect?</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">CISO</span><span style="flex:1 1 auto;line-height:1.7;">Four kinds tied to auth: Windows, proxy, AD changes, and DNS</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">CISO</span><span style="flex:1 1 auto;line-height:1.7;">Don&apos;t leave them on the machines. Collect them in one place</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">In one place, we can search them right away during an investigation</span></li></ul></div>
<!--kg-card-end: html-->
<p>The sections so far were about hardening the entry points. From here on, the topic is noticing when someone has gotten in.</p><p>To notice, you need to record authentication-related events without gaps, gather them in one place, and keep them in a form people can read. In an Active Directory environment, the following four kinds of logs form the foundation.</p>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Log</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Where to get it</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">What it tells you</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Typical fields</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Windows security log</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Domain controllers, each server, PCs</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Who signed in, when, from where, and by what method. Failures and lockouts</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Event IDs 4624 / 4625 / 4672 / 4740</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Proxy access logs</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Proxy server</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Who sent and received how much data to and from which external sites. How often authentication was requested (407)</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">User name, destination, date and time, bytes, status code</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;"><b>Active Directory changes</b></td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Domain controllers</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Creation and deletion of users and groups, additions to group membership, password changes</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Additions to groups are event IDs 4728 / 4732 / 4756</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">DNS queries</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">DNS servers</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Queries for names used by malware, surges in nonexistent names (NXDOMAIN)</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Query source, name, response code</td></tr></tbody></table><div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4624?ref=journal.qualiteg.com" style="color:#0B5C9C;">Event 4624: An account was successfully logged on</a>, <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4625?ref=journal.qualiteg.com" style="color:#0B5C9C;">Event 4625: An account failed to log on</a>, <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4672?ref=journal.qualiteg.com" style="color:#0B5C9C;">Event 4672: Special privileges assigned to new logon</a>, <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4740?ref=journal.qualiteg.com" style="color:#0B5C9C;">Event 4740: A user account was locked out</a>, <a href="https://learn.microsoft.com/en-us/windows/security/threat-protection/use-windows-event-forwarding-to-assist-in-intrusion-detection?ref=journal.qualiteg.com" style="color:#0B5C9C;">Use Windows Event Forwarding to help with intrusion detection</a></div>
<!--kg-card-end: html-->

<!--kg-card-begin: html-->
<div style="border-left:4px solid #1184DE;background:#F7FAFD;border-radius:0 10px 10px 0;padding:16px 20px;margin:24px 0;"><div style="margin:0 0 8px;font-size:15px;letter-spacing:.04em;color:#0B5C9C;font-weight:700;">Event IDs to cover first</div><ul style="margin:0;padding-left:1.2em;"><li style="margin:4px 0;"><b>4624</b> Successful logon. The logon type (Logon Type) distinguishes interactive (2), network (3), Remote Desktop (10), and so on</li><li style="margin:4px 0;"><b>4625</b> Failed logon. The first sign of brute-force attacks and password spraying</li><li style="margin:4px 0;"><b>4672</b> Logon with special privileges. Raised when a session with administrator-level rights is created. It also fires for SYSTEM logons, so collect it with exclusion conditions</li><li style="margin:4px 0;"><b>4740</b> Account lockout. A sign that failures exceeded the threshold</li></ul><div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4624?ref=journal.qualiteg.com" style="color:#0B5C9C;">Event 4624: An account was successfully logged on</a>, <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4625?ref=journal.qualiteg.com" style="color:#0B5C9C;">Event 4625: An account failed to log on</a>, <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4672?ref=journal.qualiteg.com" style="color:#0B5C9C;">Event 4672: Special privileges assigned to new logon</a>, <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4740?ref=journal.qualiteg.com" style="color:#0B5C9C;">Event 4740: A user account was locked out</a></div></div>
<!--kg-card-end: html-->
<h4 id="how-to-collect-windows-event-forwarding">How to collect: Windows Event Forwarding</h4><p>Logs are useless if they stay on the machine where they were generated, because the logs on a compromised machine get erased. Windows has a built-in way to send logs to one place without additional products, called Windows Event Forwarding (WEF).</p>
<!--kg-card-begin: html-->
<figure style="margin:24px 0;"><svg viewbox="0 0 720 280" width="100%" xmlns="http://www.w3.org/2000/svg" font-family="Inter, &apos;Segoe UI&apos;, sans-serif">
  <defs><marker id="arw2" markerwidth="10" markerheight="10" refx="8" refy="5" orient="auto"><path d="M0,0 L10,5 L0,10 z" fill="#15222B"/></marker></defs>
  <rect x="20" y="22" width="170" height="46" rx="8" fill="#F7FAFD" stroke="#D5DDE6"/>
  <text x="105" y="50" text-anchor="middle" fill="#15222B" font-size="13" font-weight="700">Domain controller</text>
  <line x1="192" y1="45" x2="300" y2="140" stroke="#8B96A4" stroke-width="1.5" marker-end="url(#arw2)"/><rect x="20" y="84" width="170" height="46" rx="8" fill="#F7FAFD" stroke="#D5DDE6"/>
  <text x="105" y="112" text-anchor="middle" fill="#15222B" font-size="13" font-weight="700">Proxy server</text>
  <line x1="192" y1="107" x2="300" y2="140" stroke="#8B96A4" stroke-width="1.5" marker-end="url(#arw2)"/><rect x="20" y="146" width="170" height="46" rx="8" fill="#F7FAFD" stroke="#D5DDE6"/>
  <text x="105" y="174" text-anchor="middle" fill="#15222B" font-size="13" font-weight="700">Member server</text>
  <line x1="192" y1="169" x2="300" y2="140" stroke="#8B96A4" stroke-width="1.5" marker-end="url(#arw2)"/><rect x="20" y="208" width="170" height="46" rx="8" fill="#F7FAFD" stroke="#D5DDE6"/>
  <text x="105" y="236" text-anchor="middle" fill="#15222B" font-size="13" font-weight="700">Client PC</text>
  <line x1="192" y1="231" x2="300" y2="140" stroke="#8B96A4" stroke-width="1.5" marker-end="url(#arw2)"/>
  <rect x="305" y="95" width="170" height="90" rx="10" fill="#0B5C9C"/>
  <text x="390" y="130" text-anchor="middle" fill="#fff" font-size="15" font-weight="700">Collector</text>
  <text x="390" y="152" text-anchor="middle" fill="#fff" font-size="11">Windows Event Forwarding</text>
  <text x="390" y="170" text-anchor="middle" fill="#fff" font-size="11">Source-initiated subscription</text>
  <line x1="477" y1="140" x2="540" y2="140" stroke="#15222B" stroke-width="2" marker-end="url(#arw2)"/>
  <rect x="545" y="95" width="155" height="90" rx="10" fill="#1184DE"/>
  <text x="622" y="130" text-anchor="middle" fill="#fff" font-size="15" font-weight="700">Analyze, store</text>
  <text x="622" y="152" text-anchor="middle" fill="#fff" font-size="12">SIEM, search, retention</text>
  <text x="622" y="170" text-anchor="middle" fill="#fff" font-size="12">Tamper detection</text>
  <text x="105" y="272" text-anchor="middle" fill="#4A463F" font-size="12">winrm qc -q on each</text>
  <text x="390" y="215" text-anchor="middle" fill="#4A463F" font-size="12">wecutil qc /q</text>
  <text x="460" y="242" text-anchor="middle" fill="#15222B" font-size="13">Two subscriptions: a minimal baseline from every</text>
  <text x="460" y="261" text-anchor="middle" fill="#15222B" font-size="13">machine, plus details from suspect machines</text>
</svg><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin-top:6px;">Figure: Gather each machine&apos;s event logs at a collector and pass them on for analysis and storage</figcaption></figure>
<!--kg-card-end: html-->
<p>There are two key settings. Configure the collection service on the collector, and enable Windows Remote Management on the sending side. Each is a one-line command.</p><pre><code class="language-powershell"># On the collector (the receiving side), run as administrator
wecutil qc /q
winrm qc -q

# On each source (the sending side). This can also be configured in bulk with Group Policy
winrm qc -q</code></pre><p>This is only initialization, and no events flow yet. To start forwarding, create a &quot;source-initiated&quot; subscription on the collector, distribute the collector&apos;s location (SubscriptionManager) to the sending side through Group Policy, confirm that the sending side has permission to read the Security log and the right audit policy, and verify that events arrive in the collector&apos;s &quot;Forwarded Events&quot; log.</p><p>Which events a subscription collects is specified with XPath. For example, to collect only successful logons, failed logons, and lockouts, use the following.</p><pre><code class="language-xml">&lt;QueryList&gt;
  &lt;Query Id=&quot;0&quot; Path=&quot;Security&quot;&gt;
    &lt;Select Path=&quot;Security&quot;&gt;
      *[System[(EventID=4624 or EventID=4625 or EventID=4740)]]
    &lt;/Select&gt;
  &lt;/Query&gt;
&lt;/QueryList&gt;</code></pre><pre><code class="language-powershell"># On the collector, check the list and status of the subscriptions you created
wecutil es
wecutil gr &quot;SubscriptionName&quot;   # Shows each source&apos;s connection status and last arrival time</code></pre>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">Microsoft&apos;s two-tier approach.</b> Microsoft describes a configuration with two subscriptions: a &quot;baseline&quot; subscription that collects a minimal set from every machine, and a &quot;targeted&quot; subscription that collects details only from suspicious machines. Collecting everything from every machine drowns you in volume, while watching only suspicious machines misses the first signs. If you track Kerberos authentication on domain controllers, also include the ticket request events (4768 / 4769 / 4771).<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/windows/win32/wec/setting-up-a-source-initiated-subscription?ref=journal.qualiteg.com" style="color:#0B5C9C;">Setting up a Source Initiated Subscription</a>, <a href="https://learn.microsoft.com/en-us/windows/security/threat-protection/use-windows-event-forwarding-to-assist-in-intrusion-detection?ref=journal.qualiteg.com" style="color:#0B5C9C;">Use Windows Event Forwarding to help with intrusion detection</a></div></div>
<!--kg-card-end: html-->
<h4 id="how-to-read-them-catching-late-night-logons">How to read them: catching late-night logons</h4><p>The first step in spotting anomalies in the collected logs can be a simple condition. For example, &quot;successful interactive logons in the last 24 hours before 6 a.m. or after 10 p.m.&quot;</p><pre><code class="language-powershell"># List interactive logons (LogonType 2 and 10) late at night or early in the morning over the last 24 hours
New-Item -ItemType Directory -Path &apos;C:\\audit&apos; -Force | Out-Null
$since = (Get-Date).AddDays(-1)
$hits = Get-WinEvent -FilterHashtable @{ LogName = &apos;Security&apos;; ID = 4624; StartTime = $since } |
  Where-Object {
    $lt = $_.Properties[8].Value
    ($lt -eq 2 -or $lt -eq 10) -and ($_.TimeCreated.Hour -lt 6 -or $_.TimeCreated.Hour -ge 22)
  } |
  Select-Object TimeCreated,
    @{ N = &apos;User&apos;;        E = { $_.Properties[5].Value } },
    @{ N = &apos;Domain&apos;;      E = { $_.Properties[6].Value } },
    @{ N = &apos;LogonType&apos;;   E = { $_.Properties[8].Value } },
    @{ N = &apos;Workstation&apos;; E = { $_.Properties[11].Value } }

$hits | Format-Table -AutoSize
$hits | Export-Csv -Path &quot;C:\\audit\\night-logons-$(Get-Date -Format yyyyMMdd).csv&quot; -NoTypeInformation -Encoding UTF8</code></pre>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">About the Properties indexes.</b> In the XML of event 4624, TargetUserName is the 6th field (5 when counting from 0), TargetDomainName is the 7th (6), LogonType is the 9th (8), and WorkstationName is the 12th (11). The indexes differ by event ID, so check the XML of the event when you handle a different one. For notifications, the current recommendation is to send them to a ticketing system or a chat webhook rather than with <code>Send-MailMessage</code>. Microsoft marks <code>Send-MailMessage</code> as obsolete, and it does not guarantee secure connections to SMTP servers.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/auditing/event-4624?ref=journal.qualiteg.com" style="color:#0B5C9C;">Event 4624: An account was successfully logged on</a>, <a href="https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/send-mailmessage?ref=journal.qualiteg.com" style="color:#0B5C9C;">Send-MailMessage</a></div></div>
<!--kg-card-end: html-->
<h3 id="722-detecting-abnormal-access-patterns">7.2.2 Detecting Abnormal Access Patterns</h3>
<!--kg-card-begin: html-->
<figure style="margin:24px auto 0;max-width:540px;"><img src="https://journal.qualiteg.com/content/images/2026/11/ad_part07_manga_7-2-2_en.png" alt="Understanding Active Directory: The Technology Behind Enterprise AI Security - Part 7: Security and Best Practices" style="width:100%;height:auto;border:1px solid #D5DDE6;border-radius:10px;display:block;"><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin:8px 0 0;line-height:1.6;">Someone who signed in from the office 30 minutes ago has also signed in over VPN from overseas. Someone else may be using the same credentials</figcaption></figure><div style="max-width:540px;margin:14px auto 28px;border:1px solid #D5DDE6;background:#fff;border-radius:10px;padding:10px 14px;font-size:13px;"><ul style="margin:0;padding:0;"><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">IT Manager</span><span style="flex:1 1 auto;line-height:1.7;">Someone in the office 30 minutes ago just logged on from overseas?</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">CISO</span><span style="flex:1 1 auto;line-height:1.7;">Nobody can travel that far in 30 minutes. Someone else is using the account</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">CISO</span><span style="flex:1 1 auto;line-height:1.7;">We catch late-night logons, mass uploads, and repeated auth failures the same way</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">IT Manager</span><span style="flex:1 1 auto;line-height:1.7;">I&apos;ll disable the account and check with the owner</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">CISO</span><span style="flex:1 1 auto;line-height:1.7;">Please do</span></li></ul></div>
<!--kg-card-end: html-->
<p>Once the logs are collected, the next step is finding what is &quot;different from usual.&quot;</p><p>To do that, you first need to know what &quot;usual&quot; is: who uses which systems, at what times, from which devices. This is called a baseline. Baselines differ by person and department, so some things slip past a single company-wide threshold.</p>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">What to watch</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Example baseline</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Example anomaly</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Main data source</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Time of day</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Interactive logons during weekday business hours</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Interactive logons late at night or on holidays</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Security log 4624</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Location and device</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Usual devices, the office network, the usual VPN egress</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Unfamiliar devices, overseas IPs, two distant places within a short time</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Security log, VPN logs</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Systems used</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Only the servers that match the role</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Network logons (Logon Type 3) to servers unrelated to the role</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Security log 4624, 4672</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Traffic volume</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Distribution of daily sent and received volume</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Large downloads or uploads in a short time</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Proxy access logs</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Authentication failures</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Occasional typos</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">A few failures each across many users (password spraying), repeated failures for one user</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Security log 4625 / 4740, proxy authentication result logs (frequent 407s are a supporting sign)</td></tr></tbody></table>
<!--kg-card-end: html-->
<h4 id="impossible-travel">Impossible travel</h4>
<!--kg-card-begin: html-->
<figure style="margin:24px 0;"><svg viewbox="0 0 720 220" width="100%" xmlns="http://www.w3.org/2000/svg" font-family="Inter, &apos;Segoe UI&apos;, sans-serif">
  <line x1="60" y1="150" x2="660" y2="150" stroke="#D5DDE6" stroke-width="4"/>
  <line x1="60" y1="142" x2="60" y2="158" stroke="#8B96A4" stroke-width="2"/><line x1="160" y1="142" x2="160" y2="158" stroke="#8B96A4" stroke-width="2"/><line x1="260" y1="142" x2="260" y2="158" stroke="#8B96A4" stroke-width="2"/><line x1="360" y1="142" x2="360" y2="158" stroke="#8B96A4" stroke-width="2"/><line x1="460" y1="142" x2="460" y2="158" stroke="#8B96A4" stroke-width="2"/><line x1="560" y1="142" x2="560" y2="158" stroke="#8B96A4" stroke-width="2"/><line x1="660" y1="142" x2="660" y2="158" stroke="#8B96A4" stroke-width="2"/>
  <text x="60" y="182" text-anchor="middle" fill="#4A463F" font-size="12">9:00</text>
  <text x="360" y="182" text-anchor="middle" fill="#4A463F" font-size="12">9:30</text>
  <text x="660" y="182" text-anchor="middle" fill="#4A463F" font-size="12">10:00</text>
  <circle cx="60" cy="150" r="12" fill="#0B5C9C"/>
  <text x="20" y="80" text-anchor="start" fill="#15222B" font-size="14" font-weight="700">Logon from the office</text>
  <text x="20" y="100" text-anchor="start" fill="#4A463F" font-size="12">Interactive (Logon Type 2)</text>
  <circle cx="360" cy="150" r="12" fill="#284C6E"/>
  <text x="360" y="80" text-anchor="middle" fill="#15222B" font-size="14" font-weight="700">VPN logon from an overseas IP</text>
  <text x="360" y="100" text-anchor="middle" fill="#4A463F" font-size="12">Same user name</text>
  <path d="M60,138 Q210,30 360,138" fill="none" stroke="#284C6E" stroke-width="2" stroke-dasharray="6 5"/>
  <circle cx="210" cy="84" r="16" fill="#fff" stroke="#284C6E" stroke-width="2"/>
  <line x1="200" y1="74" x2="220" y2="94" stroke="#284C6E" stroke-width="2.5"/>
  <line x1="220" y1="74" x2="200" y2="94" stroke="#284C6E" stroke-width="2.5"/>
  <text x="360" y="212" text-anchor="middle" fill="#15222B" font-size="13">Too far to travel in 30 minutes. Someone else may be using the same credentials</text>
</svg><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin-top:6px;">Figure: Logons with the same user name are recorded from distant places with a time gap too short to travel</figcaption></figure>
<!--kg-card-end: html-->
<p>The easiest anomaly to understand is &quot;impossible travel.&quot; Microsoft Defender for Cloud Apps defines it as activity from two geographically distant locations recorded within a time shorter than it would take to travel between them, and detects it as a sign that someone else is using the same credentials.</p><p>The idea is simple. Compute a speed from the distance between the two logon locations and the time between them, and raise an alert if it exceeds a realistic limit. Locations are derived from IP addresses at the country or region level. If you implement this yourself, logons inside the office come from private addresses that cannot be mapped to a location, so treat &quot;the office network&quot; as a single location, and for VPN logons use the source IP recorded in the VPN connection logs. You need to match authentication logs and VPN logs by user name and time.</p>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">Typical false positives.</b> When traffic goes through a VPN or a cloud proxy, the egress IP can be in a different country from where the user actually is. Registering your company&apos;s VPN egress and the routes of the cloud services you use as &quot;known locations&quot; in advance eliminates most of these alerts.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/defender-cloud-apps/anomaly-detection-policy?ref=journal.qualiteg.com" style="color:#0B5C9C;">Defender for Cloud Apps anomaly detection policies</a></div></div>
<!--kg-card-end: html-->
<h4 id="what-to-look-for-in-proxy-logs">What to look for in proxy logs</h4><p>In this series&apos; environment, all traffic to external generative AI services goes through the proxy. That means the proxy logs are the very record of &quot;who sent what outside.&quot; If they are loaded into a form you can aggregate with SQL, the following three queries are a starting point (written in MySQL 8.0 syntax).</p><pre><code class="language-sql">-- 1. Users whose total sent and received exceeded 1 GiB in the last hour (bytes is assumed to be the column for total sent and received)
SELECT username, SUM(bytes) AS total_bytes, COUNT(*) AS requests
FROM proxy_logs
WHERE ts &gt; NOW() - INTERVAL 1 HOUR
GROUP BY username
HAVING SUM(bytes) &gt; 1073741824;

-- 2. Connections to destinations not on the allow list (most frequent first). NOT IN returns nothing if the allow list contains NULL, so use NOT EXISTS
SELECT p.username, p.dst_host, COUNT(*) AS cnt
FROM proxy_logs AS p
WHERE NOT EXISTS (
  SELECT 1 FROM allowed_domains AS a WHERE a.domain = p.dst_host
)
GROUP BY p.username, p.dst_host
ORDER BY cnt DESC;

-- 3. Users who received more than 10 responses of 407 in the last 10 minutes (407 = Proxy Authentication Required)
SELECT username, COUNT(*) AS challenges
FROM proxy_logs
WHERE status_code = 407
  AND ts &gt; NOW() - INTERVAL 10 MINUTE
GROUP BY username
HAVING COUNT(*) &gt; 10;</code></pre><p>The 407 in the third query is the response RFC 9110 defines as &quot;authentication is required to use the proxy.&quot; It is not an authentication failure in itself. Even with normal authentication, the first request gets one 407 back, and the request then goes through with credentials attached. So frequent 407s mean &quot;suspected failures,&quot; not confirmed ones. To confirm, look at the proxy&apos;s own authentication result logs (for Squid, the results of the authentication helper). The first 407 often has no user name attached, so track it by source IP or per connection.</p><p>Thresholds such as 1 GiB and 10 times are only a starting point. Collect your own baseline for 1 to 2 weeks, then reset them just outside that distribution. The destination check in the second query is an exact host name match, so decide separately whether the allow list should cover subdomains.</p>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">The table and column names are examples.</b> <code>proxy_logs</code> and <code>allowed_domains</code> are example tables for proxy access logs loaded into a database. Adapt them to the actual log format and column names of Squid or whatever proxy you use.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://www.rfc-editor.org/rfc/rfc9110?ref=journal.qualiteg.com" style="color:#0B5C9C;">RFC 9110 HTTP Semantics</a></div></div>
<!--kg-card-end: html-->
<h3 id="723-conducting-regular-audits">7.2.3 Conducting Regular Audits</h3>
<!--kg-card-begin: html-->
<figure style="margin:24px auto 0;max-width:540px;"><img src="https://journal.qualiteg.com/content/images/2026/11/ad_part07_manga_7-2-3_en.png" alt="Understanding Active Directory: The Technology Behind Enterprise AI Security - Part 7: Security and Best Practices" style="width:100%;height:auto;border:1px solid #D5DDE6;border-radius:10px;display:block;"><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin:8px 0 0;line-height:1.6;">A monthly audit done by hand does not last. A script produces the lists, and people only make the judgments</figcaption></figure><div style="max-width:540px;margin:14px auto 28px;border:1px solid #D5DDE6;background:#fff;border-radius:10px;padding:10px 14px;font-size:13px;"><ul style="margin:0;padding:0;"><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Auditor</span><span style="flex:1 1 auto;line-height:1.7;">On the 1st of every month, we check these 5 items</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">IT Manager</span><span style="flex:1 1 auto;line-height:1.7;">All by hand? We haven&apos;t even finished last month&apos;s yet...</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Developer</span><span style="flex:1 1 auto;line-height:1.7;">Let&apos;s leave generating the lists to a script</span></li><li style="display:flex;gap:8px;align-items:flex-start;margin:0 0 5px;list-style:none;"><span style="flex:0 0 auto;min-width:5em;text-align:center;background:#0B5C9C;color:#fff;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:700;line-height:1.7;">Auditor</span><span style="flex:1 1 auto;line-height:1.7;">Then people only judge the rows that differ from the target state</span></li></ul></div>
<!--kg-card-end: html-->
<p>Settings drift a little at a time from the day they are put in place. Someone added to Domain Admins temporarily stays there, accounts of people who have left stay enabled, a password set to never expire as an exception is forgotten. This is called &quot;configuration drift.&quot;</p><p>The basic way to find drift and correct it is to write out the current state at fixed intervals and compare it with the target state. Once a month, check the following five items. These five are the audit items this article recommends, each mapped to what was decided in 7.1. In addition, changes to privileged group membership (event IDs 4728 / 4732 / 4756) are monitored continuously with the mechanism from 7.2.1 instead of waiting for the monthly audit, and the monthly audit serves to catch what that missed.</p>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Item</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">What to check</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Target state (what was decided in 7.1)</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">1. Privileged accounts</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Members of Domain Admins, Enterprise Admins, and Schema Admins, and their last logon</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Enterprise Admins has 0 members. Domain Admins has only a few break-glass members. No everyday-work accounts are in them</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">2. Password policy</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">The domain policy and the contents of fine-grained password policies (PSOs)</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Still the length and maximum age decided in 7.1.2. No unknown exceptions</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">3. Dormant accounts</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Enabled users with no logon for 90 days or more</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Accounts of people who have left are disabled. Long absences are known</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">4. Service accounts</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Accounts whose passwords never expire, and their rights</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Those that can be moved are migrated to gMSA. The reasons for the rest are recorded</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">5. Log settings</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Audit policy, WEF subscription status, free space at the storage location</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">The events from 7.2.1 arrive from every machine. No machine is missing</td></tr></tbody></table><div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/appendix-b--privileged-accounts-and-groups-in-active-directory?ref=journal.qualiteg.com" style="color:#0B5C9C;">Appendix B: Privileged Accounts and Groups in Active Directory</a></div>
<!--kg-card-end: html-->
<h4 id="doing-it-by-hand-does-not-last-so-automate-the-export">Doing it by hand does not last, so automate the export</h4><p>Checking five items by hand every month does not last. Leave producing the lists to a script, and have people judge only the &quot;rows that differ from the target state.&quot; The following script runs on a domain controller or a management machine with RSAT installed, and exports to CSV the lists for items 1 to 4 and the current values of the item 2 policies. Item 5 (log settings) is checked with <code>wecutil gr</code> from 7.2.1 and by reviewing the audit policy.</p><pre><code class="language-powershell"># Monthly AD audit. Export the lists to CSV (requires the ActiveDirectory module)
Import-Module ActiveDirectory
$stamp  = Get-Date -Format yyyyMM
$outDir = &quot;C:\\audit\\$stamp&quot;
New-Item -ItemType Directory -Path $outDir -Force | Out-Null

# 1. Privileged group members. Keep direct members (nested groups also appear as rows) and effective members with nesting expanded
#    in the same 7 columns (Export-Csv cannot append correctly unless the column structure matches)
$rows = foreach ($g in &apos;Domain Admins&apos;, &apos;Enterprise Admins&apos;, &apos;Schema Admins&apos;) {
  Get-ADGroupMember -Identity $g |
    Select-Object @{ N = &apos;Group&apos;; E = { $g } }, @{ N = &apos;Kind&apos;; E = { &apos;direct&apos; } },
      objectClass, SamAccountName, Name,
      @{ N = &apos;Enabled&apos;; E = { $null } }, @{ N = &apos;LastLogonDate&apos;; E = { $null } }
  Get-ADGroupMember -Identity $g -Recursive |
    Where-Object { $_.objectClass -eq &apos;user&apos; } |
    Get-ADUser -Properties LastLogonDate, Enabled |
    Select-Object @{ N = &apos;Group&apos;; E = { $g } }, @{ N = &apos;Kind&apos;; E = { &apos;effective&apos; } },
      objectClass, SamAccountName, Name, Enabled, LastLogonDate
}
$rows | Export-Csv -Path &quot;$outDir\\privileged.csv&quot; -NoTypeInformation -Encoding UTF8

# 2. Current values of the domain password policy and the fine-grained password policies (PSOs)
Get-ADDefaultDomainPasswordPolicy |
  Export-Csv -Path &quot;$outDir\\domain-password-policy.csv&quot; -NoTypeInformation -Encoding UTF8
Get-ADFineGrainedPasswordPolicy -Filter * |
  Select-Object Name, Precedence, MinPasswordLength, MaxPasswordAge, ComplexityEnabled,
    @{ N = &apos;AppliesTo&apos;; E = { ($_.AppliesTo | ForEach-Object { $_.ToString() }) -join &apos;; &apos; } } |
  Export-Csv -Path &quot;$outDir\\fine-grained-policy.csv&quot; -NoTypeInformation -Encoding UTF8

# 3. Enabled users with no logon for 90 days or more. Keep rows with an empty LastLogonDate (no record) so they can be reviewed separately
$cutoff = (Get-Date).AddDays(-90)
Get-ADUser -Filter &apos;Enabled -eq $true&apos; -Properties LastLogonDate |
  Where-Object { $null -eq $_.LastLogonDate -or $_.LastLogonDate -lt $cutoff } |
  Select-Object SamAccountName, Name, LastLogonDate |
  Export-Csv -Path &quot;$outDir\\inactive-90d.csv&quot; -NoTypeInformation -Encoding UTF8

# 4. Enabled users whose passwords never expire
Get-ADUser -Filter &apos;PasswordNeverExpires -eq $true -and Enabled -eq $true&apos; -Properties PasswordLastSet |
  Select-Object SamAccountName, Name, PasswordLastSet |
  Export-Csv -Path &quot;$outDir\\password-never-expires.csv&quot; -NoTypeInformation -Encoding UTF8

# 5. Users created in the past 30 days
Get-ADUser -Filter * -Properties whenCreated |
  Where-Object { $_.whenCreated -gt (Get-Date).AddDays(-30) } |
  Select-Object SamAccountName, Name, whenCreated |
  Export-Csv -Path &quot;$outDir\\created-30d.csv&quot; -NoTypeInformation -Encoding UTF8

Get-ChildItem $outDir | Select-Object Name, Length</code></pre>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">The precision of LastLogonDate.</b> <code>LastLogonDate</code> is derived from the <code>lastLogonTimestamp</code> attribute. This attribute is not updated at every logon; it is updated only when the previous value is older than <code>msDS-LogonTimeSyncInterval</code> (the initial update interval is 14 days minus a random amount). That is enough for coarse checks such as &quot;90 days or more,&quot; but not for day-level decisions. An empty row does not necessarily mean &quot;never signed in,&quot; so look at it together with the creation date. When you need the exact last logon, collect <code>lastLogon</code> from each domain controller and take the maximum.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://learn.microsoft.com/en-us/windows/win32/adschema/a-lastlogontimestamp?ref=journal.qualiteg.com" style="color:#0B5C9C;">Lastlogontimestamp attribute</a></div></div>
<!--kg-card-end: html-->
<p>The purpose and rights of service accounts cannot be determined from the list of non-expiring passwords alone. Check them against the service configuration (which service on which server runs under that account) and the rights delegated on OUs and servers.</p><p>The exported CSV only becomes an audit when you diff it against last month&apos;s and record, for each changed row, &quot;is this an approved change?&quot; and &quot;who will fix it?&quot; Not stopping at producing the lists is what matters most for keeping it going.</p><h4 id="how-much-log-data-to-keep-and-how">How much log data to keep, and how</h4><p>Log retention is the foundation of auditing. The retention period is determined by the regulations that apply to your industry and your contracts with business partners, so first confirm with your legal or audit department which regulations apply to your company, and write the period and its basis into your internal rules. A period with no basis, such as &quot;one year for now,&quot; turns out to be either too short or needlessly long when it is actually demanded.</p><p>Once the period is set, the requirements for how to keep the logs are the same regardless of regulation.</p>
<!--kg-card-begin: html-->
<table style="width:100%;border-collapse:collapse;font-size:15px;margin:24px 0;"><thead><tr><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Requirement</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Why</th><th style="background:#0B5C9C;color:#fff;text-align:left;padding:10px 12px;font-weight:600;">Example implementation</th></tr></thead><tbody><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Tamper-proof</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Prevent an intruder from deleting or rewriting the logs</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Write-once (WORM) storage. Compute hashes of log files finalized after compression in a separate job, and record them in a tamper-resistant store</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Searchable</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">During an investigation, narrow down by period and user within minutes</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">A SIEM, or storage indexed by time. Compress in a searchable format</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;white-space:nowrap;font-weight:600;">Provably complete</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Avoid ending up with &quot;there are no logs for this period&quot;</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;">Check arrivals from each machine daily and alert on machines that stopped</td></tr><tr><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;white-space:nowrap;font-weight:600;">Restricted readers</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">The logs themselves can contain personal activity and credentials like those in 7.1.1</td><td style="padding:10px 12px;border-bottom:1px solid #D5DDE6;vertical-align:top;background:#F7FAFD;">Limit read access to the storage to the audit team, and log the reads too</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>If you aggregate logs on a Linux log server, leave rotation and compression to logrotate. Do not compute the tamper-detection hashes inside logrotate. Compute them in a separate job on files whose compression has finished and whose contents are final, and record them in a non-rewritable store separate from the log server. If the hashes sit in ordinary files on the same server, an intruder can rewrite both the logs and the hashes.</p><pre><code class="language-bash"># /etc/logrotate.d/security-logs (handles only rotation and compression)
/var/log/security/*.log {
    daily
    # rotate is the number of rotations, not the number of days kept
    rotate 400
    compress
    delaycompress
    missingok
    notifempty
    create 0600 root root
}</code></pre>
<!--kg-card-begin: html-->
<div style="border:1px solid #D5DDE6;border-left:4px solid #284C6E;background:#fff;border-radius:0 10px 10px 0;padding:12px 18px;margin:24px 0;font-size:15px;"><b style="color:#284C6E;">rotate is not a number of days.</b> <code>rotate</code> is the number of rotations, not the retention period in days. Even with daily rotation, empty days are skipped by <code>notifempty</code>, and days when the server was down do not count. What guarantees the retention period is a mechanism on the storage side, such as the retention settings of WORM storage or object storage.<div style="font-size:13px;color:#4A463F;margin-top:8px;">Sources: <a href="https://linux.die.net/man/8/logrotate?ref=journal.qualiteg.com" style="color:#0B5C9C;">logrotate(8) manual</a></div></div>
<!--kg-card-end: html-->
<h4 id="chapter-summary">Chapter summary</h4>
<!--kg-card-begin: html-->
<figure style="margin:24px 0;"><svg viewbox="0 0 720 240" width="100%" xmlns="http://www.w3.org/2000/svg" font-family="Inter, &apos;Segoe UI&apos;, sans-serif">
  <defs><marker id="arw3" markerwidth="10" markerheight="10" refx="8" refy="5" orient="auto"><path d="M0,0 L10,5 L0,10 z" fill="#15222B"/></marker></defs>
  <rect x="20" y="60" width="200" height="110" rx="12" fill="#0B5C9C"/>
  <text x="120" y="98" text-anchor="middle" fill="#fff" font-size="18" font-weight="700">Prevent</text>
  <text x="120" y="124" text-anchor="middle" fill="#fff" font-size="12">Limit Basic, long passwords</text>
  <text x="120" y="142" text-anchor="middle" fill="#fff" font-size="12">Least privilege, tiers, PAW</text>
  <rect x="260" y="60" width="200" height="110" rx="12" fill="#1184DE"/>
  <text x="360" y="98" text-anchor="middle" fill="#fff" font-size="18" font-weight="700">Detect</text>
  <text x="360" y="124" text-anchor="middle" fill="#fff" font-size="12">Log collection, anomalies</text>
  <text x="360" y="142" text-anchor="middle" fill="#fff" font-size="12">Monthly audits, inventory</text>
  <rect x="500" y="60" width="200" height="110" rx="12" fill="#284C6E"/>
  <text x="600" y="98" text-anchor="middle" fill="#fff" font-size="18" font-weight="700">Respond</text>
  <text x="600" y="124" text-anchor="middle" fill="#fff" font-size="12">Disable accounts, fix rights</text>
  <text x="600" y="142" text-anchor="middle" fill="#fff" font-size="12">Revise policies</text>
  <line x1="222" y1="115" x2="255" y2="115" stroke="#15222B" stroke-width="2" marker-end="url(#arw3)"/>
  <line x1="462" y1="115" x2="495" y2="115" stroke="#15222B" stroke-width="2" marker-end="url(#arw3)"/>
  <path d="M600,172 Q600,215 360,215 Q120,215 120,178" fill="none" stroke="#15222B" stroke-width="2" marker-end="url(#arw3)"/>
  <text x="360" y="40" text-anchor="middle" fill="#15222B" font-size="13">Feed the configuration drift and unneeded rights found in audits back into prevention</text>
</svg><figcaption style="font-size:13px;color:#4A463F;text-align:center;margin-top:6px;">Figure: The cycle of prevention, detection, and response. Audit results feed back into the next round of prevention</figcaption></figure>
<!--kg-card-end: html-->
<p>Security is a cycle of &quot;prevention,&quot; &quot;detection,&quot; and &quot;response.&quot; Harden the entry points in 7.1 (prevention), notice drift and anomalies through the monitoring in 7.2.1 and 7.2.2 and the audits in 7.2.3 (detection), and correct what you find and feed it back into policy (response). As long as these three keep turning, your authentication infrastructure grows stronger over time instead of weaker.</p><p>Next time, in Part 8, we will turn everything so far into concrete configuration examples, and look at how to combine it with AI security tools, from small to large environments.</p>]]></content:encoded></item><item><title><![CDATA[Ubuntu 24.04 GPU Server Freezes on Shutdown. The Culprit Was the HWE Kernel]]></title><description><![CDATA[A GPU inference server on Ubuntu 24.04 froze on shutdown. The previous boot journal showed a page-management anomaly right after a CUDA process got SIGKILL, on HWE kernel 7.0 with NVIDIA 595 open. Switching the kernel and module pair to GA 6.8 fixed it, plus four layers against drifting back.]]></description><link>https://journal.qualiteg.com/ubuntu-2404-hwe-kernel-nvidia-shutdown-hang/</link><guid isPermaLink="false">6ae706d571353502c442d9a0</guid><category><![CDATA[GPU]]></category><category><![CDATA[Daily Dev Tips]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Sun, 01 Nov 2026 09:25:40 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/11/ubuntu-2404-hwe-kernel-nvidia-shutdown-hang-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/11/ubuntu-2404-hwe-kernel-nvidia-shutdown-hang-en.png" alt="Ubuntu 24.04 GPU Server Freezes on Shutdown. The Culprit Was the HWE Kernel"><p>Hello!</p><p>To move a GPU inference server running Ubuntu 24.04 to another rack, I ran <code>shutdown -h now</code>.</p><p>Ten-plus minutes later, the machine still had not powered off.</p><p>SSH got as far as the TCP handshake but never reached a login. The console on the machine showed the last target of the shutdown sequence.</p><p>It still would not go down. In the end I held the power button.</p><p><strong>The culprit was the combination of the HWE kernel 7.0 series and the NVIDIA 595 open driver.</strong></p><p>Right after a process using CUDA was sent SIGKILL, the kernel logged an anomaly in its page bookkeeping. After that, ext4 writeback and several processes stopped making progress, and the shutdown sequence never completed.</p><p>This article walks through, in the order I actually did it, the diagnosis from the previous boot&apos;s journal, the reproduction test in which only the kernel and its matching NVIDIA module pair were switched to the GA 6.8 series, pinning that configuration, and the four layers of defense that keep the machine from drifting back to HWE.</p><p>If you are walking over to a power button for the same symptom, I hope this saves you some time.</p><h2 id="1-what-happened-the-same-trigger-froze-the-machine-twice">1. What happened. The same trigger froze the machine twice</h2><p>Here is the environment.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Item</th><th>Value</th></tr></thead><tbody><tr><td>OS</td><td>Ubuntu 24.04.5 LTS (installed from the Desktop ISO, where the HWE kernel is the default)</td></tr><tr><td>Kernel (at the time)</td><td>7.0.0-42-generic (<code>linux-generic-hwe-24.04</code>)</td></tr><tr><td>NVIDIA</td><td><code>nvidia-driver-595-open</code> 595.99.02 (<code>linux-modules-nvidia-595-open-7.0.0-42-generic</code>, signed by Canonical, Secure Boot enabled)</td></tr><tr><td>GPU</td><td>NVIDIA RTX 2000 Ada &#xD7;2</td></tr><tr><td>Workload</td><td>GPU inference containers on Docker (NVIDIA Container Toolkit), using CUDA and cuDNN</td></tr><tr><td>Kernel (after recovery)</td><td>6.8.0-147-generic (GA, <code>linux-generic</code>)</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The first time was an ordinary <code>shutdown -h now</code>.</p><p>The shutdown sequence progressed, but Docker, NetworkManager and wpa_supplicant would not die even on SIGKILL, and everything stopped at the final stage. SSH behaved differently as time passed. At first it was &quot;Connection closed&quot;: ssh.socket accepted the connection but sshd could not be started. After a while it became &quot;Connection refused&quot;, with only the kernel&apos;s IP stack still alive. Nothing could be done remotely, and I held the power button.</p><p>The second time happened in the middle of the countermeasures.</p><p>While I was stopping the CUDA containers one at a time with <code>docker stop -t 300</code>, a container that did not respond to SIGTERM was sent SIGKILL after 300 seconds, and the same breakage reproduced right away. SSH got through the TCP connection and the banner, but the login that followed, that is, the disk wait in user space, made no progress. The power button again.</p><p>Twice, from the same trigger. This was a reproducible defect, not &quot;a machine having a bad day&quot;.</p><h2 id="2-start-the-diagnosis-with-journalctlb1k">2. Start the diagnosis with journalctl -b -1 -k</h2><p>After the reboot, the kernel log of the previous boot still held the whole sequence.</p><pre><code class="language-bash">journalctl -b -1 -k</code></pre><p><code>journalctl -k</code> only shows the current boot, so the record of the boot that froze is read with <code>-b -1</code>, which selects the previous boot.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig1_hang_flow_en.png" class="kg-image" alt="Ubuntu 24.04 GPU Server Freezes on Shutdown. The Culprit Was the HWE Kernel" loading="lazy" width="2000" height="1257" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig1_hang_flow_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig1_hang_flow_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig1_hang_flow_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig1_hang_flow_en.png 2100w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 1  The shutdown hang unfolded in this order. From the kernel log of the previous boot (diagram by Qualiteg)</span></figcaption></figure><p>About 10 seconds after shutdown started, dockerd sent SIGKILL to several of the GPU containers because they &quot;did not exit within 10 s&quot;. Up to this point, that is Docker working as designed.</p><p>The problem came right after. In the teardown path of the python process using CUDA, <code>do_exit &#x2192; uvm_release [nvidia_uvm] &#x2192; exit_mmap &#x2192; zap_present_ptes</code>, these lines appeared in the log (excerpt, with timestamps and the hostname removed).</p><pre><code class="language-text">WARNING: fs/ext4/inode.c ext4_dirty_folio
BUG: Bad rss-counter state (dockerd)
BUG: Bad page map in process python
Bad page state
rcu: INFO: rcu_preempt detected expedited stalls
Processes still around after SIGKILL</code></pre><p>The anomaly was recorded on pages belonging to the cuDNN shared library <code>libcudnn_engines_precompiled.so.9</code>. The mapcount was -1, meaning the bookkeeping of page mapping references had broken down.</p><p>From then on, the kworker responsible for ext4 writeback kept spinning, and dockerd, runc, wpa_supplicant and the CUDA threads entered D state. SIGKILL has no effect on a process in D state. systemd went through stop-sigterm timeout, SIGKILL, &quot;Processes still around after SIGKILL&quot; and final SIGKILL for docker.service, and then stalled at the final stage of shutdown.</p><p>What the log preserves is this order of events. I concluded that this chain of anomalies led to the shutdown hang.</p><p>The Tainted flags were <code>G B W O</code>. EDAC and MCE showed only the initialization lines for a non-ECC configuration, with no memory errors recorded.</p><p><strong>A process that survives SIGKILL is stuck inside the kernel.</strong></p><p>A user-space problem goes away with SIGKILL. If it does not, the kernel side is the suspect. Keep this pattern in mind and you will not hesitate the next time it happens.</p><h2 id="3-the-other-machines-are-fine-came-down-to-version-differences">3. &quot;The other machines are fine&quot; came down to version differences</h2><p>Three machines of the same model, built with the same procedure, had been running for weeks. Comparing them answered the question &quot;why only this one&quot; in two parts.</p><p>First, none of the three had ever been through a &quot;shutdown while force-killing GPU containers&quot; up to that point. Their logs held zero docker force-kill records. The trigger itself had never been pulled.</p><p>Second, the problem machine alone had a newer kernel, a newer driver and a newer GPU generation. The three ran 7.0.0-34 and 7.0.0-38 with 595.91.07. The problem machine ran 7.0.0-42 with 595.99.02. The build procedure was the same, but a machine built later gets whatever versions are current. The older versions had disappeared from the archive, and the newer ones went in.</p><p>Here are the conditions side by side.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Item</th><th>Problem machine</th><th>The 3 older-config machines</th></tr></thead><tbody><tr><td>Kernel</td><td>7.0.0-42</td><td>7.0.0-34 on two, 7.0.0-38 on one</td></tr><tr><td>NVIDIA</td><td>595.99.02 open</td><td>595.91.07</td></tr><tr><td>GPU</td><td>NVIDIA RTX 2000 Ada &#xD7;2</td><td>Previous-generation GPU</td></tr><tr><td>Secure Boot</td><td>Enabled</td><td>Enabled on one, disabled on two</td></tr><tr><td>Went through a forced kill of GPU containers?</td><td>Yes (hung twice)</td><td>Two of them did, during the rollout shutdown (Section 7)</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>From here, &quot;a version combination problem&quot; looked far more likely than &quot;a hardware fault in one unit&quot;.</p><h2 id="4-recovery-and-prevention-switch-the-kernel-configuration-to-the-ga-68-series">4. Recovery and prevention. Switch the kernel configuration to the GA 6.8 series</h2><p>There were three candidates: switch the driver to the proprietary build, go back to the GA kernel, or run a memory test.</p><p>Change several things at once and you lose track of what worked. So I first moved only the kernel from 7.0.0-42 back to GA 6.8.0-147 and kept the 595.99.02 open driver as it was.</p><h3 id="4-1-install-the-ga-metapackages-and-the-signed-nvidia-modules-for-68">4-1. Install the GA metapackages and the signed NVIDIA modules for 6.8</h3><p>With Secure Boot enabled I did not use DKMS; I used the prebuilt modules signed by Canonical. First, check that modules for 6.8 are published. At the time they existed for 595-open, 595, 580, 580-open and 615-open, and not for the 590 series.</p><pre><code class="language-bash">apt-cache policy linux-modules-nvidia-595-open-6.8.0-147-generic
apt-get install linux-generic linux-image-generic linux-headers-generic linux-modules-nvidia-595-open-generic</code></pre><h3 id="4-2-pin-the-grub-default-to-the-actual-68-entry-name-and-reboot">4-2. Pin the GRUB default to the actual 6.8 entry name and reboot</h3><p>The 7.0 side stays installed until the tests pass. With a way back secured, only the default is changed to 6.8.</p><p>First, check the menu names GRUB actually generates.<code>GRUB_DEFAULT</code> has no effect unless it matches this output exactly.</p><pre><code class="language-bash">grep -E &quot;submenu |menuentry &quot; /boot/grub/grub.cfg</code></pre><p>Next, in <code>/etc/default/grub</code>, rewrite the <code>GRUB_DEFAULT</code> line with the actual entry name you just confirmed.</p><pre><code class="language-text">GRUB_DEFAULT=&quot;Advanced options for Ubuntu&gt;Ubuntu, with Linux 6.8.0-147-generic&quot;</code></pre><p>After the edit, apply the configuration and confirm that the target kernel has the NVIDIA modules before rebooting.</p><pre><code class="language-bash">cp /etc/default/grub /etc/default/grub.bak
grep &apos;^GRUB_DEFAULT=&apos; /etc/default/grub
update-grub
modinfo -k 6.8.0-147-generic -F filename nvidia
systemctl reboot</code></pre><p><code>modinfo</code> returns a path such as <code>/lib/modules/6.8.0-147-generic/kernel/nvidia-595-open/nvidia.ko</code>, the modules are present for the target kernel. If it reports ERROR, install the modules for that version before rebooting. In my case, I confirmed that nvidia.ko and the other modules were placed at that path right after the <code>apt-get install</code> in 4-1, and then rebooted. This pre-reboot check was built into the verification script for the rollout described later, and all four machines passed it.</p><h3 id="4-3-three-rounds-of-start-and-kill9-each-exited-within-a-second-with-zero-warnings">4-3. Three rounds of start and kill -9. Each exited within a second, with zero warnings</h3><p>In a conda environment on the host (the CUDA build of PyTorch), I started a python process running cuDNN convolutions on both GPUs and killed it with <code>kill -9</code>, three rounds in total. This deliberately reproduces the very trigger that was hit on the 7.0 series.</p><p>All three rounds exited within a second. Zero kernel warnings, zero processes in D state, GPU memory normal, and no effect on the running containers.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig2_variable_matrix_en.png" class="kg-image" alt="Ubuntu 24.04 GPU Server Freezes on Shutdown. The Culprit Was the HWE Kernel" loading="lazy" width="2000" height="1067" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig2_variable_matrix_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig2_variable_matrix_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig2_variable_matrix_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig2_variable_matrix_en.png 2100w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 2  Driver and hardware unchanged; switching back only the kernel and module pair made the hang disappear (diagram by Qualiteg)</span></figcaption></figure><p>With the hardware and the driver untouched, going back only on the kernel made the same SIGKILL test run clean. Changing the kernel configuration was effective at avoiding this failure, which strongly supports the combination of 7.0.0-42 and 595.99.02 open as the cause.</p><p>Note that changing the kernel also swaps the NVIDIA kernel modules for the 6.8 builds. What was switched is the pair of the kernel and its matching NVIDIA kernel modules.</p><h2 id="5-four-layers-to-keep-from-drifting-back-to-70">5. Four layers to keep from drifting back to 7.0</h2><p>Stop here, and the next <code>apt upgrade</code> brings HWE right back. So I put the defense in four layers.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig3_four_layers_en.png" class="kg-image" alt="Ubuntu 24.04 GPU Server Freezes on Shutdown. The Culprit Was the HWE Kernel" loading="lazy" width="2000" height="1143" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig3_four_layers_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig3_four_layers_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig3_four_layers_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig3_four_layers_en.png 2100w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 3  Four layers of defense against an accidental return to 7.0 (diagram by Qualiteg)</span></figcaption></figure><p>Layer 1 removes the HWE metapackages. List the installed HWE metapackages, purge them, and <code>apt upgrade</code> no longer pulls in 7.0.</p><pre><code class="language-bash">dpkg -l | awk &apos;/^ii/ &amp;&amp; $2 ~ /^linux-.*-hwe-[0-9.]+$/ {print $2}&apos;
apt-get purge linux-generic-hwe-24.04 linux-image-generic-hwe-24.04 linux-headers-generic-hwe-24.04 linux-modules-nvidia-595-open-generic-hwe-24.04</code></pre><p>Layer 2 holds the four GA metapackages. The kernel and the NVIDIA modules are then upgraded by a person, as a pair.</p><pre><code class="language-bash">apt-mark hold linux-generic linux-image-generic linux-headers-generic linux-modules-nvidia-595-open-generic</code></pre><p>What is pinned here is not &quot;the 6.8 series&quot; as a line but the 6.8.0-147 ABI whose behavior was verified. GA 6.8 is a continuously maintained kernel, so this hold also stops the GA security updates. When applying an update, upgrade the kernel and the matching NVIDIA modules together, run the same SIGKILL test and reboot test as in 4-3, and then hold again.</p><p>Layer 3 purges the 7.0 series packages themselves. Rather than deleting blindly with a wildcard, list the installed 7.0 series packages and purge those. The entries disappear from the GRUB menu, and once you set <code>GRUB_DEFAULT=0</code> back and run <code>update-grub</code>, 6.8 is at the top.</p><pre><code class="language-bash">dpkg -l | awk &apos;/^ii/ &amp;&amp; $2 ~ /7\.0\.0-/ {print $2}&apos;
apt-get purge &lt;the 7.0 series packages listed above&gt;
apt-get autoremove</code></pre><p>Layer 4 is an apt pin. Even an explicit <code>apt install linux-generic-hwe-24.04</code> ends with &quot;has no installation candidate&quot;.</p><pre><code class="language-text"># /etc/apt/preferences.d/qualiteg-no-hwe-kernel
Package: linux-generic-hwe-24.04 linux-image-generic-hwe-24.04 linux-headers-generic-hwe-24.04 linux-modules-nvidia-*-hwe-24.04 linux-tools-generic-hwe-24.04
Pin: release *
Pin-Priority: -1

Package: linux-image-7.* linux-modules-7.* linux-modules-extra-7.* linux-headers-7.* linux-tools-7.* linux-modules-nvidia-*-7.* linux-objects-nvidia-*-7.* linux-signatures-nvidia-7.* linux-hwe-7.*
Pin: release *
Pin-Priority: -1</code></pre><p>Checking that it works takes one command. If Candidate is <code>(none)</code>, it worked.</p><pre><code class="language-bash">apt-cache policy linux-generic-hwe-24.04</code></pre><p>This setting went into all four machines. On all four, Candidate became <code>(none)</code> and an explicit <code>apt install linux-generic-hwe-24.04</code> stopped with &quot;has no installation candidate&quot;. That is what I measured.</p><p>The apt_preferences(5) man page shows <code>Pin: version *</code> as the way to say &quot;any version&quot; of a specific package. The form above works on my machines; to follow the man page, change the two <code>Pin:</code> lines to <code>Pin: version *</code>.</p><p>The state with all four layers in place can be checked with the following commands.<code>uname -r</code> is checked after rebooting into 6.8.</p><pre><code class="language-bash">uname -r
dpkg -l | grep -E &apos;linux-(image|headers|modules).*7\.0&apos;
apt-mark showhold
apt-cache policy linux-generic-hwe-24.04</code></pre><p>If any one layer is lost, the remaining layers still stop the slide back. I wanted to be able to answer &quot;yes&quot; when asked whether it was double- and triple-protected.</p><h2 id="6-rebooted-with-containers-running-back-unattended-in-59-seconds">6. Rebooted with containers running. Back unattended in 59 seconds</h2><p>To finish, I ran <code>systemctl reboot</code> with the containers still running.</p><p>The shutdown sequence took 29 seconds, and SSH was back 59 seconds after the command, with nobody at the machine. Later, when the same machine was taken down with <code>shutdown -h now</code>, the shutdown sequence took 14 seconds to power off.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/11/fig4_stop_times_en.png" class="kg-image" alt="Ubuntu 24.04 GPU Server Freezes on Shutdown. The Culprit Was the HWE Kernel" loading="lazy" width="2000" height="952" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/11/fig4_stop_times_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/11/fig4_stop_times_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/11/fig4_stop_times_en.png 1600w, https://journal.qualiteg.com/content/images/2026/11/fig4_stop_times_en.png 2100w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 4  Time taken by the shutdown. When it happened the machine never powered off after ten-plus minutes; after the fix, the shutdown stage of a reboot took 29 seconds (SSH back in 59 seconds) and a power-off took 14 seconds (diagram by Qualiteg)</span></figcaption></figure><p>The gap from &quot;ten-plus minutes, then the power button&quot; is itself the evidence of recovery.</p><h2 id="7-a-trap-hit-during-the-rollout-a-held-hwe-metapackage-gets-in-the-way">7. A trap hit during the rollout. A held HWE metapackage gets in the way</h2><p>The same countermeasures went into the remaining three machines. Same procedure, but with one trap.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Trap</th><th>Fix</th></tr></thead><tbody><tr><td>The held HWE metapackage <code>linux-modules-nvidia-595-open-generic-hwe-24.04</code> blocks installing the 595 modules for GA. apt stops with &quot;Depends: linux-modules-nvidia-595-open-7.0.0-38-generic (= &#x2026;) but it is not going to be installed&quot;</td><td><code>apt-mark unhold</code> the two HWE metapackages first, then install. As a side effect a newer 7.0 comes in as well, but pin the GRUB default to the actual 6.8 entry name, reboot, and purge the 7.x series together after verification</td></tr><tr><td><code>journalctl -k</code> covers the current boot only</td><td>Read the boot that froze with <code>journalctl -b -1 -k</code>. It cannot be used for a comparison such as &quot;zero Bad page across all boots&quot;</td></tr><tr><td>With a worker that ignores SIGTERM, every shutdown goes through the SIGKILL path</td><td>On my machines I set <code>shutdown-timeout</code> to 120 seconds for the Docker daemon&apos;s shutdown (daemon.json) and <code>TimeoutStopSec=180s</code> for docker.service on the systemd side. The per-container SIGTERM grace period is separate and is managed with <code>default-stop-timeout</code> or the container&apos;s own stop timeout. The real fix is for the application to handle SIGTERM</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The rollout&apos;s own shutdown work doubled as a test of the old configuration. Before moving to the GA 6.8 series, that is, still on 7.0.0-34 and 595.91.07, I stopped the containers with <code>docker stop</code>. On two machines, containers that ignored SIGTERM were sent SIGKILL after the 120-second grace period, with zero kernel warnings and no hang. On the remaining machine (7.0.0-38), every container stopped on SIGTERM, so the SIGKILL path was never taken.</p><p>All three were then moved to the GA 6.8 series. The same &quot;start and kill -9, three rounds&quot; test as in 4-3 produced zero warnings, and a reboot with the containers running took 14 to 20 seconds to shut down.</p><p>On the pre-migration combination of 7.0.0-34 and 595.91.07, sending SIGKILL to a container that ignored SIGTERM produced no kernel warning. I take this to be behavior specific to the 7.0.0-42 and 595.99.02 combination. The three machines each had pending updates that would have put them on that very combination at the next <code>apt upgrade</code> and reboot, so pinning was the right call.</p><h2 id="8-what-changed-in-our-operations">8. What changed in our operations</h2><p>This incident changed five of our standards for GPU servers.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Change</th><th>Why</th></tr></thead><tbody><tr><td>Stop the CUDA containers gracefully before a shutdown or reboot</td><td>The SIGKILL path was the trigger. Containers that do not handle SIGTERM get fixed on the application side. Extending the grace period at the OS level only treats the symptom</td></tr><tr><td>GPU servers built from the Desktop ISO get a &quot;revert to GA&quot; step in the build procedure</td><td>The Desktop ISO defaults to HWE. For server use, the GA 6.8 series is sufficient</td></tr><tr><td>Check the versions that will be installed with <code>apt-cache policy</code> first, record them as the standard, and pin them with hold</td><td>The same procedure yields different versions depending on when the machine is built</td></tr><tr><td>Review GA series updates on a schedule</td><td>The hold pins the 6.8.0-147 ABI, so it also stops security updates. When applying one, upgrade the kernel and the NVIDIA modules together, run the same tests, and pin again</td></tr><tr><td>Add a hardware watchdog and automatic reboot on panic</td><td>Being unable to recover remotely when a machine freezes is a real risk. That is covered in a separate article</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The three lessons from this incident: when you see a process that survives SIGKILL, suspect the kernel; switch only the kernel and module pair to confirm the effect; and pin the configuration that worked with four layers.</p><p>Next time, I will write about adding a hardware watchdog and automatic reboot on panic, so that a frozen machine no longer means a walk to the power button.</p><p>See you in the next article!</p><h2 id="references">References</h2><ul><li>Ubuntu, &quot;Kernel lifecycle&quot; (primary source; how the GA and HWE kernels are positioned) <a href="https://ubuntu.com/kernel/lifecycle?ref=journal.qualiteg.com" rel="noreferrer">https://ubuntu.com/kernel/lifecycle</a></li><li>Ubuntu manpage, &quot;apt_preferences(5)&quot; (primary source; Pin-Priority semantics) <a href="https://manpages.ubuntu.com/manpages/noble/man5/apt_preferences.5.html?ref=journal.qualiteg.com" rel="noreferrer">https://manpages.ubuntu.com/manpages/noble/man5/apt_preferences.5.html</a></li><li>Ubuntu manpage, &quot;journalctl(1)&quot; (primary source; the <code>-b</code> option) <a href="https://manpages.ubuntu.com/manpages/noble/man1/journalctl.1.html?ref=journal.qualiteg.com" rel="noreferrer">https://manpages.ubuntu.com/manpages/noble/man1/journalctl.1.html</a></li><li>Our article: Ubuntu&apos;s Automatic Updates Bumped the Kernel Alone, and the GPU Vanished After a Reboot <a href="https://journal.qualiteg.com/ubuntu-kernel-update-nvidia-driver-lost/" rel="noreferrer">https://journal.qualiteg.com/ubuntu-kernel-update-nvidia-driver-lost/</a></li></ul>]]></content:encoded></item><item><title><![CDATA[[AI×CAD] Part 6: Show What Changed in 3D — Compare and Share STEP and STL Revisions in Your Browser]]></title><description><![CDATA[A moved sensor, larger mounting holes, and added or removed parts: use an interactive CADAS comparison to see how design changes can be shared with procurement and assembly teams.]]></description><link>https://journal.qualiteg.com/ai-cad-revision-comparison-part6/</link><guid isPermaLink="false">6a9ead5047721380cb5d2aa3</guid><category><![CDATA[3D CAD]]></category><dc:creator><![CDATA[Qualiteg Consulting]]></dc:creator><pubDate>Tue, 27 Oct 2026 19:57:45 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/10/ai-cad-step-viewer-part6-en-1.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/10/ai-cad-step-viewer-part6-en-1.png" alt="[AI&#xD7;CAD] Part 6: Show What Changed in 3D &#x2014; Compare and Share STEP and STL Revisions in Your Browser"><p>Hello!</p><p>&quot;Here is the revised STEP file.&quot; A separate email explains which parts changed. You read the drawing&apos;s revision notes, open the previous model, and switch between views to find the differences. Does this sound like your design-change workflow?</p><p>A larger mounting hole, a moved sensor, an added stiffener: a small change for the designer can mean different parts to order or a different mounting position for assembly. If you share only the finished revision, the recipient has to start by finding those differences.</p><p>In this article, we use the free, registration-free <a href="https://cadas-ai.com/?hl=en&amp;ref=journal.qualiteg.com">CADAS</a> to overlay and compare the before-and-after geometry. Part 6 is about <strong>sharing what changed in a form people outside design can inspect for themselves</strong>.</p>
<!--kg-card-begin: html-->
<style>.cadas-article-navigation{border:1px solid #c9dcec;border-radius:8px;padding:18px 20px;margin:0 0 28px;background:#f5f9fc;font-size:16px;line-height:1.65}.cadas-article-navigation p{margin:0 0 8px;font-size:18px}.cadas-article-navigation ol{margin:0;padding-left:22px}.cadas-article-navigation li{margin:3px 0;break-inside:avoid}.cadas-article-navigation span{color:#586879}.cadas-article-navigation .article-contents{margin-top:16px;padding-top:14px;border-top:1px solid #d8e4ed}@media(min-width:700px){.cadas-article-navigation ol{columns:2;column-gap:28px}}</style><div class="cadas-article-navigation" data-cadas-navigation="series-1-7"><nav aria-label="Series contents"><p><strong>Series contents: AI &#xD7; CAD &amp; Design Information (Parts 1&#x2013;7)</strong></p><ol><li><a href="https://journal.qualiteg.com/ai-cad-step-viewer-part1/">Part 1: View &amp; share 3D</a></li><li><a href="https://journal.qualiteg.com/ai-cad-feature-recognition-part2/">Part 2: Find holes &amp; counterbores</a></li><li><a href="https://journal.qualiteg.com/ai-cad-mold-dfm-part3/">Part 3: Check mold DFM</a></li><li><a href="https://journal.qualiteg.com/ai-cad-ai-design-review-part4/">Part 4: Review designs with AI</a></li><li><a href="https://journal.qualiteg.com/ai-cad-design-knowledge-part5/">Part 5: Preserve design decisions</a></li><li><strong><a href="https://journal.qualiteg.com/ai-cad-revision-comparison-part6/" aria-current="page">Part 6: Compare design revisions</a></strong></li><li>Part 7: Check motion &amp; interference <span>(Coming soon)</span></li></ol></nav><nav class="article-contents" aria-label="In this article"><p><strong>In this article</strong></p><ol><li><a href="#start-by-looking-at-where-the-sensor-moved">Start by looking at where the sensor moved</a></li><li><a href="#still-25-parts-but-four-changes">Still 25 parts, but four changes</a></li><li><a href="#for-your-own-step-or-stl-files-open-the-earlier-revision-first">For your own STEP or STL files, open the earlier revision first</a></li><li><a href="#move-from-finding-differences-to-deciding-who-should-check-what">Move from finding differences to deciding who should check what</a></li><li><a href="#share-both-revisions-as-one-package">Share both revisions as one package</a></li><li><a href="#colors-show-geometry-differences-people-record-the-reasons">Colors show geometry differences; people record the reasons</a></li><li><a href="#start-by-explaining-one-change">Start by explaining one change</a></li></ol></nav></div>
<!--kg-card-end: html-->
<figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/cadas-series-1-7-en.png" class="kg-image" alt="[AI&#xD7;CAD] Part 6: Show What Changed in 3D &#x2014; Compare and Share STEP and STL Revisions in Your Browser" loading="lazy" width="1672" height="941" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/09/cadas-series-1-7-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/09/cadas-series-1-7-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/09/cadas-series-1-7-en.png 1600w, https://journal.qualiteg.com/content/images/2026/09/cadas-series-1-7-en.png 1672w" sizes="(min-width: 720px) 720px"><figcaption>AI &#xD7; CAD &amp; Design Information: series overview, Parts 1&#x2013;7</figcaption></figure><h2 id="start-by-looking-at-where-the-sensor-moved">Start by looking at where the sensor moved</h2><p>The demo below compares revisions A and B of the CADAS linear-stage sample. It is interactive 3D, not a static image. Select &quot;Moved &#xB7; limit-sensor&quot; to focus on the sensor before and after the change. Orange shows the earlier revision; green shows the later revision.</p><figure class="kg-card kg-embed-card"><iframe src="https://cadas-ai.com/e/aa2e62ae39d7f9ef15c4d120353b7ed0?hl=en" width="100%" height="600" style="border:1px solid #ccc;border-radius:8px" allowfullscreen loading="lazy" title="3D comparison of linear-stage revisions A and B"></iframe></figure><p>Switch between Before and After to see the new position. The list reports a center movement of 15.000 / 0.000 / 0.000mm. In this sample, the sensor moved 15mm along X. You can inspect both the difference and its location in the browser.</p><p>On a small screen, <a href="https://cadas-ai.com/e/aa2e62ae39d7f9ef15c4d120353b7ed0?hl=en&amp;ref=journal.qualiteg.com">open the comparison demo directly</a> for more room to work.</p><h2 id="still-25-parts-but-four-changes">Still 25 parts, but four changes</h2><p>The sample combines guide rails, a screw shaft, a moving carriage, a motor, a sensor, and other components. Both revisions have 25 parts. Equal part counts do not mean identical content.</p><p>To validate the comparison, we deliberately made four changes in revision B. Dimensions in the following table are the sample&apos;s design values. CADAS did not infer the purpose of the hole diameter or automatically generate this explanation.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Part</th><th>Change made in the sample</th><th>Comparison result</th></tr></thead><tbody>
<tr><td>Mounting plate: tooling-carriage</td><td>Mounting-hole diameter changed from 6.6mm to 8.5mm</td><td>Mesh changed</td></tr>
<tr><td>Sensor: limit-sensor</td><td>Moved 15mm along X</td><td>Moved</td></tr>
<tr><td>Stiffener: rear-stiffener</td><td>Added in revision B</td><td>Added</td></tr>
<tr><td>Cable clip: cable-clip</td><td>Removed in revision B</td><td>Removed</td></tr>
</tbody></table>
<!--kg-card-end: html-->
<p>The actual comparison reported 22 matching meshes, one moved part, one mesh change, one added part, one removed part, and zero matches needing review. Because additions and removals are counted separately, the list contains 26 parts across the two revisions. Revision B itself still has 25 parts.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/part6-sensor-comparison-en.jpg" class="kg-image" alt="[AI&#xD7;CAD] Part 6: Show What Changed in 3D &#x2014; Compare and Share STEP and STL Revisions in Your Browser" loading="lazy" width="2000" height="912" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/09/part6-sensor-comparison-en.jpg 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/09/part6-sensor-comparison-en.jpg 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/09/part6-sensor-comparison-en.jpg 1600w, https://journal.qualiteg.com/content/images/size/w2400/2026/09/part6-sensor-comparison-en.jpg 2400w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 1: An actual comparison with the moved sensor selected. Inspect the geometry and displacement together.</span></figcaption></figure><p>Select the mounting plate and inspect the area around its holes for different overlay colors. Then select the added stiffener and removed clip. Filtering the subject by change type helps you decide what to inspect before comparing all 25 parts individually.</p><h2 id="for-your-own-step-or-stl-files-open-the-earlier-revision-first">For your own STEP or STL files, open the earlier revision first</h2><p>Load the earlier STEP or STL file in CADAS, then choose the revised file through File &gt; Compare revisions. The comparison window shows both filenames and the result for each part.</p><p>Export both files with the same origin and orientation to make comparison easier. CADAS compares geometry in the coordinates stored in the source files. A pose temporarily adjusted with the rotation tool is not treated as a design revision.</p><p>STEP can retain an assembly&apos;s part structure. STL is also supported, useful for comparing shapes exported for 3D printing. Choose units when opening STL. One STL is treated as one part, so STEP with assembly structure makes it easier to narrow down differences between individual components.</p><p>Normal loading and comparison run in the browser. You do not need to issue a share link to inspect differences in your own data.</p><h2 id="move-from-finding-differences-to-deciding-who-should-check-what">Move from finding differences to deciding who should check what</h2><p>Once a difference is visible, the next questions concern the work. If a mounting hole grew, do its mating part, fastener, or machining instructions need review? If a sensor moved, how did its detection position or relationship to nearby components change? If a stiffener was added, is it reflected in purchasing and assembly instructions?</p><p>These are examples of questions to ask when reviewing changes. The responsible people use the differences shown by CADAS to decide what their own work needs checked.</p><p>Designers can point to the changed part. Procurement can start with additions and removals. Assembly staff can rotate the changed location into a useful view. Browser-based comparison lets everyone discuss the same revision before learning CAD editing operations.</p><h2 id="share-both-revisions-as-one-package">Share both revisions as one package</h2><p>Export a share package (.cadas) while the comparison is active to bundle both models. The recipient can reopen the comparison in CADAS, reducing the need to explain which separate attachment contains the earlier revision.</p><p>For models that may be made public, use Embed / share comparison to include the current view state. Both revisions are uploaded, and CADAS issues a share URL and an iframe for embedding in a blog. This article&apos;s demo was made that way.</p><p>For parts that cannot be public, create a .cadas file locally and send it through an approved company channel. In either case, include the model revisions and what you want checked so the recipient can start with a clear purpose.</p><h2 id="colors-show-geometry-differences-people-record-the-reasons">Colors show geometry differences; people record the reasons</h2><p>CADAS compares meshes normalized in 0.001mm increments. This is not a measurement-accuracy specification or a manufacturing-tolerance guarantee. Changes in STEP export software or tessellation can produce differences even without an intended design change.</p><p>Use the list to narrow down the inspection. For a match marked as needing review, confirm that the two entries should be treated as the same part. Verify final hole dimensions and tolerances against the original CAD information or drawings.</p><p>The colors tell you that a shape or position differs. Ask the designer why it changed and record the answer. Combine the decision records from Part 5 with this before-and-after comparison, and a successor can read the reason while inspecting the change.</p><h2 id="start-by-explaining-one-change">Start by explaining one change</h2><p><a href="https://cadas-ai.com/?hl=en&amp;ref=journal.qualiteg.com">CADAS</a> is ready to try: choose File &gt; Comparison sample (linear stage A &#x2192; B). Start with the sensor. Switch between before and after, see where it moved, and explain that change to a colleague. Having the recipient inspect the same location makes the value of 3D sharing concrete.</p><p>For your own organization, begin with one pair of revisions. If checking changes takes too long, or handoffs from design to procurement miss details, we can examine how you store the data and pass it on. We support investigation, proof of concept, and implementation to integrate comparison results into internal review and change-management workflows.</p><p><a href="https://qualiteg.com/consulting/technology?hl=en&amp;ref=journal.qualiteg.com#ai-cad">Talk to Qualiteg about AI &#xD7; CAD and Design Information Consulting</a></p><p>Next, we look at interference that a stationary assembly view cannot reveal. We will move a two-axis pan/tilt assembly in the browser and share the pose and motion range being checked. See you in the next article!</p>]]></content:encoded></item><item><title><![CDATA[Building a Coding Agent from Scratch, Part 5: Making Local LLMs First-Class Citizens. Getting Two 16GB GPUs to Build a Web System Unattended]]></title><description><![CDATA[Starting on a single RTX A5000 (24GB), seven local models were all wiped out. Configuration and pitfalls on the way to two RTX A4000 (16GB) cards running Qwen3.6-35B-A3B in AWQ 4-bit with an fp8 KV cache and a 98k context, building an authenticated web system three times in a row at 93.8 to 96.5.]]></description><link>https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/</link><guid isPermaLink="false">6ab38de82ead0f114b6f0974</guid><category><![CDATA[AI Agents]]></category><category><![CDATA[LLM]]></category><category><![CDATA[IT & AI Technology]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Sun, 25 Oct 2026 05:11:14 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/10/build-coding-agent-from-scratch-part5-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/10/build-coding-agent-from-scratch-part5-en.png" alt="Building a Coding Agent from Scratch, Part 5: Making Local LLMs First-Class Citizens. Getting Two 16GB GPUs to Build a Web System Unattended"><p>Hello!</p><p>Every experiment in this series runs on local LLMs on our own GPUs.</p><p>What we started with was a single RTX A5000 (24GB) card.</p><p>We tried seven models that fit on it and had them build a task-management web app with authentication. The result: <strong>all seven disqualified</strong>.</p><p>Keeps writing things that won&apos;t start. Installs forbidden external packages. Gets the database API wrong every time. Output collapses into the same word repeated.</p><p>To put the conclusion first, we now have exactly one local configuration confirmed on our machines that can &quot;build a web system from scratch, unattended.&quot;</p><p>It is <strong>two RTX A4000 (16GB) cards running Qwen3.6-35B-A3B (MoE, 3B active) in AWQ 4-bit, with an 8-bit KV cache to secure a 98k context</strong>.</p><p>Three runs of the same task: 94.9, 93.8, 96.5. But by the time this configuration was confirmed, we had fixed 64 defects on the harness side. Changing the model alone did not get us there.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig05_1_two_gpu-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 5: Making Local LLMs First-Class Citizens. Getting Two 16GB GPUs to Build a Web System Unattended" loading="lazy"><figcaption>Figure 1: Fitting a 98k context on two 16GB cards (diagram: Qualiteg)</figcaption></figure><p>This article is Part 5 of the series &quot;Building a Coding Agent from Scratch.&quot; This time we focus on local LLMs: model quirks, GPU memory, context length, and how to repair broken tool calls, all with measured numbers.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Part</th><th>Theme</th></tr></thead><tbody><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Part 1</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Why &quot;finishing the job&quot; is hard. Separating the turn runner from the stop decider</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">Part 2</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">&quot;Done&quot; means &quot;it runs,&quot; not &quot;it&apos;s written.&quot; The completion gate and how to write a pushback</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">Part 3</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">&quot;It didn&apos;t stop&quot; is not &quot;it made progress.&quot; Count idle turns, and cap them without cutting the thinking</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Part 4</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Autonomy rules are for when nobody is around. If a human is present, stop and ask</a></td></tr><tr><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Part 5 (this post)</a></strong></td><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Making local LLMs first-class citizens. Getting two 16GB GPUs to build a web system</a></strong></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">Part 6</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">How to build the evaluation. Have it build something that runs, auto-grade it, and doubt the grader itself</a></td></tr><tr><td>Part 7</td><td>Logging and regression. Everything is found in the event log. And building with zero dependencies</td></tr></tbody></table>
<!--kg-card-end: html-->
<h2 id="why-a-small-model-and-small-gpus">Why a Small Model and Small GPUs</h2><p>First, let me explain why this combination. What we are aiming for with this agent is a <strong>coding agent that builds a web system all the way to the end on a single machine at hand</strong>. If all we wanted was something that runs on a large cloud model, Parts 1 through 4 would suffice. What this part tackles, on top of that, is making sure that &quot;a small model and small GPUs can do the same thing properly.&quot;</p><p>There are three reasons.</p><p>The first is that there are workplaces where code cannot leave the building. In development that handles customer source code, or data that contracts forbid sending outside the company, you cannot simply hand it to a cloud API. With an agent that runs entirely on an in-house workstation, the problem never arises in the first place.</p><p>The second is the shape of the cost. An autonomous agent runs 40 to 144 turns on a single task, and each time it loads the whole, ever-longer history into the input. It is obvious when you think about it, but with a pay-as-you-go API this repetition piles straight onto the bill. With your own GPUs, the hardware is a fixed cost at the time of purchase, and there is no charge per run. And since these are small GPUs, they are within reach of individuals too.</p><p>The third is that <strong>running a small model exposes every weakness in the harness</strong>. A smart model fills the holes in the harness with its own judgment, so the defects stay hidden. A small model does not fill them. Many of the &quot;64 harness defects fixed&quot; that appear in this part were found because a small model fell through them. Fixes for defects found with a small model are not wasted when you load a large model either, because completion judgment, idle detection, and tool-call validation are mechanisms you need regardless of how smart the model is.</p><p>Settling the hardware on &quot;two 16GB cards&quot; was for the same reason. Not data-center GPUs, but cards you can plug into a workstation under the desk: we wanted to find out what can and cannot be done within that range.</p><p>The RTX A4000 is a single-slot card whose power and cooling needs are met by an ordinary PC. With two of them, this part presents one configuration that can build a web system with authentication all the way through with nobody watching.</p><h2 id="1-first-class-citizen-means-the-core-knows-no-providers-vocabulary">1. &quot;First-Class Citizen&quot; Means the Core Knows No Provider&apos;s Vocabulary</h2><p>First, just one point about design.</p><p>When putting a local LLM on board, the very first thing we decided was that <strong>the agent&apos;s core knows no particular provider&apos;s vocabulary</strong>. The core deals only with three kinds of normalized blocks: &quot;text,&quot; &quot;tool call,&quot; and &quot;tool result.&quot; Cloud APIs and local inference servers each get an adapter that converts to and from their own wire format in both directions.</p><pre><code class="language-text">(pseudocode) The boundary between core and adapters
core  --&gt; ModelClient (abstract)
             |-- adapter for cloud A
             |-- adapter for OpenAI-compatible APIs   &lt;- local inference servers go here
             |-- adapter for cloud B
             `-- fake adapter for tests (no network needed)</code></pre><p>Adding a new provider means adding one adapter class. Not a single line of the core or the tools changes. If you feel the urge to change them, the abstraction is leaking.</p><p>Why did we decide this first? Because local models return &quot;broken responses&quot; more often than cloud models. JSON cut off midway. Thinking mixed into the body. Tool call arguments in a different order. Unless handling of these quirks is confined to the adapters, the core gets polluted with the quirks of specific models.</p><h2 id="2-the-local-models-we-tried-and-one-control">2. The Local Models We Tried, and One Control</h2><p>These are the results of running the same task (a task-management web app with authentication, 12 APIs, SQLite, a UI, and tests). The seven models first tried on a single RTX A5000 (24GB) were all disqualified: the six in the table other than the 35B-A3B, plus the previous-generation Qwen3-30B-A3B, which is not in the table (its habit of installing forbidden external packages never went away). Of those, the 27B, the Coder variant, and Gemma 4 were rerun on two RTX A4000s, while the main candidate, the 35B-A3B, was tried for the first time on two 16GB cards. The table includes the results from both configurations.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig05_2_model_scores_v2-en.png" class="kg-image" alt="Building a Coding Agent from Scratch, Part 5: Making Local LLMs First-Class Citizens. Getting Two 16GB GPUs to Build a Web System Unattended" loading="lazy"><figcaption>Figure 2: Scores by local model on the same task (diagram: Qualiteg)</figcaption></figure>
<!--kg-card-begin: html-->
<table><thead><tr><th>Model</th><th>Quantization / shape</th><th>GPU</th><th>Context</th><th>Score</th><th>Per turn</th><th>In a word</th></tr></thead><tbody><tr><td>Qwen3.6-35B-A3B</td><td>AWQ 4-bit / MoE</td><td>16GB x2</td><td>98k</td><td><strong>94.9 / 93.8 / 96.5</strong></td><td>25 to 36 s</td><td>The main candidate. Reads the fine print of the spec</td></tr><tr><td>Qwen3.6-27B</td><td>int4 / Dense</td><td>16GB x2</td><td>98k</td><td>75.5</td><td>54 s</td><td>Passes, but slow</td></tr><tr><td>Qwen3-Coder-30B-A3B</td><td>AWQ 4-bit / MoE</td><td>16GB x2, 24GB</td><td>98k, 32k</td><td>Disqualified x9</td><td>14 to 24 s</td><td>Fast, but gets the DB API wrong every time</td></tr><tr><td>Gemma 4 26B-A4B</td><td>AWQ 4-bit / MoE</td><td>24GB, 16GB x2</td><td>32k, 98k</td><td>9 / 11.5</td><td>Varies</td><td>Keeps writing things that won&apos;t start</td></tr><tr><td>Devstral-Small-2 24B</td><td>AWQ 4-bit / Dense</td><td>24GB</td><td>32k</td><td>Disqualified</td><td>Varies</td><td>Output collapses into the same word repeated</td></tr><tr><td>Qwen3-14B</td><td>AWQ / Dense (thinking)</td><td>24GB</td><td>32k</td><td>Disqualified</td><td>Varies</td><td>Thinking took up 93% of the history</td></tr><tr><td>Qwen3-4B-Instruct</td><td>None</td><td>24GB</td><td>32k</td><td>Disqualified</td><td>Varies</td><td>Startup and UI work. APIs: 0</td></tr><tr><td>Control: a small paid cloud model</td><td>API</td><td>None</td><td>API side</td><td>89.5</td><td>Not measured</td><td>Completed in 33 turns. Old harness, before the defect fixes</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>One caveat about the cloud control model. This 89.5 is a number from the <strong>old harness</strong> at the stage when 16 harness defects had been fixed. On that same old harness, the local models were wiped out with at best 6.4 of the 35 points allotted to APIs. The local 35B&apos;s three runs (94.9 / 93.8 / 96.5) are numbers from the harness at the stage when 64 defects had been fixed. On the latest harness, after continued fixes brought the total to 105, the same task scored 98.3. Since the versions differ, <strong>this is not a comparison of model quality</strong>. What can be read from it is rather that &quot;loading a smart model alone does not get you there.&quot;</p><h3 id="2-1-the-main-candidate-35b-a3b-reads-the-specification">2-1. The main candidate, 35B-A3B, reads the specification</h3><p>It implements even the default value of <code>status</code> and the sort order of queries that other models drop. On the third run, all 22 API checks passed.</p><p>It has weaknesses, though. It re-saves the same TODO for 11 turns. It tries to write 27KB in one go and gets cut off. It forgets to write the README. Turn counts vary from 40 to 144, but scores stay within &#xB1;1.5 of the average (Part 3).</p><h3 id="2-2-the-coder-variant-is-fast-but-doesnt-know">2-2. The Coder variant is &quot;fast but doesn&apos;t know&quot;</h3><p>The Coder variant is 1.5 to 2 times faster (14 to 24 s/turn), but it got the API of Node.js&apos;s built-in SQLite module wrong in all nine attempts (there is no function <code>open</code>; the correct one is the <code>DatabaseSync</code> class), and never produced something that worked. <strong>A gap in knowledge cannot be filled by the harness.</strong></p><p>Change the shape of the task, though, and a different face appears. Fixing a bug in existing code: 12 turns, 35 seconds, 92.5 points. Adding a feature to existing code: 20 turns, 70 seconds, 96.7. Building a CLI from scratch: 70. A server with background processing: 24.3 (reported done without the worker ever running).</p><p><strong>It splits into four tiers by task shape</strong> (Web 0 / background processing 24 / CLI 70 / fixes and feature additions 92 to 97). Declaring a model &quot;usable&quot; or &quot;unusable&quot; from the score on one task is dangerous; even a weak model is practical for &quot;work where the structure already exists.&quot; Part 6 returns to this as &quot;adding more kinds of tasks.&quot;</p><h3 id="2-3-quantization-degradation-looks-like-a-bad-model">2-3. Quantization degradation looks like &quot;a bad model&quot;</h3><p>The 4-bit version of Devstral collapsed into repeating the same words: &quot;Let me create files are files as files are files...&quot; Looking only at the 33% idle rate, it appears to lack ability, but the cause is the interaction between quantization and long outputs. A different quantization of the same model used up the whole 24GB on weights alone and could not start.</p><p>As written in Part 3, <strong>do not blame the model&apos;s ability for a low score on a run whose idle rate exceeds 30%.</strong> Simply changing the quantization build can change the result.</p><h2 id="3-fitting-a-98k-context-on-two-16gb-cards">3. Fitting a 98k Context on Two 16GB Cards</h2><p>This is the main configuration (Figure 1 at the top).</p><p>The model is cyankiwi&apos;s AWQ 4-bit build. Qwen&apos;s official quantized versions are BF16 and FP8 only, with no 4-bit, so fitting it on two 16GB cards means using a third-party quantization. cyankiwi is an account specializing in quantization that has continuously published AWQ builds of Qwen and Gemma, and the same model is used in NVIDIA&apos;s Jetson AI Lab instructions. The weights are distributed in safetensors format, so loading them requires no custom Python code.</p><pre><code class="language-bash"># Starting the inference server (vLLM 0.28). 16GB x 2, Ubuntu on WSL2
export NCCL_P2P_DISABLE=1 NCCL_SHM_DISABLE=1 NCCL_CUMEM_ENABLE=0 NCCL_IB_DISABLE=1
vllm serve cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
  --tensor-parallel-size 2 --kv-cache-dtype fp8 \
  --max-model-len 98304 --gpu-memory-utilization 0.92 \
  --max-num-batched-tokens 2048 --max-num-seqs 4 \
  --compilation-config &apos;{&quot;cudagraph_capture_sizes&quot;:[1,2,4]}&apos; \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice</code></pre><p>The weights are 4-bit and split across two cards (tensor parallel), the KV cache is 8-bit, and the context is 98,304 tokens. Here is a table of the pitfalls we stepped on along the way.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Pitfall</th><th>What happens</th><th>Fix</th></tr></thead><tbody><tr><td><code>--enforce-eager</code> (disabling CUDA graphs)</td><td>6.8 tokens/s, 63 s/turn. Unusable</td><td>Keep CUDA graphs on but <strong>narrowed</strong>. Concurrency 4, capture sizes 1/2/4. Graph memory fits in 0.51GiB, 30 to 39 tokens/s</td></tr><tr><td>Memory utilization 0.95</td><td>Won&apos;t start (14.77/16GiB free)</td><td>0.92</td></tr><tr><td>0.92 with default CUDA graphs</td><td>&quot;No memory for KV&quot;</td><td>The narrowing above</td></tr><tr><td>Context length 131072</td><td>KV needs 3.0GiB and won&apos;t fit</td><td>98304 (KV 2.42GiB)</td></tr><tr><td>Default PyPI wheel (cu130)</td><td>&quot;driver too old&quot; on our driver (CUDA 12.8)</td><td>Specify the cu129 wheel</td></tr><tr><td>Tensor parallel on WSL2</td><td>NCCL hangs</td><td><code>NCCL_P2P_DISABLE=1 NCCL_SHM_DISABLE=1 NCCL_CUMEM_ENABLE=0 NCCL_IB_DISABLE=1</code></td></tr><tr><td>Triton JIT</td><td>Missing headers</td><td><code>python3-dev</code></td></tr><tr><td>Starting in the background</td><td><code>nohup</code> dies when the WSL session ends</td><td>Create the process from the Windows side and keep a foreground</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The first row mattered most. When memory is short and you disable CUDA graphs, speed drops to one fifth. Instead of disabling them, narrow the concurrency and capture sizes and keep them on. That difference took a turn from 63 seconds to around 30.</p><p>The NCCL problem on WSL2 is covered in detail in an earlier article (see Related Articles).</p><h2 id="4-context-length-matters-not-just-to-fit-but-to-avoid-the-cycle">4. Context Length Matters Not Just to &quot;Fit&quot; but to &quot;Avoid the Cycle&quot;</h2><p>We ran the same 27B model with a 16k context on a single 24GB card and with a 98k context on two 16GB cards.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig05_3_context_cycle-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 5: Making Local LLMs First-Class Citizens. Getting Two 16GB GPUs to Build a Web System Unattended" loading="lazy"><figcaption>Figure 3: A short context creates a re-reading cycle (diagram: Qualiteg)</figcaption></figure><p>At 16k, it entered a compaction cycle re-reading four files 48 times each and was cut off. At 98k it passed with 75.5.</p><p>The mechanism is as described in Part 3. Read results accumulate in the window, compaction drops them, and since they were dropped, it reads again. On the single 24GB card at 32k, compaction ran 6 to 54 times in 300 turns; at 98k, 0 to 5 times. <strong>Context length is not only about &quot;whether the model fits&quot; but about whether the length avoids the re-reading cycle.</strong></p><p>One more unglamorous pitfall. The inference server was listening at 98,304, but the harness side was using the 32,768 from the model catalog. Compaction started at one third. We changed it to read the context length not from the catalog but from <code>/v1/models</code> returned by the server&apos;s <code>max_model_len</code>, and swap it in at startup. The output window is widened to match the window too.</p><h2 id="5-the-thinking-model-pitfall-before-comparing-check-that-both-get-the-same-treatment">5. The Thinking-Model Pitfall: Before Comparing, Check That Both Get the Same Treatment</h2><p>This is from when we loaded a model of the type that emits reasoning.</p><p>The inference server separates thinking from the body into its own field only when <code>--reasoning-parser</code> is specified. Without it, the <code>&lt;think&gt;</code> tags arrive mixed into the body. The first version did not know this, and <strong>thinking took up 93% of the history</strong>. Most of the model&apos;s response was thinking, all of it was appended to the history, and it became the input for the next turn.</p><p>To avoid depending on the server&apos;s configuration, we added a layer on the harness side that strips <code>&lt;think&gt;</code> from the body.</p><p>The important point here is to <strong>check, before comparing, that both sides of the comparison are getting the same treatment</strong>. In this case only the thinking model was being disadvantaged. The conclusion &quot;14B is weak&quot; was a conclusion manufactured by a harness defect.</p><h2 id="6-a-broken-tool-call-cannot-be-detected-by-the-shape-of-its-arguments">6. A Broken Tool Call Cannot Be Detected by the Shape of Its Arguments</h2><p>This is where the handling of local and cloud models differed the most.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig05_4_truncated_call-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 5: Making Local LLMs First-Class Citizens. Getting Two 16GB GPUs to Build a Web System Unattended" loading="lazy"><figcaption>Figure 4: A call cut off at the output limit turns into &quot;invalid arguments&quot; (diagram: Qualiteg)</figcaption></figure><p>One model had a habit of emitting <code>content</code> (the body) first and <code>file_path</code> (the path) afterwards in the arguments of a file write. When it is cut off at the output limit (4096 at the time), the path is lost. So far, expected.</p><p>What we did not expect was that <strong>the inference server&apos;s parser closes the truncated JSON and returns it</strong>. It arrives at the harness as well-formed arguments &quot;without a path.&quot; The harness sends it back as &quot;invalid arguments,&quot; and the model <strong>repeated a write of the same size 18 times</strong>. The fact that it had been cut off was recorded nowhere.</p><p>The fix is to judge &quot;did the whole response hit the limit&quot; not from the shape of the arguments but from <code>finish_reason</code> and the output token count. If it did, write-type tools do not write and instead return &quot;split it up, and put the short arguments first.&quot;</p><p>We made one decision here. <strong>We do not repair arguments cut off mid-value.</strong> Closing them and treating them as success would look like it worked, but it would write the truncated body to the file and fabricate the contents. Return broken as broken, and have the model redo it.</p><p>Two other fixes that helped with local models are worth noting.</p><p>The edit tool falls back through matching modes in order: exact match, then tolerating whitespace differences, then searching by surrounding anchors. Local models wobble on indentation, so with exact match alone the edit failure rate jumps.</p><p>We narrowed the tool count to 19. The implementation we referenced had 42, but a small model with many options gets the selection itself wrong.</p><h2 id="summary-you-need-both-the-model-and-the-harness">Summary: You Need Both the Model and the Harness</h2>
<!--kg-card-begin: html-->
<table><thead><tr><th>What to look at</th><th>What we measured</th></tr></thead><tbody><tr><td>GPU memory</td><td>Two 16GB cards: 4-bit 35B-A3B + fp8 KV + 98k context. Narrow CUDA graphs, don&apos;t disable them</td></tr><tr><td>Context length</td><td>Re-reading cycle at 16k, pass at 98k. Ask the server and swap it in</td></tr><tr><td>Model shape</td><td>MoE 35B (3B active) is the main candidate. Dense 27B passes but is twice as slow. The Coder variant fails on missing knowledge</td></tr><tr><td>Quantization</td><td>Some builds collapse into repeated words. Don&apos;t blame the model for a low score with idle rate over 30%</td></tr><tr><td>Thinking models</td><td>Without the parser setting, thinking mixes into the body. Strip it on the harness side</td></tr><tr><td>Tool calls</td><td>Calls cut off at the output limit arrive well-formed. Judge by whether the limit was hit, and don&apos;t repair truncated arguments</td></tr><tr><td>Task shape</td><td>Same model: Web 0 / background processing 24 / CLI 70 / fixes and additions 92 to 97</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>It was neither &quot;load a smart model and it works&quot; nor &quot;train the harness and even a weak model works.&quot; Both were needed. The two-16GB configuration was confirmed only after 64 harness defects had been fixed.</p><p>Next time: the evaluation rig that produces every number in this series. The mechanism that has the agent build something that runs and grades it automatically, the pass/fail criteria of running five kinds of tasks three times each, and the story of the grader itself being broken for three whole days.</p><p>See you next time!</p><h2 id="related-articles">Related Articles</h2><ul><li><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">Building a Coding Agent from Scratch, Part 3: &quot;It Didn&apos;t Stop&quot; Is Not &quot;It Made Progress&quot;</a></li><li><a href="https://journal.qualiteg.com/wsl2-multi-gpu-nccl-error-vllm/">NCCL error: unhandled cuda error &#x2014; Getting Stuck (and Unstuck) with WSL2 + Multi-GPU + vLLM</a></li><li><a href="https://journal.qualiteg.com/gpu_capability_and_fp8_native_support/">Native FP8 and FP4 Support, and &quot;fp8&quot; Quantization with vLLM</a></li><li><a href="https://journal.qualiteg.com/about_paged_attention_for_llm_serving/">PagedAttention for More Efficient LLM Serving</a></li><li><a href="https://journal.qualiteg.com/jisaku_gpu_pc_00/">Building a GPU PC, Part 0: Starting Our GPU Machine Build Journal</a></li></ul>]]></content:encoded></item><item><title><![CDATA[Building a Coding Agent from Scratch, Part 4: Autonomy Rules Are for When Nobody Is Around. If a Human Is Present, Stop and Ask]]></title><description><![CDATA[The "don't stop" discipline honed in evaluation runs was firing even when a human could answer: building unasked after a question, re-asking a declined confirmation seven times. From 19 defects found by role-playing six kinds of usage, we describe the design for when a human is present.]]></description><link>https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/</link><guid isPermaLink="false">6ab38d942ead0f114b6f096c</guid><category><![CDATA[AI Agents]]></category><category><![CDATA[LLM]]></category><category><![CDATA[IT & AI Technology]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Fri, 23 Oct 2026 05:27:00 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/10/build-coding-agent-from-scratch-part4-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/10/build-coding-agent-from-scratch-part4-en.png" alt="Building a Coding Agent from Scratch, Part 4: Autonomy Rules Are for When Nobody Is Around. If a Human Is Present, Stop and Ask"><p>Hello!</p><p>Around the time our own coding agent had started completing evaluation tasks reliably, we tried using it &quot;as a person&quot; for the first time. We opened the chat screen in an empty directory and typed:</p><p>&quot;Make something.&quot;</p><p>The model&apos;s first response was, &quot;What are you looking for? What kind of thing would you like to build?&quot; That is correct. A vague instruction should be met with a question.</p><p>But the harness judged that as &quot;did nothing,&quot; pushed it back, and the model went ahead and built a Python web app on its own and reported &quot;Done.&quot; I had not answered anything.</p><p>To put the conclusion first, the <strong>&quot;don&apos;t stop&quot; discipline we had honed in unattended evaluation runs was entirely for situations where nobody is around to answer</strong>. If a person is in front of the screen and can answer, the right thing is to stop and ask. Applying the discipline honed in evaluation runs unchanged to interactive situations surfaced as 19 defects in a test where we role-played six kinds of usage.</p><p>This article is Part 4 of the series &quot;Building a Coding Agent from Scratch.&quot; This time we describe the flip side of &quot;don&apos;t stop&quot;: the design for situations where a human is present.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Part</th><th>Theme</th></tr></thead><tbody><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Part 1</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Why &quot;finishing the job&quot; is hard. Separating the turn runner from the stop decider</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">Part 2</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">&quot;Done&quot; means &quot;it runs,&quot; not &quot;it&apos;s written.&quot; The completion gate and how to write a pushback</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">Part 3</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">&quot;It didn&apos;t stop&quot; is not &quot;it made progress.&quot; Count idle turns, and cap them without cutting the thinking</a></td></tr><tr><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Part 4 (this post)</a></strong></td><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Autonomy rules are for when nobody is around. If a human is present, stop and ask</a></strong></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Part 5</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Making local LLMs first-class citizens. Getting two 16GB GPUs to build a web system</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">Part 6</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">How to build the evaluation. Have it build something that runs, auto-grade it, and doubt the grader itself</a></td></tr><tr><td>Part 7</td><td>Logging and regression. Everything is found in the event log. And building with zero dependencies</td></tr></tbody></table>
<!--kg-card-end: html-->
<h2 id="1-defects-that-evaluation-runs-cant-find-wont-be-found-by-evaluation-runs">1. Defects That Evaluation Runs Can&apos;t Find Won&apos;t Be Found by Evaluation Runs</h2><p>Why had not a single one of these defects appeared in the more than 100 evaluation runs up to then?</p><p>An evaluation run hands over a specification, runs unattended, and grades the deliverable. It never gives a vague instruction, nobody declines a confirmation, and it never creates a situation where someone else&apos;s process is holding a port. Defects that appear in evaluation runs only appear in the form of evaluation runs.</p><p>We are a one-person development team, so we cannot have other people try it. So the developer took on other personas and used it. Six of them.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Persona</th><th>What they do</th></tr></thead><tbody><tr><td>The first-timer</td><td>&quot;Make something&quot; in an empty directory. Hits Enter three times in a row. &quot;Thanks&quot;</td></tr><tr><td>The impatient one</td><td>Queues two instructions while it&apos;s running. Stops it midway with Esc. &quot;Continue&quot;</td></tr><tr><td>The maintainer</td><td>&quot;Fix the failing test. Don&apos;t commit&quot;</td></tr><tr><td>The cautious one</td><td>Declines every execution confirmation with <code>n</code>. Switches to read-only and then asks for a write</td></tr><tr><td>The web UI user</td><td>Opens two browser tabs. Closes one. Queues, cancels, stops, answers a question</td></tr><tr><td>The misconfigurer</td><td>Starts on a machine with no API key. Mistypes the model name. Points at a nonexistent working directory</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>We prepared a 1-to-5-minute script per persona and played them in order, and each surfaced 2 to 5 different holes. The rest of this article is what they were.</p><h2 id="2-the-same-situation-has-a-different-right-answer-depending-on-whether-a-human-is-present">2. The Same Situation Has a Different Right Answer Depending on Whether a Human Is Present</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig04_1_with_or_without_human-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 4: Autonomy Rules Are for When Nobody Is Around. If a Human Is Present, Stop and Ask" loading="lazy"><figcaption>Figure 1: The same situation, a different right answer depending on whether a human is present (diagram: Qualiteg)</figcaption></figure><h3 id="2-1-if-a-human-is-present-a-question-stops-and-waits">2-1. If a human is present, a question stops and waits</h3><p>This is the &quot;make something&quot; case from the opening.</p><p>The pseudocode shown in Part 1 is the form after we fixed it through this persona test. Even before the fix, &quot;when the user&apos;s answer is needed&quot; was one of the reasons it may stop. But that judgment only fired when the model explicitly called the question tool, and <strong>a response that merely asked a question in its body was classified as &quot;did nothing&quot;</strong>. In an unattended evaluation run that is correct, because it is a situation where we should push back with &quot;don&apos;t ask; assume and proceed.&quot;</p><p>On the chat screen it is different. If someone can answer, a question on the first turn simply stops and waits for the answer. If nobody is present, push back with &quot;assume and proceed&quot; attached. <strong>We made the judgment on the same response split according to whether a human is present</strong>.</p><h3 id="2-2-a-declined-confirmation-is-not-asked-again-until-the-next-instruction">2-2. A declined confirmation is not asked again until the next instruction</h3><p>The &quot;cautious one&quot; script produced the most defects.</p><p>With execution confirmation enabled, ask it to &quot;run the tests&quot; and answer <code>n</code> to the confirmation. After the refusal, the harness&apos;s advice was &quot;ask the user with the question tool, then retry.&quot; The model then asked a nonsensical question: &quot;Do I need to use the standard profile?&quot;</p><p>We changed the advice to &quot;if declined, don&apos;t ask again. Proceed without it, or report and stop&quot; and reran. This time the model tried <strong>four times in different forms</strong>. <code>npm test</code> was declined, so it tried <code>node --test</code>, and when that was declined, <code>cd tests &amp;&amp; node --test</code>. Each time a confirmation appeared, and the user had to stop it every time.</p><p>Worse, at that moment the harness was pushing back with &quot;don&apos;t repeat the same failure; fix the cause.&quot; It was counting the declined confirmation as a &quot;failure to be fixed.&quot; The model took that and went looking for a way around. <strong>The harness was egging on the search for a workaround.</strong></p><p>We fixed three things. Failures caused by the user&apos;s policy (declined confirmations, denied permissions) are not counted in the continuation judgment. A confirmation of the declined kind (the kind of permission, such as execute or write) is not asked again until the user gives the next instruction, and is automatically failed. And in that profile, automated verification does not run tests on its own either.</p><p>The last one was actually a defect where, right after the user declined <code>n</code>, the harness&apos;s automated verification ran the very same <code>npm test</code> on its own. We were conveying the user&apos;s intent to the model while not honoring it ourselves. Unless execution permission is &quot;allowed,&quot; automated verification is now limited to a syntax check.</p><p>In the fourth rerun, the confirmation appeared exactly once, the subsequent five attempts failed automatically, and the user was not asked.</p><h3 id="2-3-for-dont-ask-again-a-key-of-the-same-turn-was-not-enough">2-3. For &quot;don&apos;t ask again,&quot; a key of &quot;the same turn&quot; was not enough</h3><p>A small detail, but here is one number we measured.</p><p>We first implemented &quot;don&apos;t re-ask a declined kind of confirmation&quot; as &quot;within the same turn.&quot; Large models call multiple tools per turn, so we thought that would suffice. But a small model that calls only one tool per turn <strong>asked again seven times across different turns</strong> and ran straight to the 40-turn limit.</p><p>Changing the key to &quot;until the user&apos;s next instruction&quot; settled it. A hole we only saw because we checked with a small model.</p><h2 id="3-dont-let-the-model-decide-who-started-this-process">3. Don&apos;t Let the Model Decide &quot;Who Started This Process&quot;</h2><p>The &quot;impatient one&quot; script produced a different kind of defect.</p><p>The model tried to start a server on port 3000, and the port was in use (EADDRINUSE). That port was held by a different app unrelated to this run. The harness&apos;s advice was:</p><p>&quot;If it&apos;s a process started by this run, kill it. To kill it: <code>taskkill /PID &lt;number&gt;</code>&quot;</p><p>It delegated the judgment to the model, and even showed how to kill it. This time the model killed only the process it had started itself, so no harm was done, but it could very well have killed someone else&apos;s app.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig04_2_who_started-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 4: Autonomy Rules Are for When Nobody Is Around. If a Human Is Present, Stop and Ask" loading="lazy"><figcaption>Figure 2: The machine knows who started a process (diagram: Qualiteg)</figcaption></figure><p>What we realized here is that <strong>the harness knows &quot;who started it&quot;</strong>. It remembers the PIDs of the processes it started itself. So the judgment need not be left to the model; it can be separated mechanically.</p><p>Processes started by this run may be killed. Processes started by others go to the user for confirmation (or, unattended, fail and suggest a different port). The harness&apos;s own processes are refused. And forms that kill by name in bulk (<code>taskkill /IM node.exe</code>, <code>pkill node</code>) are refused by the form itself, without looking at the arguments.</p><p>The last one has a backstory. In an evaluation run, the model executed <code>Get-Process -Name node | Stop-Process</code>, and <strong>the harness itself was killed and the run vanished</strong>. Looking only at &quot;the arguments of the kill command&quot; cannot stop a form that collects via a pipeline and then kills. We had to look at the whole form of &quot;collect by name, then kill.&quot;</p><p>Cleanup was fixed along the same lines. When a session ends, stop the servers the model started and left behind. At one point three servers were left on the machine, a defect invisible in evaluation runs because the evaluation rig had been stopping them. Furthermore, chains like <code>npm run web &amp;</code> (cmd -&gt; node -&gt; cmd -&gt; node) leave parts behind on a busy machine because enumeration of descendants cannot keep up, so we also kill &quot;anything born in the meantime that has lost its parent&quot; as a batch.</p><h2 id="4-if-something-was-started-in-the-background-say-so">4. If Something Was Started in the Background, Say So</h2><p>Running the same scripts with a large model, the skeleton of the dialogue (queue, interrupt, resume, refuse, clean up, log) held. Instead, one different friction appeared.</p><p>When the model starts a server in the background with <code>node server.js &amp;</code>, the shell exits immediately. All the model gets back is &quot;exit code: 0&quot; and empty output. It cannot tell whether it worked. The model tried the same startup <strong>five ways</strong> (<code>&amp;</code>, two forms of <code>start /b</code>, with a redirect to a log, and PowerShell&apos;s <code>Start-Process</code>), two of which hit the 120-second timeout, concluded &quot;<code>&amp;</code> does not work in this shell,&quot; and burned about four minutes.</p><p>The harness had adopted the background child process and remembered its PID. It knew, but it kept quiet toward the model. When the command is written in a way that clearly backgrounds a process, we now attach to the result: &quot;Started in the background (pid n). Don&apos;t repeat the same startup. Verify with a different command, and stop it when done.&quot;</p><p>This was a form hard to see in evaluation runs. There, the main path was starting in the foreground and detecting the listener; it only showed up in numbers when a person asked &quot;write tests too&quot; and had the model start the server itself.</p><h2 id="5-configuration-problems-one-line-and-the-next-step-not-a-stack-trace">5. Configuration Problems: One Line and the Next Step, Not a Stack Trace</h2><p>The &quot;misconfigurer&quot; script is the plainest, and the very first place a user steps.</p><p>Starting on a machine with no API key. Mistyping the model name and getting a 404. The endpoint being down and unreachable. Pointing at a nonexistent working directory.</p><p>The first version stopped with a stack trace in every case. Worse, 404 and 401 were treated the same as &quot;temporary failures&quot; and resent five times, after which the circuit breaker tripped and it stopped. From the user&apos;s point of view, they waited tens of seconds without knowing what was wrong.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig04_3_config_error-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 4: Autonomy Rules Are for When Nobody Is Around. If a Human Is Present, Stop and Ask" loading="lazy"><figcaption>Figure 3: Stop configuration problems with one line and the next step (diagram: Qualiteg)</figcaption></figure><p>A 401 or 403 for a missing key, a 404 for a wrong model name, and a 400 for a bad setting will not pass no matter how often you resend. Stop within one turn and report the reason in one line plus the next step (the command to check the setting). The context-overflow 400 from Part 3 is different: that one is compacted and resent. If the endpoint is merely down temporarily, retry up to the limit as in Part 1, then stop. We also gave it a distinct exit code as &quot;a failure the user fixes.&quot;</p><p>One more thing: operations that only read the record (listing or aggregating past sessions) were demanding the model connection settings, because a read-only operation was assembling the entire agent. If it only reads, assemble only the record component. Even a small decomposition like this changes whether you can read the record on a machine without a key.</p><h2 id="6-six-personas-19-defects">6. Six Personas, 19 Defects</h2><p>The persona tests found and fixed 19 defects. Laid out in a table, most of them fall under &quot;applying evaluation-run discipline unchanged to interactive use.&quot;</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Persona</th><th>Count</th><th>Defects found (summary)</th></tr></thead><tbody><tr><td>The first-timer</td><td>2</td><td>Pushes back a question and builds on its own. Brings up the previous run&apos;s verification results in reply to &quot;thanks.&quot; Replies in a different language (model)</td></tr><tr><td>The impatient one</td><td>4</td><td>Queued instruction is displayed twice. Shows how to kill someone else&apos;s process. Background startup returns only &quot;exit 0.&quot; Doesn&apos;t stop leftover servers on exit</td></tr><tr><td>The maintainer</td><td>2</td><td>Creates a summary file nobody asked for. Working directory shows up in <code>git status</code></td></tr><tr><td>The cautious one</td><td>4</td><td>Asks again after being declined. Asks again seven times in different forms. Counts refusals as failures and pushes back. Automated verification runs tests on its own</td></tr><tr><td>The web UI user</td><td>2</td><td>Not a single session is saved. Fails 12 times when the TODO argument shape differs</td></tr><tr><td>The misconfigurer</td><td>5</td><td>Stack traces. Demands a key just to read the record. Resends a 404 five times</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Today, this agent stops and asks on vague instructions. Instructions given mid-run are queued and picked up on the next turn. Esc stops it even mid-thought, and the next instruction resumes from there. A declined confirmation is not asked again. Other people&apos;s processes are not killed. Configuration problems stop with one line. Leftover servers are stopped on exit.</p><p>None of this adds a single point to the evaluation score.</p><h2 id="summary-dont-stop-and-stop-and-ask-are-decided-by-the-same-role">Summary: &quot;Don&apos;t Stop&quot; and &quot;Stop and Ask&quot; Are Decided by the Same Role</h2>
<!--kg-card-begin: html-->
<table><thead><tr><th>Situation</th><th>Nobody present (unattended run)</th><th>Human present (chat screen)</th></tr></thead><tbody><tr><td>The model asked a question</td><td>Push back with an assumption attached</td><td>Stop and ask</td></tr><tr><td>A confirmation was declined</td><td>Look for a move that needs no confirmation</td><td>Don&apos;t re-ask the same kind of confirmation until the next instruction</td></tr><tr><td>Someone else&apos;s process holds the port</td><td>Fail and suggest a different port</td><td>Send to the user for confirmation</td></tr><tr><td>Execution permission requires confirmation</td><td>(Unattended, grant it explicitly)</td><td>Automated verification is syntax check only</td></tr><tr><td>Configuration problem</td><td>Stop with one line and the next step</td><td>Same</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The &quot;stop decider&quot; described in Part 1 was never a role just for &quot;not stopping.&quot; It takes whether a human is present as an input and returns the opposite judgment for the same situation.</p><p>Next time: the local LLMs that underpin every experiment in this series. How we handled model quirks, quantization degradation, GPU memory, and repairing broken tool calls on the way to getting two 16GB cards to build an entire web system. With tables of measured numbers.</p><p>See you next time!</p><h2 id="related-articles">Related Articles</h2><ul><li><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Building a Coding Agent from Scratch, Part 1: Why &quot;Finishing the Job&quot; Is Hard, and Separating the Turn Runner from the Stop Decider</a></li><li><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">Building a Coding Agent from Scratch, Part 3: &quot;It Didn&apos;t Stop&quot; Is Not &quot;It Made Progress&quot;</a></li><li><a href="https://journal.qualiteg.com/claude-code-read-deny-grep-permission-prompt/">Claude Code Suddenly Asks for Approval on grep: v2.1.259 Now Applies Read Deny Rules to Bash grep</a></li><li><a href="https://journal.qualiteg.com/designing-accountability-for-ai-agents-part1/">Making AI Agents Business-Ready [Part 1]</a></li></ul>]]></content:encoded></item><item><title><![CDATA[Japanese LLM Rankings 2026 — Benchmark Analysis Report (October 22 Edition)]]></title><description><![CDATA[<h2 id="the-bottom-line-first-llm-rankings-as-of-october-22-2026">The bottom line first: LLM rankings as of October 22, 2026</h2><p>This is the October 22, 2026 edition of our Japanese-capable LLM rankings. We will save the preamble for later and show you the latest rankings first.</p><h3 id="latest-frontier-model-rankings-top-20-epoch-capabilities-index">Latest frontier model rankings TOP 20 (Epoch Capabilities Index)</h3>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rank</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Model</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Developer</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Access</th></tr></thead></table></div>]]></description><link>https://journal.qualiteg.com/llm-ranking-2026/</link><guid isPermaLink="false">6ad958502ead0f114b6f0aa1</guid><category><![CDATA[LLM]]></category><category><![CDATA[Generative AI Frontlines]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Thu, 22 Oct 2026 00:21:50 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/10/2026_llm_ranking_1021-en.jpg" medium="image"/><content:encoded><![CDATA[<h2 id="the-bottom-line-first-llm-rankings-as-of-october-22-2026">The bottom line first: LLM rankings as of October 22, 2026</h2><img src="https://journal.qualiteg.com/content/images/2026/10/2026_llm_ranking_1021-en.jpg" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)"><p>This is the October 22, 2026 edition of our Japanese-capable LLM rankings. We will save the preamble for later and show you the latest rankings first.</p><h3 id="latest-frontier-model-rankings-top-20-epoch-capabilities-index">Latest frontier model rankings TOP 20 (Epoch Capabilities Index)</h3>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rank</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Model</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Developer</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Access</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Release date (per Epoch AI)</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">ECI</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 5.5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/22</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">168.8</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">2</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-6 Astra</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/03</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">168.8</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-6.1 Sol</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/29</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">168.4</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">4</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Fable 5.1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/01</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">166.5</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 4 Argon</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/30</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">166.4</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">6</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Sonnet 5.5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/28</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">166.2</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">7</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">07/24</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">163.9</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-6 Sol</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/22</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">163.7</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">9</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Fable 5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">06/09</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">163.4</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">10</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.5 Pro</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">04/23</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">162.9</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">11</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.6 Sol</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">07/09</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">162.8</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">12</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Muse Wonder</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Meta AI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">10/12</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">161.8</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">13</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.6 Terra</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">07/09</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">160.1</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">14</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">04/23</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">159.7</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">15</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.4 Pro</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">03/05</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">159.6</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">16</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Haiku 5.5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">10/07</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">159.3</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">17</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 4.8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">05/28</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">159.0</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">18</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3.7 Flash</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">08/13</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">158.0</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">19</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Kimi K3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Moonshot</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Open</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">07/16</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">157.9</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">20</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3.8 Flash</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/02</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">157.4</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<p>* An index of overall capability (Epoch AI, updated October 21). How to read it, and our analysis, are in Part 1.</p><h3 id="measured-rankings-on-japanese-tasks-top-10-nejumi-leaderboard-4">Measured rankings on Japanese tasks TOP 10 (Nejumi Leaderboard 4)</h3>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rank</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Model</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Category</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Total Score</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-opus-5: adaptive-thinking-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8720</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">2</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-fable-5: adaptive-thinking-max with fallback to Opus 4.8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8699</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">Qwen/Qwen3.8-2.4T-A95B: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8598</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">4</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-opus-4.8: adaptive-thinking-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8523</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-opus-4.7: adaptive-thinking-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8509</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">6</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.6-sol: max-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8499</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">7</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.6-terra: max-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8440</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">moonshotai/kimi-k3: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8432</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">9</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemini-3.1-pro-preview</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8430</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">10</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">moonshotai/Kimi-K3: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8425</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<p>* Total score on Japanese tasks (per evaluation run). Rows in light blue are open models. Second-place Claude Fable 5 was evaluated in a configuration that includes fallback to Opus 4.8. The full TOP 50 table and analysis are in Part 2.</p><p>Here are this edition&apos;s three big topics.</p><ul><li><strong>Frontier update: Claude Opus 5.5 and GPT-6 Astra tie at the top</strong>.<br> Both score 168.8 on Epoch AI&apos;s ECI. Claude Fable 5.1, Gemini 4 Argon, and Claude Sonnet 5.5 follow in the 166 range, and none of them has been evaluated on Nejumi yet<br></li><li><strong>On Nejumi (Japanese tasks), the leader Claude Opus 5 (0.8720) holds firm</strong>. The top 13 are exactly the same as last edition, and 13 of the 14 newcomers are open models<br></li><li><strong>Meta and Japan&apos;s llm-jp-4.1 debut in the Japanese rankings</strong>.<br>Meta&apos;s open model Muse Spark Open ranks 14th overall (0.8379), and the 33B version of llm-jp-4.1 from the National Institute of Informatics becomes the top Japanese-made open model</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-topics3-v3-en.jpg" class="kg-image" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)" loading="lazy" width="2000" height="1116" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/llm-ranking-202610-topics3-v3-en.jpg 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/llm-ranking-202610-topics3-v3-en.jpg 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/llm-ranking-202610-topics3-v3-en.jpg 1600w, https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-topics3-v3-en.jpg 2000w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">This edition&apos;s three big topics (chart by Qualiteg)</span></figcaption></figure><p>(We update these LLM rankings regularly. Follow our <a href="https://x.com/qualiteg_hq?ref=journal.qualiteg.com">X</a> (formerly Twitter) account to get notified of updates.)</p><hr><h2 id="why-this-edition-has-two-parts-and-where-the-three-rankings-fit">Why this edition has two parts, and where the three rankings fit</h2><p>Up to the previous <a href="https://journal.qualiteg.com/llm-ranking-2026-09/">September 1, 2026 edition</a>, this series relied solely on data from <a href="https://nejumi.ai/?ref=journal.qualiteg.com">Nejumi Leaderboard 4</a> (Weights &amp; Biases Japan). Starting with this edition, we also draw on new ranking sources that can track models right after release.<br><br>From this edition on, Part 1 draws on independent evaluation data that tracks models right after release, and Part 2 holds the detailed Nejumi analysis.<br><br>The three sources we use from now on are as follows. Because they measure different things, <strong>we do not place the three scores side by side for comparison</strong>.</p>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Ranking</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Operator</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">What it measures</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Coverage</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Latest update</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;">Epoch Capabilities Index (ECI)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Epoch AI (U.S. research organization)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">An index of overall capability that integrates more than 50 benchmarks, combining Epoch&apos;s own evaluations, independent leaderboards, and developer-reported figures</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Both commercial APIs and open models</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">October 21</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;">Arena, Japanese category</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Arena (formerly LMArena)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Ratings computed with the Bradley-Terry method from votes by people comparing answers from two models. Only Japanese-language prompts are counted</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Both commercial APIs and open models</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">October 20</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;">Nejumi Leaderboard 4</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Weights &amp; Biases Japan</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Measures real performance on Japanese tasks across 35 columns on two axes, General Language Performance (GLP) and Alignment (ALT)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Both commercial APIs and open models</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Latest run October 19 (the last entry from the American commercial API camp was July 25)</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<p>* Rows in light blue are used in Part 1, white rows in Part 2.</p><ul><li><strong>Epoch Capabilities Index (ECI)</strong>. An index from the U.S. research organization Epoch AI that combines results from more than 50 benchmarks, such as GPQA Diamond and SWE-bench Verified, into a single measure of overall capability. In addition to Epoch&apos;s own evaluations, it incorporates values from independent leaderboards and developer-reported figures, so new models appear within days of release. Higher is better, and each estimate comes with a confidence interval (<a href="https://epoch.ai/eci?ref=journal.qualiteg.com">Epoch AI&apos;s public data</a>)</li><li><strong>Arena, Japanese category</strong>. Ratings computed from users submitting the same prompt to two models and voting for the better answer. It is not a benchmark but a tally of which model people prefer in Japanese, and rankings can shift for models with few votes (<a href="https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset?ref=journal.qualiteg.com">Arena&apos;s public data</a>)</li><li><strong>Nejumi Leaderboard 4</strong>. Measures real Japanese-task performance across 35 columns on two axes, GLP and ALT, from translation, summarization, reasoning, and coding to toxicity, bias, and truthfulness. Its unique value, which the other two lack, is showing what each model is good at and where it is risky, down to the breakdown</li></ul><p>In addition, the independent evaluators <a href="https://artificialanalysis.ai/?ref=journal.qualiteg.com">Artificial Analysis</a> and <a href="https://www.vals.ai/benchmarks/vals_index?ref=journal.qualiteg.com">Vals AI</a> also list the newest models quickly. We do not reproduce their figures in this article and provide links only.</p><hr><h2 id="part-1-latest-frontier-model-rankings">Part 1: Latest frontier model rankings</h2><h3 id="the-latest-frontier-by-overall-capability-epoch-capabilities-index">The latest frontier by overall capability (Epoch Capabilities Index)</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig7-eci-top20-1-en.png" class="kg-image" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)" loading="lazy" width="1840" height="1632" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/llm-ranking-202610-fig7-eci-top20-1-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/llm-ranking-202610-fig7-eci-top20-1-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/llm-ranking-202610-fig7-eci-top20-1-en.png 1600w, https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig7-eci-top20-1-en.png 1840w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">ECI TOP 20. Light blue = not yet on Nejumi, navy = listed on Nejumi. Horizontal lines are the confidence intervals published by Epoch AI</span></figcaption></figure><p>Here again is the TOP 20 table shown at the top of this article.</p>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rank</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Model</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Developer</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Access</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Release date (per Epoch AI)</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">ECI</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 5.5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/22</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">168.8</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">2</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-6 Astra</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/03</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">168.8</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-6.1 Sol</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/29</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">168.4</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">4</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Fable 5.1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/01</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">166.5</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 4 Argon</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/30</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">166.4</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">6</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Sonnet 5.5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/28</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">166.2</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">7</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">07/24</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">163.9</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-6 Sol</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/22</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">163.7</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">9</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Fable 5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">06/09</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">163.4</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">10</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.5 Pro</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">04/23</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">162.9</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">11</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.6 Sol</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">07/09</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">162.8</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">12</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Muse Wonder</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Meta AI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">10/12</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">161.8</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">13</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.6 Terra</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">07/09</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">160.1</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">14</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">04/23</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">159.7</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">15</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.4 Pro</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">OpenAI</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">03/05</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">159.6</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">16</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Haiku 5.5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">10/07</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">159.3</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">17</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 4.8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">05/28</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">159.0</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">18</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3.7 Flash</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">08/13</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">158.0</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">19</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Kimi K3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Moonshot</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Open</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">07/16</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">157.9</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">20</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3.8 Flash</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">API</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">09/02</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">157.4</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<p>* Release dates are as stated by Epoch AI.</p><p>The ECI leaders are <strong>Claude Opus 5.5 (168.8) and GPT-6 Astra (168.8)</strong>, with point estimates only about 0.02 apart. The confidence intervals published by Epoch AI (165.3 to 174.0 for Opus 5.5 and 165.5 to 174.5 for Astra) overlap heavily, so this data does not support a clear verdict on which is stronger. Third-place GPT-6.1 Sol (168.4) sits at nearly the same level.</p><p>Below them, Claude Fable 5.1 (166.5), Gemini 4 Argon (166.4), and Claude Sonnet 5.5 (166.2) cluster in the 166 range. All six of these models are <strong>absent from Nejumi</strong>.</p><p>Claude Opus 5, the Nejumi leader, ranks 7th on ECI at 163.9. In other words, six unevaluated models sit above the Nejumi leader. ECI and Nejumi measure different things, so rankings do not transfer directly, but once these models are evaluated on Nejumi, the top of the Japanese rankings could very well move.</p><p>Three other points stand out.</p><ul><li><strong>Meta enters the frontier</strong>. Muse Wonder (161.8), released on October 12, is the first commercial API model from Meta to reach the ECI top 12. It is a higher-end model, distinct from the open-weight Muse Spark Open (155.1) covered in Part 2</li><li><strong>Small models catch up with last generation&apos;s flagships</strong>. Claude Haiku 5.5 (159.3) is on par with Claude Opus 4.8 (159.0), the flagship released in May. Claude Sonnet 5.5 (166.2) even outscores the Nejumi leader Opus 5 (163.9)</li><li><strong>Kimi K3 is the top open model</strong>. The highest open-weight score is Kimi K3 at 157.9 (19th), followed by GLM-5.3 (156.0), DeepSeek V4.1 Flash (155.9), and Muse Spark Open (155.1). On ECI, the gap between the open camp and the top API models is about 11 points</li></ul><p>Because ECI also incorporates developer-reported figures, the influence of vendor claims cannot be ruled out entirely. Even so, since it combines independent leaderboard values with Epoch&apos;s own evaluations, it is currently the most practical indicator for gauging where a just-released model stands.</p><h3 id="human-evaluation-in-japanese-arena-japanese-category">Human evaluation in Japanese (Arena, Japanese category)</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig8-arena-japanese-en.png" class="kg-image" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)" loading="lazy" width="1840" height="1344" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/llm-ranking-202610-fig8-arena-japanese-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/llm-ranking-202610-fig8-arena-japanese-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/llm-ranking-202610-fig8-arena-japanese-en.png 1600w, https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig8-arena-japanese-en.png 1840w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Arena Japanese category TOP 15. Light blue = not yet on Nejumi, navy = listed on Nejumi. Numbers on the right are vote counts</span></figcaption></figure>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rank</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Model (Arena naming)</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Developer</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rating</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Votes</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Fable 5.1 (Max)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,562</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">282</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">2</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 5.5 (High)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,534</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">179</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 5 (Max)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,520</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">594</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">4</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 5 (High)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,516</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,163</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3.7 Flash (High)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,511</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">476</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">6</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3 Pro</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,509</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">476</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">7</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3.8 Flash (High)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,507</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">762</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Fable 5 (High)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,504</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">643</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">9</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">GPT-5.5 (High)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">openai</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,499</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">907</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">10</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen3.5-Max-Preview</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">alibaba</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,495</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">209</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">11</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Kimi K3 (Max)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">moonshot</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,494</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">662</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">12</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3.1 Pro Preview</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,491</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,582</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">13</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3 Flash</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,489</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">340</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">14</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 4.6 (High)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">anthropic</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,488</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">950</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">15</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Gemini 3.5 Flash (High)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">google</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1,486</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">700</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<p>* Parentheses show the reasoning settings that Arena distinguishes.</p><p>In Arena&apos;s Japanese category, which tallies only Japanese-language prompts, <strong>Claude Fable 5.1 (1562) leads and Claude Opus 5.5 (1534) is second</strong>. The Nejumi leader Claude Opus 5 holds 3rd and 4th, so the same picture as on ECI, with Fable 5.1 and Opus 5.5 above Opus 5, appears in Japanese human evaluation as well.</p><p>That said, Fable 5.1 has only 282 votes so far, and its rating may still shift. Opus 5 (High) has 1,163 votes and Gemini 3.1 Pro Preview 1,582, a larger body of comparisons, but the reliability of any ranking must be judged not only by vote counts but also by the estimation error in the ratings.</p><p>Another point worth noting is that GPT-6 Astra, tied for first on ECI, does not appear in the Japanese-category top 15 we retrieved. The same goes for Gemini 4 Argon and Muse Wonder. This table alone cannot tell us whether they are not listed in the Japanese category, have too few votes yet, or simply rank 16th or lower, so we withhold any relative assessment in Japanese for now.</p><p>Among open models, Kimi K3 (1494) came in 11th. It is also the second open model and 10th overall on Nejumi in Part 2, and it leads the open camp in Japanese human evaluation too.</p><h3 id="part-1-summary-who-will-move-the-top-of-nejumi-next">Part 1 summary: who will move the top of Nejumi next?</h3><ul><li>On both overall capability (ECI) and Japanese human evaluation (Arena), <strong>Claude Opus 5.5 and Claude Fable 5.1 rank above the Nejumi leader Claude Opus 5</strong>. They are strong candidates to reach the top of Nejumi as well, but a change of leader on Japanese tasks will have to wait for actual evaluation results</li><li><strong>GPT-6 Astra</strong> is tied for first in overall capability but is not yet visible near the top in Japanese human evaluation. Its Japanese-task performance awaits Nejumi&apos;s evaluation</li><li><strong>Claude Sonnet 5.5 and Claude Haiku 5.5</strong> show lower-priced models matching last generation&apos;s flagship class, making them strong candidates for cost-sensitive use cases</li></ul><p>With that, let&apos;s move to Part 2 and take a detailed look at Japanese-task performance using the Nejumi Leaderboard 4 data.</p><hr><h2 id="part-2-rankings-of-japanese-capable-models-nejumi-leaderboard-4">Part 2: Rankings of Japanese-capable models (Nejumi Leaderboard 4)</h2><p>Part 2 is an analysis based on the <a href="https://nejumi.ai/?ref=journal.qualiteg.com">Nejumi Leaderboard 4</a> benchmark data (October 22, 2026 snapshot). This time the top of the board, leader included, did not move at all, while all 14 newly added models are either open models or from outside the United States. A quiet edition, but one with the feel of tectonic plates shifting!</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig2-trend-en.png" class="kg-image" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)" loading="lazy" width="1840" height="864" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/llm-ranking-202610-fig2-trend-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/llm-ranking-202610-fig2-trend-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/llm-ranking-202610-fig2-trend-en.png 1600w, https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig2-trend-en.png 1840w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">The top score holds at 0.8720. The number of 0.80+ models (per model) rose from 42 to 46, and every addition is an open model</span></figcaption></figure><h3 id="about-open-source-models">About open-source models</h3><p>Models with open weights are sometimes called &quot;open-source models&quot; or &quot;OSS models.&quot; Since not all of them release their training data and training code, this article uses the term &quot;open models&quot; throughout.</p><h3 id="about-this-benchmark-analysis">About this benchmark analysis</h3><p>This report presents trends and characteristics read from benchmark data, as reference information for LLM selection. For actual deployment, we recommend validating candidate models in your own environment for your specific use case.</p><p>Also, this edition&apos;s tally is based on data retrieved from the evaluation runs currently listed on the leaderboard (excluding archived ones). Note that the same model may appear as separate entries when evaluated under different settings such as reasoning/thinking modes.</p><p>With that, let&apos;s look at the overall rankings of Japanese-capable LLMs as of October 22, 2026.</p><hr><h2 id="overall-score-rankings-top-50">Overall Score Rankings: TOP 50</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig1-top15-en.png" class="kg-image" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)" loading="lazy" width="1840" height="1312" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/llm-ranking-202610-fig1-top15-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/llm-ranking-202610-fig1-top15-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/llm-ranking-202610-fig1-top15-en.png 1600w, https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig1-top15-en.png 1840w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Overall TOP 15. The top 13 are the same faces as last edition, and Meta&apos;s Muse Spark Open debuts at 14th</span></figcaption></figure><p>Here is the full TOP 50 table, including the TOP 10 shown at the top of this article.</p>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rank</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Model</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Category</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Total Score</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-opus-5: adaptive-thinking-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8720</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">2</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-fable-5: adaptive-thinking-max with fallback to Opus 4.8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8699</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">Qwen/Qwen3.8-2.4T-A95B: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8598</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">4</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-opus-4.8: adaptive-thinking-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8523</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-opus-4.7: adaptive-thinking-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8509</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">6</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.6-sol: max-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8499</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">7</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.6-terra: max-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8440</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">moonshotai/kimi-k3: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8432</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">9</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemini-3.1-pro-preview</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8430</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">10</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">moonshotai/Kimi-K3: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8425</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">11</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.5-2026-04-23: xhigh-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8411</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">12</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.4-2026-03-05: high-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8397</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">13</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-opus-4-6: extended-thinking</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8394</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">14</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">meta-models/Muse-Spark-Open-816B-A42B: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8379</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">15</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">tencent/Hy4-preview: reasoning-high</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8344</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">16</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemini-3.6-flash</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8312</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">17</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">qwen/qwen3.6-max-preview: openrouter-reasoning</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8295</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">18</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">Qwen/Qwen3.8-Flash-Next: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8291</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">19</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">zai-org/GLM-5.3: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8287</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">20</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.4-2026-03-05: xhigh-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8286</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">21</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.2-2025-12-11: xhigh-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8285</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">22</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">zai-org/GLM-5.3-Flash: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8260</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">23</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemini-3.5-flash</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8249</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">24</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">thinkingmachines/Inkling: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8244</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">25</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-sonnet-4.6: extended-thinking</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8230</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">26</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">thinkingmachines/Inkling-Small: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8222</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">27</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">tencent/Hy3: reasoning-high</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8200</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">28</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">Qwen/Qwen3.5-397B-A17B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8191</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">29</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">deepseek-ai/DeepSeek-V4-Flash-0731: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8169</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">30</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">ornith-ai/Ornith-1.5-397B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8161</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">31</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">XiaomiMiMo/MiMo-V2.6-Pro-MOPD: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8159</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">32</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">deepseek-ai/DeepSeek-V4.1-Flash: reasoning-effort-100</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8158</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">33</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">zai-org/GLM-5.2: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8156</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">34</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">xai/grok-4.5: reasoning-high</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8156</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">35</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemini-3-flash-preview</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8155</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">36</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.6-luna: max-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8145</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">37</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemini-3-pro-preview</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8134</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">38</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">Qwen/Qwen3.5-122B-A10B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8094</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">39</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">Qwen/Qwen3.8-27B: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Medium (10B-30B)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8091</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">40</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">openai/gpt-5.1-2025-11-13: high-effort</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8085</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">41</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">deepseek-ai/DeepSeek-V4-Flash-Vision-Exp: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8077</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">42</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">google/gemma-4-31b-it</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8077</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">43</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">deepseek-ai/DeepSeek-V4-Pro: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8067</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">44</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-opus-4.5-20251125: extended-thinking</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8064</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">45</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">MiniMaxAI/MiniMax-M3: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8061</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">46</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">Qwen/Qwen3.5-27B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Medium (10B-30B)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.8049</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">47</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">z-ai/glm-5.2: openrouter-reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8040</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">48</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">anthropic/claude-opus-4-1-20250805: extended-thinking</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">api</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.7992</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">49</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">deepseek-ai/DeepSeek-V4-Pro-0813: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.7984</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">50</td><td style="border:1px solid #c8d6e5;padding:2px 8px;background:#EAF3FC;">Qwen/Qwen3.6-35B-A3B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;background:#EAF3FC;">0.7978</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<p>* Rows highlighted in light blue are open models.</p><h2 id="overall-score-trends-and-analysis">Overall Score Trends and Analysis</h2><p>The defining feature of this edition is, without question, that <strong>the top of the board did not move</strong>. From the leader Claude Opus 5 (0.8720), second-place Claude Fable 5 (0.8699), and third-place Qwen3.8-2.4T-A95B (0.8598) down to 13th-place Claude Opus 4.6 (0.8394), the top 13 are identical to last edition in membership, order, and score.</p><ul><li><strong>Five models score 0.85 or higher</strong>, the same as last edition (Opus 5 / Fable 5 / Qwen3.8 / Opus 4.8 / Opus 4.7)</li><li><strong>46 models score 0.80 or higher</strong> (up from 42 last edition. Both are per-model counts, with multiple evaluation runs of the same model consolidated into its best score. The TOP 50 table counts evaluation runs, so the numbers do not match)</li><li><strong>Four newcomers entered the TOP 50</strong>: Muse Spark Open (14th), MiMo-V2.6-Pro (31st), DeepSeek-V4.1-Flash (32nd), and DeepSeek-V4-Flash-Vision-Exp (41st), all of them open models</li><li>Last edition&apos;s 47th to 50th (GPT-5, the API version of DeepSeek V4 Pro, Qwen3.6-27B, and Claude Sonnet 4.5) were pushed down to 51st through 55th. Their scores are unchanged</li></ul><h3 id="the-first-edition-of-2026-in-which-the-leader-stays-put">The first edition of 2026 in which the leader stays put</h3><p>In this series, the top spot has changed hands with every update so far in 2026: Gemini 3.1 Pro in the March 6 edition, Claude Opus 4.8 on July 10, and Claude Opus 5 on September 1. This is the first edition this year in which the leader holds.</p><p>The reason is simple. Among the evaluation runs added to the leaderboard over these seven weeks, not a single one came from the American commercial API camp of Anthropic, OpenAI, Google, or xAI. Of the 15 new runs (14 models), the only commercial API was Solar Pro 4 from Korea&apos;s Upstage, and the remaining 13 models were all open models.</p><p>As we saw in Part 1, GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1, Gemini 4 Argon, and others were released during this period, but none has been evaluated on Nejumi yet. This edition&apos;s rankings are therefore rankings among evaluated models only, and we note up front that the top could move once these are evaluated.</p><p>Within Nejumi, these seven weeks left the frontier ceiling untouched and only deepened the field. Of the 46 models above 0.80, 23 are commercial APIs and 23 are open models, an exact split.</p><h3 id="the-14-newcomers">The 14 newcomers</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig3-newcomers-en.png" class="kg-image" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)" loading="lazy" width="1840" height="1280" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/llm-ranking-202610-fig3-newcomers-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/llm-ranking-202610-fig3-newcomers-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/llm-ranking-202610-fig3-newcomers-en.png 1600w, https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig3-newcomers-en.png 1840w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">The 14 models added since the previous (September 1) edition. Solar Pro 4 is the only commercial API among them</span></figcaption></figure><p>The chart above lists the 14 models added to the leaderboard since the previous edition. Here are the newcomers worth highlighting.</p><ul><li><strong>Muse Spark Open 816B-A42B (14th, 0.8379)</strong>. Meta&apos;s open model debuts in the overall TOP 15 and third among open models (more in the open models section)</li><li><strong>MiMo-V2.6-Pro (31st, 0.8159)</strong>. Xiaomi&apos;s MoE with 1.02T total and 42B active parameters. It adds roughly 0.035 over the previous MiMo-V2.5-Pro (0.7812), making it Xiaomi&apos;s first model above 0.80</li><li><strong>DeepSeek-V4.1-Flash (32nd, 0.8158)</strong>. DeepSeek&apos;s new generation. It beats its predecessor on GLP (language performance) but falls just short of the older V4-Flash-0731 (0.8169) on the total score (more in the next section)</li><li><strong>Solar Pro 4 (52nd, 0.7960)</strong>. A commercial API from Korea&apos;s Upstage and the only API newcomer this time. It officially lists Japanese alongside English and Korean as supported languages, and came within a hair of 0.80</li><li><strong>llm-jp-4.1 series (102nd, 117th, and 124th)</strong>. Three domestic models from Japan&apos;s National Institute of Informatics (NII) arrive at once. The 33B version (0.7206) is now the top Japanese-made open model (more in the open models section)</li></ul><h3 id="the-glp-x-alt-landscape-this-editions-lesson-is-that-a-new-version-does-not-always-beat-the-old-one">The GLP x ALT landscape: this edition&apos;s lesson is that a new version does not always beat the old one</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig5-glp-alt-en.png" class="kg-image" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)" loading="lazy" width="1840" height="1280" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/llm-ranking-202610-fig5-glp-alt-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/llm-ranking-202610-fig5-glp-alt-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/llm-ranking-202610-fig5-glp-alt-en.png 1600w, https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig5-glp-alt-en.png 1840w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Upper right is better. DeepSeek-V4.1-Flash beats its predecessor on GLP but sinks far on ALT</span></figcaption></figure><p>The scatter plot above maps the major models on two axes: GLP (language performance) and ALT (alignment). The overall top models all sit in the upper right. In other words, they <strong>combine &quot;smarts&quot; with safety and controllability</strong>.</p><p>This edition&apos;s case study is the newcomer <strong>DeepSeek-V4.1-Flash</strong>. Its GLP of 0.8135 beats the 0.8002 of the older V4-Flash-0731. Mathematical reasoning at 0.9683 and abstract reasoning at 0.81 show solid gains in the reasoning categories.</p><p>But its truthfulness score of 0.497 was the lowest in the TOP 50 (the next lowest being Gemini 3 Flash Preview at 0.518). Comparing the ALT breakdown with the previous version, controllability, ethics, toxicity, and robustness all improved, and bias slipped only marginally (0.870 to 0.864), but truthfulness alone plunged from 0.760 to 0.497. Driven mainly by that one category, ALT fell from 0.8603 to 0.8373, and the total of 0.8158 landed just below the old version&apos;s 0.8169. A rare reversal in which a new generation fails to overtake its predecessor.</p><p>Following Gemini 3.6 Flash (truthfulness 0.607) last edition, low truthfulness again stood out this time in a speed- and cost-oriented model. Two cases are not enough to generalize, but if you are adopting a fast, low-cost model, we recommend checking this axis individually.</p><p>Incidentally, the reversal we reported last time, GPT-5.4&apos;s high setting (0.8397) outscoring its xhigh setting (0.8286), remains unchanged in this edition&apos;s data.</p><h3 id="how-to-interpret-benchmark-results">How to interpret benchmark results</h3><p>The scores in this report are, in the end, results from benchmark tests. Benchmarks are a useful tool for comparing LLM performance objectively, but please keep the following in mind.</p><ul><li>Benchmarks evaluate models on a specific task set, so models well suited to that task mix tend to score higher</li><li>Real-world usability and usefulness for a specific purpose cannot be fully captured by benchmark scores alone</li><li>Some models have unique strengths that benchmarks simply do not measure</li></ul><p>Next, let&apos;s narrow the focus to open models. Almost everything that moved this time happened in this division.</p><hr><h2 id="open-model-overall-score-rankings-top-20">Open Model Overall Score Rankings: TOP 20</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig4-open-en.png" class="kg-image" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)" loading="lazy" width="1840" height="1440" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/llm-ranking-202610-fig4-open-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/llm-ranking-202610-fig4-open-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/llm-ranking-202610-fig4-open-en.png 1600w, https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig4-open-en.png 1840w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Open TOP 15. Text inside the bars shows the parameter configuration (as stated on the HF model cards. GLM-5.3 and Ornith disclose total parameters only. DeepSeek-V4.1-Flash&apos;s active count is 8B for prefill / 16B for decode). The dashed line marks the gap to the API leader</span></figcaption></figure>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rank</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Model</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Size Class</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Total Score</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.8-2.4T-A95B: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8598</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">2</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">moonshotai/Kimi-K3: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8425</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">meta-models/Muse-Spark-Open-816B-A42B: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8379</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">4</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">tencent/Hy4-preview: reasoning-high</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8344</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.8-Flash-Next: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8291</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">6</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">zai-org/GLM-5.3: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8287</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">7</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">zai-org/GLM-5.3-Flash: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8260</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">8</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">thinkingmachines/Inkling: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8244</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">9</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">thinkingmachines/Inkling-Small: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8222</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">10</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">tencent/Hy3: reasoning-high</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8200</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">11</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.5-397B-A17B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8191</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">12</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">deepseek-ai/DeepSeek-V4-Flash-0731: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8169</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">13</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">ornith-ai/Ornith-1.5-397B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8161</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">14</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">XiaomiMiMo/MiMo-V2.6-Pro-MOPD: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8159</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">15</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">deepseek-ai/DeepSeek-V4.1-Flash: reasoning-effort-100</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8158</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">16</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">zai-org/GLM-5.2: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8156</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">17</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.5-122B-A10B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8094</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">18</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.8-27B: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Medium (10B-30B)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8091</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">19</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">deepseek-ai/DeepSeek-V4-Flash-Vision-Exp: reasoning-max</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8077</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">20</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemma-4-31b-it</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">Large (30B+)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8077</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<p>* Rankings are per model (for models with multiple evaluation runs, only the best score is used).</p><h2 id="open-model-score-trends-and-analysis">Open Model Score Trends and Analysis</h2><h3 id="meta-enters-at-third-among-open-models-muse-spark-open">Meta enters at third among open models: Muse Spark Open</h3><p>The biggest news in this edition&apos;s open division is the arrival of Meta&apos;s <strong>Muse Spark Open 816B-A42B (0.8379)</strong>. It takes third place behind the open leader Qwen3.8-2.4T-A95B (0.8598) and second-place Kimi K3 (0.8425), overtaking Tencent Hy4 (0.8344).</p><p>Released by Meta Superintelligence Labs on October 15, it is an MoE with 816B total and 42B active parameters, using 8 of 256 experts, and it includes an encoder for image input. According to Meta, it is an open-weight release of the same base model as Muse Spark 1.3, which is available via API. Following the distilled Muse Glimmer 30B (0.7754) we covered last time, a full-scale Meta model has finally landed in the Japanese rankings.</p><p>Looking at the breakdown, mathematical reasoning at 0.9633 and logical reasoning at 0.97 are strong, and its ALT (alignment) of 0.8990 is among the best in the open camp. On the other hand, coding at 0.7121 and function calling at 0.5636 lag the top models, which is why its GLP of 0.8034 fell short of Qwen3.8 and Kimi K3.</p><p>One caveat. The license is not Apache 2.0 or MIT but Meta&apos;s own Muse Model License, which attaches conditions to commercial use. Also, while the training languages are described as more than 100, Japanese is not named specifically. We recommend checking the terms before commercial use.</p><p>In terms of depth, the number of open models above 0.80 grew from 19 last edition to <strong>23</strong>. That makes open models exactly half of the 46 models above 0.80.</p><h3 id="chinese-newcomers-xiaomi-deepseek-nex-agi-and-inclusionai">Chinese newcomers: Xiaomi, DeepSeek, Nex-AGI, and inclusionAI</h3><p>The rush of new open models from China continues this edition.</p><ul><li><strong>MiMo-V2.6-Pro (0.8159) / MiMo-V2.6-Flash (0.7600)</strong>. Xiaomi&apos;s new generation. Pro is an MoE with 1.02T total and 42B active parameters, Flash roughly 310B total and 15B active, both under the MIT license. Xiaomi describes the evaluated MOPD versions as having been on-policy distilled from multiple teacher models to curb repetitive tool calling (vendor description)</li><li><strong>DeepSeek-V4.1-Flash (0.8158)</strong>. The new generation released on September 10. Its configuration pairs a 552B main model with 196B of Engram conditional memory, with active parameters stated as 8B during prefill and 16B during decode (MIT license). The October 17 update &quot;1017&quot; (0.7541) is also listed, but its evaluation run name carries no reasoning setting, so it appears to have been evaluated without reasoning</li><li><strong>DeepSeek-V4-Flash-Vision-Exp (0.8077)</strong>. An experimental version of V4-Flash with a vision module added and further trained. It was listed on the leaderboard on September 1, but its API offering ended on September 10 when it was replaced by V4.1-Flash</li><li><strong>Nex-N2.5 series (Pro 0.7836 / Max 0.7711 / mini 0.7038)</strong>. Apache 2.0 models from Shanghai-based Nex-AGI. Pro has 397B total and 17B active parameters (based on Qwen3.5-397B-A17B), Max 1.6T total and 49B active (based on DeepSeek-V4-Pro-Base), and mini 35B total and 3B active (based on Qwen3.5-35B-A3B). Notably, the smaller Pro outscores the larger Max</li><li><strong>Ling-3.0-flash-VL (0.7767)</strong>. inclusionAI&apos;s multimodal version with 124B total and 5.5B active parameters under the MIT license. It scored slightly below the text-only Ling-3.0-flash (0.7924)</li></ul><h3 id="a-new-leader-among-japanese-made-open-models-llm-jp-41">A new leader among Japanese-made open models: llm-jp-4.1</h3><p>From Japan, the <strong>llm-jp-4.1 series</strong> from the Research and Development Center for Large Language Models at the National Institute of Informatics (NII) arrived with three models at once. They are the 33B dense version (0.7206), the 8B dense version (0.6921), and the 32B-A3B MoE version (0.6840), all under Apache 2.0 and all explicitly supporting Japanese.</p><p>Compared with the previous generation, the 33B version adds about 0.045 over llm-jp-4-33b-thinking (0.6760), and the 32B-A3B version rose from 0.6679 to 0.6840. According to NII, 4.1 is an improved post-training (SFT and DPO) release. The 33B version overtakes GPT-OSS-Swallow-120B-RL-v0.1 (0.6914) from the Swallow team at Institute of Science Tokyo to become the top Japanese-made open model.</p><p>In the breakdown, the 33B version&apos;s mathematical reasoning of 0.93 and translation of 0.9029 hold up well. Toxicity at 0.5887, however, is far below the top models (0.78 to 0.85), and abstract reasoning was extremely low at 0.14. The ALT side lags the GLP gains, so for work where factuality and safety matter, we recommend validating this axis.</p><h3 id="nearby-moves-solar-pro-4-from-koreas-upstage">Nearby moves: Solar Pro 4 from Korea&apos;s Upstage</h3><p>Closer to home, <strong>Solar Pro 4 (0.7960)</strong>, a commercial API model from Korea&apos;s Upstage, arrived as the only API newcomer this edition. Its parameter count is undisclosed, but the official documentation explicitly lists English, Korean, and Japanese as supported input and output languages. The large gap from its no-reasoning evaluation run (0.7207) suggests it is meant to be used with reasoning enabled.</p><p>Together with SK Telecom&apos;s A.X-K2 (0.7041) and LG AI&apos;s K-EXAONE-236B-A23B (0.7186) from last edition, the range of Korean models that officially cover Japanese is widening.</p><p>Next, let&apos;s look at mid-size models in the 10B to 30B range. The nice thing about mid-size models is that they run on GPUs that individuals can obtain relatively easily.</p><hr><h2 id="mid-size-model-10b-30b-overall-score-rankings">Mid-size Model (10B-30B) Overall Score Rankings</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig6-mid-small-en.png" class="kg-image" alt="Japanese LLM Rankings 2026 &#x2014; Benchmark Analysis Report (October 22 Edition)" loading="lazy" width="1840" height="1024" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/llm-ranking-202610-fig6-mid-small-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/llm-ranking-202610-fig6-mid-small-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/llm-ranking-202610-fig6-mid-small-en.png 1600w, https://journal.qualiteg.com/content/images/2026/10/llm-ranking-202610-fig6-mid-small-en.png 1840w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">The local LLM battleground. Top mid-size (left) and small (right) models. The dashed line marks 0.80</span></figcaption></figure>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rank</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Model</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Total Score</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.8-27B: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8091</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">2</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.5-27B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.8049</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.6-27B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.7955</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">4</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemma-4-26B-A4B-it</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.7872</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">meta-models/Muse-Glimmer-30B: reasoning-xhigh</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.7754</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">6</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.5-9B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.7485</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">7</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3-14B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.7233</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<p>* Because the tally is based on evaluation runs currently listed on the leaderboard, some models from the previous edition are excluded this time due to archiving. The mid-size / small classification (such as Qwen3.5-9B being treated as Medium) also follows the leaderboard&apos;s size categories.</p><h2 id="mid-size-model-score-trends-and-analysis">Mid-size Model Score Trends and Analysis</h2><p>The mid-size category saw no newcomers in these seven weeks, and <strong>all seven models hold the same positions as last edition</strong>.</p><p>The leader is still <strong>Qwen3.8-27B (0.8091)</strong>. A 27B dense model with a multimodal configuration that also accepts image and video input, released under Apache 2.0. Together with second-place Qwen3.5-27B (0.8049), the picture of Qwen&apos;s two models as the top picks for 30B-class local LLMs is unchanged.</p><p>A model of this size, which with quantization can run on a single GPU, scoring above 0.80 overall tops GPT-5 (high setting, 0.7970), released in August 2025, on the same leaderboard (required VRAM varies with precision and context length). It remains a strong, practical choice for on-prem and sensitive-data use cases.</p><p>Fourth-place Gemma 4 26B-A4B (0.7872) and fifth-place Muse-Glimmer-30B (0.7754) are still going strong. In future editions we will also be watching how the 32B-A3B version of llm-jp-4.1 is treated in this category (this time the leaderboard classifies it as Large).</p><p>Finally, let&apos;s look at small models. These compact models run on relatively inexpensive consumer GPUs, making them the prime candidates for edge and on-prem use.</p><hr><h2 id="small-model-under-10b-overall-score-rankings">Small Model (under 10B) Overall Score Rankings</h2>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Rank</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Model</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Total Score</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen/Qwen3.5-4B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.7352</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">2</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.7111</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">llm-jp/llm-jp-4.1-8b-thinking: reasoning-high</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.6921</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">4</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">ornith-ai/Ornith-1.5-9B: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.6909</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">5</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemma-4-E4B-it</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.6692</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">6</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">google/gemma-4-E2B-it</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.6564</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">7</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">tokyotech-llm/Qwen3-Swallow-8B-RL-v0.2: reasoning-enabled</td><td style="border:1px solid #c8d6e5;padding:2px 8px;text-align:center;">0.6552</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<h2 id="small-model-score-trends-and-analysis">Small Model Score Trends and Analysis</h2><p>Among small models, <strong>Qwen3.5-4B (0.7352)</strong> remains in first place. Scoring above 0.73 with just 4B parameters, it outperforms Qwen3-30B-A3B-Thinking-2507 (0.7331), released in July 2025, on the same leaderboard, a symbol of how far miniaturization has come.</p><p>Second-place <strong>NVIDIA Nemotron Nano 9B v2 Japanese (0.7111)</strong> is also still going strong. As a small model further trained for Japanese, it remains one of the leading options.</p><p>This edition&apos;s newcomer is third-place <strong>llm-jp-4.1-8b-thinking (0.6921)</strong>. It is the 8B dense version of the llm-jp-4.1 series introduced in the open models section, under Apache 2.0. As a Japanese-made small model that explicitly supports Japanese, it entered just ahead of Ornith-1.5-9B (0.6909). Its mathematical reasoning of 0.925 is eye-catching for a small model, while its SWE-Bench-based score of 0.0625 and toxicity of 0.5515 are major weaknesses.</p><p>Gemma 4&apos;s on-device E4B (0.6692) and E2B (0.6564), along with Qwen3-Swallow-8B-RL-v0.2 (0.6552) from the Swallow team at Institute of Science Tokyo, remain small-model staples.</p><hr><h2 id="summary-a-guide-toward-full-scale-deployment">Summary: a guide toward full-scale deployment</h2><p>This edition analyzed three data sources: Epoch AI, Arena, and Nejumi Leaderboard 4. Here are the key takeaways.</p><h3 id="the-frontier-has-moved-and-the-japanese-rankings-have-stopped-just-short-of-it">The frontier has moved, and the Japanese rankings have stopped just short of it</h3><p>There are two main points this time. <strong>On the frontier, Claude Opus 5.5 and GPT-6 Astra are tied in overall capability, and Claude Fable 5.1 leads Arena&apos;s Japanese category</strong>, and <strong>on Nejumi, which measures Japanese tasks directly, these models remain unevaluated and the top 13, led by Claude Opus 5, are unchanged from last edition</strong>.</p><p>What moved within Nejumi was the open camp: 13 of the 14 newcomers are open models, and open models above 0.80 reached 23, half of the 46. The trend we called the dawn of the open 0.85 era last time continues, now with a broader base.</p><ul><li>Commercial APIs: In overall capability, Anthropic (Opus 5.5 / Fable 5.1 / Sonnet 5.5) and OpenAI (GPT-6 Astra / GPT-6.1 Sol) form one leading group, joined by Google (Gemini 4 Argon) and Meta (Muse Wonder). Their Japanese evaluation on Nejumi is still to come</li><li>Open models: On Nejumi, Meta&apos;s Muse Spark Open breaks into third behind the two leaders Qwen3.8 and Kimi K3. Chinese newcomers from Xiaomi, DeepSeek, Nex-AGI, and others keep coming</li><li>Japanese-made models: llm-jp-4.1 arrives with three models, and the 33B version takes the top spot among domestic open models. With the Swallow RL versions and the Japanese Nemotron, Japanese-specialized options keep expanding steadily</li></ul><h3 id="characteristics-by-model-size">Characteristics by model size</h3>
<!--kg-card-begin: html-->
<div style="overflow-x:auto;"><table style="border-collapse:collapse;font-size:12px;font-family:sans-serif;">
  <thead>
    <tr>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Category</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Candidate Models</th>
      <th style="border:1px solid #c8d6e5;padding:4px 8px;background:#0B5C9C;color:#ffffff;">Notes</th>
    </tr>
  </thead>
  <tbody>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;">Maximum performance (commercial API)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Opus 5.5 / GPT-6 Astra / Claude Fable 5.1</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Top-tier in overall capability (ECI). Not yet evaluated on Nejumi, so for Japanese-language work, also validate the Nejumi leader Claude Opus 5</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;">Balancing cost and performance (commercial API)</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Claude Sonnet 5.5 / Claude Haiku 5.5 / Gemini 3.8 Flash</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">On ECI, Sonnet 5.5 outscores Opus 5 and Haiku 5.5 matches last generation&apos;s flagship Opus 4.8. Suited to high-volume, everyday work (not yet evaluated on Nejumi)</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;">Best-in-class open models</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen3.8-2.4T-A95B / Kimi K3</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">0.84 to 0.86 total on Nejumi, on par with top API models; can run on your own infrastructure (check the custom license terms)</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;">Single-GPU-class local deployment</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen3.8-27B / Qwen3.5-27B</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">27B class above 0.80 total; single-GPU setups feasible with quantization. The practical choice for on-prem and sensitive data</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;">Small / edge</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen3.5-4B / Gemma 4 E4B</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Qwen3.5-4B scores above 0.73 with 4B parameters. Along with Gemma 4 E4B (0.6692), a candidate for on-device and low-resource environments</td></tr>
    <tr><td style="border:1px solid #c8d6e5;padding:2px 8px;">Japanese-specialized</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">NVIDIA Nemotron Nano 9B v2 Japanese / llm-jp-4.1-33b-thinking</td><td style="border:1px solid #c8d6e5;padding:2px 8px;">Models trained on Japanese data; for domestic operation and Japanese-language work. llm-jp-4.1 is the new leader among Japanese-made open models</td></tr>
  </tbody>
</table></div>
<!--kg-card-end: html-->
<h3 id="toward-full-scale-llm-adoption-choosing-the-right-model-for-each-use-case">Toward full-scale LLM adoption: choosing the right model for each use case</h3><p>This edition has used total scores as the main lens for positioning each model, but real deployment decisions are not that simple.</p><p>Benchmarks are a valuable reference, but in practice you need a multifaceted assessment that goes beyond accuracy to inference cost, response speed, handling of sensitive data, and integration with existing systems. Especially in a situation like this one, where the choices keep multiplying, what matters on the ground is less about picking the single highest-scoring model and more about building a setup in which several models can be used appropriately for different purposes.</p><p>To support exactly this kind of multi-model operation, we offer <a href="https://bestllam.com/?ref=journal.qualiteg.com">Bestllam </a>, an integrated AI platform that lets you use multiple LLMs on a single platform. We can support you from model selection through integration into business workflows and operational design built around AI agents.</p><p>Beyond providing tools, we also offer <a href="https://qualiteg.com/consulting/business?ref=journal.qualiteg.com#ai-transformation">BPR consulting focused on bringing AI into your business operations</a>, accompanying you from business analysis through BPR planning, success metric design, and deployment support.</p><p><strong>Please feel free to reach out to us.</strong><br><a href="https://qualiteg.com/contact?inquiry=consulting&amp;ref=journal.qualiteg.com">https://qualiteg.com/contact?inquiry=consulting</a></p>]]></content:encoded></item><item><title><![CDATA[Qualiteg at CEATEC 2026: Exhibition Report]]></title><description><![CDATA[Qualiteg's four days at CEATEC 2026 (Makuhari Messe, October 13 to 16, booth 4H339), day by day: official daily visitor counts, 20 to 30 percent international visitors on Day 1, the first public showing of ANNNA REAL, a packed final day with returning visitors, and twice as many business meetings.]]></description><link>https://journal.qualiteg.com/ceatec-2026-report/</link><guid isPermaLink="false">6ad503db2ead0f114b6f0a7e</guid><category><![CDATA[Event]]></category><dc:creator><![CDATA[Qualiteg Business Development | Marketing]]></dc:creator><pubDate>Sun, 18 Oct 2026 12:00:00 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/10/cover_ceatec2026_report_en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/10/cover_ceatec2026_report_en.png" alt="Qualiteg at CEATEC 2026: Exhibition Report"><p>Hello! From Tuesday, October 13 to Friday, October 16, 2026, Qualiteg exhibited for four days at CEATEC 2026, held at Makuhari Messe.</p><p>This was our second year in a row, following CEATEC 2025. (Last year&apos;s show is covered in our <a href="https://journal.qualiteg.com/ceatec-2025-report/" rel="noreferrer">CEATEC 2025 Exhibition Report</a>, and this year&apos;s preview is in <a href="https://journal.qualiteg.com/ceatec-2026-announcement/" rel="noreferrer">this article</a>.)</p><p>According to the organizer, registered attendance over the four days of CEATEC 2026 reached <strong>112,373</strong>, up from 98,884 last year.</p><p>Our booth also welcomed more visitors than last year, and the number of business meetings doubled.</p><p>In this report we look back on the show in order, from setup day to the final day.</p><p>We also include the official daily visitor counts, which we hope will be useful to anyone exhibiting next year (and to ourselves, as a memo for next time).</p><h2 id="contents">Contents</h2><ul><li><a href="#ch1" rel="noreferrer">1. CEATEC 2026 Exhibit Overview</a></li><li><a href="#ch2" rel="noreferrer">2. Setup Day (October 12)</a></li><li><a href="#ch3" rel="noreferrer">3. DAY 1 (October 13): Opening Day, 20 to 30 Percent International Visitors</a></li><li><a href="#ch4" rel="noreferrer">4. DAY 2 (October 14): English Handouts, and Student Visitors</a></li><li><a href="#ch5" rel="noreferrer">5. DAY 3 (October 15): The First Public Showing of ANNNA REAL</a></li><li><a href="#ch6" rel="noreferrer">6. DAY 4 (October 16): Final Day. Short-Handed Afternoons and Returning Visitors</a></li><li><a href="#ch7" rel="noreferrer">7. What We Exhibited (1): MotionVox</a></li><li><a href="#ch8" rel="noreferrer">8. What We Exhibited (2): ANNNA REAL</a></li><li><a href="#ch9" rel="noreferrer">9. Our Commitment to In-House Technology</a></li><li><a href="#ch10" rel="noreferrer">10. Summary: Four Days in Numbers, and Notes for Next Year</a></li></ul>
<!--kg-card-begin: html-->
<div id="ch1"></div>
<!--kg-card-end: html-->
<h2 id="1-ceatec-2026-exhibit-overview">1. CEATEC 2026 Exhibit Overview</h2><p>CEATEC 2026 is a comprehensive technology exhibition organized by the Japan Electronics and Information Technology Industries Association (JEITA).</p><p>This year&apos;s theme was <strong>&quot;Transformation: Companies, industries, and society are changing.&quot;</strong></p><p>A total of 821 companies and organizations from Japan and abroad exhibited, and the show was held jointly with the Japan Automobile Manufacturers Association&apos;s &quot;Japan Mobility Show Bizweek 2026.&quot;</p><p>Our booth was <strong>4H339</strong>, in the Next Generation Park area of Hall 4.</p><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1245" height="920" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image.png 1245w" sizes="(min-width: 720px) 720px"></figure><p>The centerpiece of our exhibit was the latest version of MotionVox&#xAE;, our AI avatar video generation service. Enter a script, choose an avatar and a voice, and you get a video of a lifelike avatar speaking with natural expressions, with no filming, no appearing on camera, and no voice recording.</p><p>And on the third day, we unveiled ANNNA REAL, our real-time conversational AI human service, for the first time. More on that later.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-1.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1015" height="632" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-1.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-1.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-1.png 1015w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">ANNNA REAL, our real-time AI human service, shown in public for the first time</span></figcaption></figure>
<!--kg-card-begin: html-->
<table>
<thead>
<tr><th>Item</th><th>Details</th></tr>
</thead>
<tbody>
<tr><td>Dates</td><td>Tuesday, October 13 to Friday, October 16, 2026, 10:00 to 17:00</td></tr>
<tr><td>Venue</td><td>Makuhari Messe</td></tr>
<tr><td>Booth</td><td>Hall 4, Next Generation Park, 4H339</td></tr>
<tr><td>Exhibits</td><td>MotionVox&#xAE; (AI avatar video generation), ANNNA REAL (real-time AI human, preview release)</td></tr>
<tr><td>Total registered attendance</td><td>112,373 (organizer&apos;s figure, four days, combined with Japan Mobility Show Bizweek 2026)</td></tr>
</tbody>
</table>
<!--kg-card-end: html-->

<!--kg-card-begin: html-->
<div id="ch2"></div>
<!--kg-card-end: html-->
<h2 id="2-setup-day-october-12">2. Setup Day (October 12)</h2><p>Readers of last year&apos;s report may remember this.</p><p>Last year we arrived late at Makuhari Messe and set an odd record by building our booth late at night in an empty hall.</p><p>This year we learned our lesson and finished setup on October 12, the day before the show.</p><p>We arrived early, unpacked, put up the panels, started the demo machines, and got the booth into show-ready condition.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-2.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="713" height="777" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-2.png 600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-2.png 713w"><figcaption><span style="white-space: pre-wrap;">Setup finished early this year</span></figcaption></figure><p>Behind the demos runs our own in-house GPU cluster.</p><p>Both MotionVox avatar generation and ANNNA REAL&apos;s real-time conversation are processed on this cluster.</p><p>That evening, after finishing setup, one of our team members posted this on LinkedIn.</p><blockquote>We have finished setting up our CEATEC booth. Starting tomorrow (well, today), we will be exhibiting for four days with live AI avatar demos. Our team, and the GPU cluster working behind the scenes, look forward to welcoming you!</blockquote>
<!--kg-card-begin: html-->
<div id="ch3"></div>
<!--kg-card-end: html-->
<h2 id="3-day-1-october-13-opening-day-20-to-30-percent-international-visitors">3. DAY 1 (October 13): Opening Day, 20 to 30 Percent International Visitors</h2><p>Day 1 was cloudy. Registered attendance, according to the organizer, was <strong>25,637</strong> (Day 1 last year was 20,474).</p><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-3.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="510" height="867"></figure><p>As soon as the doors opened at 10:00, people started gathering at the booth.</p><p>At the front of the booth we played a video cycling through more than 1,000 avatars.</p><p><strong>&quot;There are this many?&quot;</strong></p><p>Many visitors stopped at that, and from there we moved naturally into the live demo of how a single script becomes a video.</p><figure class="kg-card kg-video-card kg-width-regular kg-card-hascaption" data-kg-thumbnail="https://journal.qualiteg.com/content/media/2026/10/ceatec2026r_x_20261013_avatars_thumb.jpg" data-kg-custom-thumbnail>
            <div class="kg-video-container">
                <video src="https://journal.qualiteg.com/content/media/2026/10/ceatec2026r_x_20261013_avatars.mp4" poster="https://img.spacergif.org/v1/1920x1080/0a/spacer.png" width="1920" height="1080" playsinline preload="metadata" style="background: transparent url(&apos;https://journal.qualiteg.com/content/media/2026/10/ceatec2026r_x_20261013_avatars_thumb.jpg&apos;) 50% 50% / cover no-repeat;"></video>
                <div class="kg-video-overlay">
                    <button class="kg-video-large-play-icon" aria-label="Play video">
                        <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                            <path d="M23.14 10.608 2.253.164A1.559 1.559 0 0 0 0 1.557v20.887a1.558 1.558 0 0 0 2.253 1.392L23.14 13.393a1.557 1.557 0 0 0 0-2.785Z"/>
                        </svg>
                    </button>
                </div>
                <div class="kg-video-player-container">
                    <div class="kg-video-player">
                        <button class="kg-video-play-icon" aria-label="Play video">
                            <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                                <path d="M23.14 10.608 2.253.164A1.559 1.559 0 0 0 0 1.557v20.887a1.558 1.558 0 0 0 2.253 1.392L23.14 13.393a1.557 1.557 0 0 0 0-2.785Z"/>
                            </svg>
                        </button>
                        <button class="kg-video-pause-icon kg-video-hide" aria-label="Pause video">
                            <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                                <rect x="3" y="1" width="7" height="22" rx="1.5" ry="1.5"/>
                                <rect x="14" y="1" width="7" height="22" rx="1.5" ry="1.5"/>
                            </svg>
                        </button>
                        <span class="kg-video-current-time">0:00</span>
                        <div class="kg-video-time">
                            /<span class="kg-video-duration">0:24</span>
                        </div>
                        <input type="range" class="kg-video-seek-slider" max="100" value="0">
                        <button class="kg-video-playback-rate" aria-label="Adjust playback speed">1&#xD7;</button>
                        <button class="kg-video-unmute-icon" aria-label="Unmute">
                            <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                                <path d="M15.189 2.021a9.728 9.728 0 0 0-7.924 4.85.249.249 0 0 1-.221.133H5.25a3 3 0 0 0-3 3v2a3 3 0 0 0 3 3h1.794a.249.249 0 0 1 .221.133 9.73 9.73 0 0 0 7.924 4.85h.06a1 1 0 0 0 1-1V3.02a1 1 0 0 0-1.06-.998Z"/>
                            </svg>
                        </button>
                        <button class="kg-video-mute-icon kg-video-hide" aria-label="Mute">
                            <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                                <path d="M16.177 4.3a.248.248 0 0 0 .073-.176v-1.1a1 1 0 0 0-1.061-1 9.728 9.728 0 0 0-7.924 4.85.249.249 0 0 1-.221.133H5.25a3 3 0 0 0-3 3v2a3 3 0 0 0 3 3h.114a.251.251 0 0 0 .177-.073ZM23.707 1.706A1 1 0 0 0 22.293.292l-22 22a1 1 0 0 0 0 1.414l.009.009a1 1 0 0 0 1.405-.009l6.63-6.631A.251.251 0 0 1 8.515 17a.245.245 0 0 1 .177.075 10.081 10.081 0 0 0 6.5 2.92 1 1 0 0 0 1.061-1V9.266a.247.247 0 0 1 .073-.176Z"/>
                            </svg>
                        </button>
                        <input type="range" class="kg-video-volume-slider" max="100" value="100">
                    </div>
                </div>
            </div>
            <figcaption><span style="white-space: pre-wrap;">More than 1,000 AI avatars are waiting for you. Makuhari Messe, booth 4H339, Qualiteg. (October 13, 2026, <a href="https://x.com/qualiteg_hq/status/2109878419454668858?ref=journal.qualiteg.com">original post on X</a>)</span></figcaption>
        </figure><p>What surprised us on Day 1 was the number of international visitors.</p><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-4.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="944" height="909" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-4.png 600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-4.png 944w" sizes="(min-width: 720px) 720px"></figure><p>By our rough estimate, 20 to 30 percent of the people who came to the booth were given the explanation in English.</p><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_linkedin_day1_en_demo.jpg" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="800" height="600" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_linkedin_day1_en_demo.jpg 600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_linkedin_day1_en_demo.jpg 800w" sizes="(min-width: 720px) 720px"></figure><p>MotionVox can produce videos in 46 languages with the same avatar and the same voice. When we showed English-speaking visitors the English version of the demo, the conversation moved quickly.</p><p>On the other hand, our handouts were only in Japanese. We were asked &quot;Do you have an English flyer?&quot; many times, which we were grateful for, so on the evening of Day 1 we decided to print English handouts in a hurry.</p>
<!--kg-card-begin: html-->
<div id="ch4"></div>
<!--kg-card-end: html-->
<h2 id="4-day-2-october-14-english-handouts-and-student-visitors">4. DAY 2 (October 14): English Handouts, and Student Visitors</h2><p>Day 2 was cloudy, with rain later. Registered attendance was <strong>25,649</strong> (Day 2 last year was 23,433).</p><p>In the morning we brought in the English handouts printed the night before. From this day on, explaining in English became even easier.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_x_20261014_day2.jpg" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="960" height="1280" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_x_20261014_day2.jpg 600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_x_20261014_day2.jpg 960w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Good morning, it is Day 2 of CEATEC 2026! The hall is cool and comfortable today. Live demos of real AI avatars at Hall 4, booth 4H339. (October 14, 2026, <a href="https://x.com/qualiteg_hq/status/2110202730082177298?ref=journal.qualiteg.com">original post on X</a>)</span></figcaption></figure><p>The change we felt on Day 2 was in who came. In addition to corporate visitors, we saw more students, people from research institutions, and other non-corporate visitors.</p><p>Students asked how the avatars are made and where the voices come from, questions about the mechanism itself, and those were enjoyable conversations for us too.</p><p>And this year, once again, we were asked:</p><p><strong>&quot;Does this avatar video use an external API?&quot;</strong> <strong>&quot;Which AI are you using?&quot;</strong></p><p>Just like last year, when we answered</p><p><strong>&quot;No, the avatar engine is developed in house,&quot;</strong></p><p>visitors were surprised, again and again.</p><p>Many visitors assume we are calling some AI service&apos;s API. In fact, we handle the whole pipeline ourselves:</p><p><strong>Training data sourcing (safe data, with rights and permissions secured)</strong> <strong>&#x2193;</strong> <strong>Cleansing</strong> <strong>&#x2193;</strong> <strong>AI training on our own GPU cluster</strong> <strong>&#x2193;</strong> <strong>Building the inference infrastructure</strong> <strong>&#x2193;</strong> <strong>Delivering it as an application</strong></p><p>We are deeply committed to developing our technology in house, with a team of specialists who have extensive experience and a track record in AI.</p><p>We were delighted that visitors could experience, first hand, what that commitment produces.</p><p>We come back to this in <a href="#ch9" rel="noreferrer">Chapter 9</a>.</p><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-29.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1920" height="1440" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-29.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-29.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/ceatec2026r_image-29.png 1600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-29.png 1920w" sizes="(min-width: 720px) 720px"></figure><p>Across the venue, after public hours ended that day, a joint networking event was held with the exhibitors of Japan Mobility Show Bizweek 2026.</p>
<!--kg-card-begin: html-->
<div id="ch5"></div>
<!--kg-card-end: html-->
<h2 id="5-day-3-october-15-the-first-public-showing-of-annna-real">5. DAY 3 (October 15): The First Public Showing of ANNNA REAL</h2><p>Day 3 was cloudy. Registered attendance was <strong>28,397</strong> (Day 3 last year was 24,601). By our own feel, the number of visitors at our booth was about 1.5 times that of Day 1.</p><p>On this day we unveiled a new service for the first time: ANNNA REAL, our real-time conversational AI human service.</p><figure class="kg-card kg-video-card kg-width-regular kg-card-hascaption" data-kg-thumbnail="https://journal.qualiteg.com/content/media/2026/10/ceatec2026r_x_20261015_annna_real_thumb.jpg" data-kg-custom-thumbnail>
            <div class="kg-video-container">
                <video src="https://journal.qualiteg.com/content/media/2026/10/ceatec2026r_x_20261015_annna_real.mp4" poster="https://img.spacergif.org/v1/1024x768/0a/spacer.png" width="1024" height="768" playsinline preload="metadata" style="background: transparent url(&apos;https://journal.qualiteg.com/content/media/2026/10/ceatec2026r_x_20261015_annna_real_thumb.jpg&apos;) 50% 50% / cover no-repeat;"></video>
                <div class="kg-video-overlay">
                    <button class="kg-video-large-play-icon" aria-label="Play video">
                        <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                            <path d="M23.14 10.608 2.253.164A1.559 1.559 0 0 0 0 1.557v20.887a1.558 1.558 0 0 0 2.253 1.392L23.14 13.393a1.557 1.557 0 0 0 0-2.785Z"/>
                        </svg>
                    </button>
                </div>
                <div class="kg-video-player-container">
                    <div class="kg-video-player">
                        <button class="kg-video-play-icon" aria-label="Play video">
                            <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                                <path d="M23.14 10.608 2.253.164A1.559 1.559 0 0 0 0 1.557v20.887a1.558 1.558 0 0 0 2.253 1.392L23.14 13.393a1.557 1.557 0 0 0 0-2.785Z"/>
                            </svg>
                        </button>
                        <button class="kg-video-pause-icon kg-video-hide" aria-label="Pause video">
                            <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                                <rect x="3" y="1" width="7" height="22" rx="1.5" ry="1.5"/>
                                <rect x="14" y="1" width="7" height="22" rx="1.5" ry="1.5"/>
                            </svg>
                        </button>
                        <span class="kg-video-current-time">0:00</span>
                        <div class="kg-video-time">
                            /<span class="kg-video-duration">1:51</span>
                        </div>
                        <input type="range" class="kg-video-seek-slider" max="100" value="0">
                        <button class="kg-video-playback-rate" aria-label="Adjust playback speed">1&#xD7;</button>
                        <button class="kg-video-unmute-icon" aria-label="Unmute">
                            <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                                <path d="M15.189 2.021a9.728 9.728 0 0 0-7.924 4.85.249.249 0 0 1-.221.133H5.25a3 3 0 0 0-3 3v2a3 3 0 0 0 3 3h1.794a.249.249 0 0 1 .221.133 9.73 9.73 0 0 0 7.924 4.85h.06a1 1 0 0 0 1-1V3.02a1 1 0 0 0-1.06-.998Z"/>
                            </svg>
                        </button>
                        <button class="kg-video-mute-icon kg-video-hide" aria-label="Mute">
                            <svg xmlns="http://www.w3.org/2000/svg" viewbox="0 0 24 24">
                                <path d="M16.177 4.3a.248.248 0 0 0 .073-.176v-1.1a1 1 0 0 0-1.061-1 9.728 9.728 0 0 0-7.924 4.85.249.249 0 0 1-.221.133H5.25a3 3 0 0 0-3 3v2a3 3 0 0 0 3 3h.114a.251.251 0 0 0 .177-.073ZM23.707 1.706A1 1 0 0 0 22.293.292l-22 22a1 1 0 0 0 0 1.414l.009.009a1 1 0 0 0 1.405-.009l6.63-6.631A.251.251 0 0 1 8.515 17a.245.245 0 0 1 .177.075 10.081 10.081 0 0 0 6.5 2.92 1 1 0 0 0 1.061-1V9.266a.247.247 0 0 1 .073-.176Z"/>
                            </svg>
                        </button>
                        <input type="range" class="kg-video-volume-slider" max="100" value="100">
                    </div>
                </div>
            </div>
            <figcaption><span style="white-space: pre-wrap;">ANNNA REAL, our real-time AI human service, made its public debut today and was very well received. &quot;Anna ni real. Kaiwa mo real.&quot; (That real. And the conversation is real too.) ANNNA REAL pairs a photorealistic avatar with natural, real-time conversation on the spot. (October 15, 2026, <a href="https://x.com/qualiteg_hq/status/2110724674485444976?ref=journal.qualiteg.com">original post on X</a>)</span></figcaption>
        </figure><p>Its tagline, in Japanese, is a play on words:</p><p><strong>&quot;Anna ni real. Kaiwa mo real.&quot;</strong> (That real. And the conversation is real too.)</p><p>At the booth, a visitor speaks to the avatar on screen, and it answers on the spot, with voice and facial expression.</p><p><strong>&quot;What is this booth about?&quot;</strong> <strong>&quot;What can MotionVox do?&quot;</strong></p><p>The demo answered questions like these in conversation, drawing on information about the venue.</p><p>ANNNA REAL is built on the MotionVox avatar generation engine, with the processing made real time. It keeps the realism of the facial expressions and lip movement, while adding live, in-person dialogue.</p><p>Visitors who saw ANNNA REAL asked things like:</p><p>&quot;We would like to put this at our front desk.&quot; &quot;We would like it to explain products in our store.&quot; &quot;Can it guide people in multiple languages?&quot;</p><p>Almost everyone was interested in the same thing: places where you cannot keep a person standing by.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_x_20261015_day3.jpg" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1280" height="960" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_x_20261015_day3.jpg 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_x_20261015_day3.jpg 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_x_20261015_day3.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Day 3 of CEATEC 2026. Many visitors came by and tried MotionVox for themselves. Thank you all! (October 15, 2026, <a href="https://x.com/qualiteg_hq/status/2110721880999227902?ref=journal.qualiteg.com">original post on X</a>)</span></figcaption></figure><p>According to the organizer, this was also the day when members of the Diet and senior officials from the Ministry of Economy, Trade and Industry and the Ministry of Internal Affairs and Communications visited the show.</p>
<!--kg-card-begin: html-->
<div id="ch6"></div>
<!--kg-card-end: html-->
<h2 id="6-day-4-october-16-final-day-short-handed-afternoons-and-returning-visitors">6. DAY 4 (October 16): Final Day. Short-Handed Afternoons and Returning Visitors</h2><p>The final day was the only sunny day of the four. Registered attendance was <strong>32,690</strong>, the highest of the show (the final day last year was 30,376).</p><p>Our booth, too, was simply packed all day. By feel, about twice the number of visitors of Day 1.</p><p>From a little after 13:00, there were repeated stretches when even four of us explaining at once were not enough. To everyone we kept waiting, please accept our apologies.</p><p>And the happiest thing about the final day was how many visitors came back.</p><p>People who had visited the booth on Day 1, Day 2, or Day 3 came to see us again on the final day.</p><p>Some had taken the story back to their company, discussed it, and returned. Some had compared us with other booths and then said,</p><p><strong>&quot;We would like to hear it once more.&quot;</strong></p><p>Some came back bringing a decision maker or a colleague with them.</p><p>With these returning visitors, the conversation very often turned into a formally scheduled business meeting right there, and we are truly grateful for that.</p><p>The fact that people thought <strong>&quot;Let&apos;s go back one more time&quot;</strong> is, to us, the best evaluation our four days of exhibiting could have received.</p><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-7.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1280" height="1231" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-7.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-7.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-7.png 1280w" sizes="(min-width: 720px) 720px"></figure>
<!--kg-card-begin: html-->
<div id="ch7"></div>
<!--kg-card-end: html-->
<h2 id="7-what-we-exhibited-1-motionvox">7. What We Exhibited (1): MotionVox</h2><p>From here, let us introduce what we exhibited once more.</p><p>MotionVox&#xAE; is a service that generates videos of a lifelike avatar speaking, simply from a script and your choice of avatar and voice. No filming, no appearing on camera, and no voice recording.</p><p>A year on from Expression Aware&#xAE; v2.0, which we first showed at CEATEC 2025, the main advances are as follows.</p><ul><li>Supported languages have grown from more than 30 to <strong>46</strong>. You can generate multilingual videos in one go, with the same avatar and the same voice</li><li>PowerPoint files, selfie videos, and single images have been added as input material</li><li>Choose from <strong>more than 1,000</strong> avatars, or create an original avatar from a prompt or a photo</li><li>Video creation now works by picking a template and filling in the content, so it takes shape even without video production skills</li><li>Emotes such as nods and surprise can be placed anywhere on the timeline</li></ul><p>At the booth we demonstrated, live, how a video is made from a news-program-style template.</p><figure class="kg-card kg-embed-card"><iframe width="200" height="113" src="https://www.youtube.com/embed/3oyyI7f11e8?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen title="MotionVox template: News Duo, two AI avatars in conversation"></iframe></figure><p>The feature that automatically generates a video of an avatar presenting a PowerPoint file also drew interest for training and manual use cases.</p><figure class="kg-card kg-embed-card"><iframe width="200" height="113" src="https://www.youtube.com/embed/VjUVRzlQbig?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen title="Generate a video from PowerPoint: an AI avatar presents your slides"></iframe></figure><p>For details of the features, please see our preview article and the MotionVox website.</p><ul><li><a href="https://journal.qualiteg.com/ceatec-2026-announcement/" rel="noreferrer">CEATEC 2026 Preview (highlights in detail)</a></li><li><a href="https://motionvox.com/?ref=journal.qualiteg.com" rel="noreferrer">MotionVox website</a></li></ul>
<!--kg-card-begin: html-->
<div id="ch8"></div>
<!--kg-card-end: html-->
<h2 id="8-what-we-exhibited-2-annna-real">8. What We Exhibited (2): ANNNA REAL</h2><p>ANNNA REAL is a real-time AI human service: speak to it, and a lifelike avatar answers on the spot. Ahead of the official launch, we gave it a preview showing at CEATEC 2026.</p><figure class="kg-card kg-embed-card"><iframe width="200" height="150" src="https://www.youtube.com/embed/YAQrsGTF5ew?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen title="ANNNA REAL: a real-time AI avatar that talks with you like a person (CEATEC 2026 preview)"></iframe></figure><p><strong>What it can do</strong></p><ul><li>Real-time dialogue. It responds on the spot to the question in front of it</li><li>Guidance tailored to a facility. Load it with exhibit content, product features, or how to use the facility, and it explains in response to each question</li><li>Multilingual communication. It can guide international visitors in their own languages</li><li>Avatar, expressions and gestures, and speaking style can all be set to match the location, from polite guidance to a more characterful manner</li></ul><p><strong>Where we expect it to be used</strong></p><p>Trade shows and showrooms, airports, art and science museums, amusement parks and theme parks, stations and tourist facilities, and commercial facilities. The idea is to place a guide you can talk to wherever people gather.</p><p><strong>Relationship to MotionVox</strong></p><p>MotionVox is an application for creating and editing avatar videos. ANNNA REAL is a separate service built on the same engine and made real time.</p><p>A standalone configuration that runs without an internet connection is also available (in this configuration, the naturalness of speech may be reduced).</p><p>For details, please see the <a href="https://annnareal.com/?ref=journal.qualiteg.com" rel="noreferrer">ANNNA REAL website</a>.</p>
<!--kg-card-begin: html-->
<div id="ch9"></div>
<!--kg-card-end: html-->
<h2 id="9-our-commitment-to-in-house-technology">9. Our Commitment to In-House Technology</h2><p>The questions we mentioned in the Day 2 section,</p><p><strong>&quot;Does this use an external API?&quot;</strong></p><p><strong>&quot;Which AI are you using?&quot;</strong></p><p>came up almost every day again this year.</p><p>MotionVox&apos;s expression generation engine, Expression Aware&#xAE;, and its voice expression technology, Voice Wrapper Technology&#xAE;, were developed by Qualiteg and run on our own GPU cluster. ANNNA REAL likewise uses a real-time expression generation engine based on the MotionVox engine for its facial expressions, and our own AI digital human technology for generating the dialogue.</p><p>Our reason for insisting on in-house development has not changed since last year.</p><p>We want to build an environment in which Japanese companies and organizations can use AI with confidence.</p><ul><li>The data we use to create avatars and voices comes only from performers with whom we have signed contracts and obtained the necessary permissions. We never scrape images or voices of people from the internet for training</li><li>Because the engine is our own,<ul><li>We offer a <strong>domestic-server plan</strong> that completes all processing in Japan, without sending input data to servers overseas</li><li>We can also support on-premises configurations that run entirely inside the customer&apos;s environment</li></ul></li></ul><p>In last year&apos;s report we wrote that Japan must not become a mere &quot;AI consumer nation.&quot; This year, too, many visitors told us they are rooting for AI technology that comes from Japan, and our thinking has not changed.</p>
<!--kg-card-begin: html-->
<div id="ch10"></div>
<!--kg-card-end: html-->
<h2 id="10-summary-four-days-in-numbers-and-notes-for-next-year">10. Summary: Four Days in Numbers, and Notes for Next Year</h2><p>Here are the official visitor counts for the four days, side by side with last year&apos;s figures.</p><p>These are the numbers from the Daily News that the organizer (the CEATEC Secretariat) distributed to exhibitors.</p>
<!--kg-card-begin: html-->
<table>
<thead>
<tr><th></th><th>DAY 1, Oct 13 (Tue)</th><th>DAY 2, Oct 14 (Wed)</th><th>DAY 3, Oct 15 (Thu)</th><th>DAY 4, Oct 16 (Fri)</th><th>Total</th></tr>
</thead>
<tbody>
<tr><td>Weather</td><td>Cloudy</td><td>Cloudy, then rain</td><td>Cloudy</td><td>Sunny</td><td></td></tr>
<tr><td>Registered attendance 2026</td><td>25,637</td><td>25,649</td><td>28,397</td><td><strong>32,690</strong></td><td><strong>112,373</strong></td></tr>
<tr><td>Registered attendance 2025</td><td>20,474</td><td>23,433</td><td>24,601</td><td>30,376</td><td>98,884</td></tr>
<tr><td>Of which general visitors 2026</td><td>18,731</td><td>20,550</td><td>23,697</td><td>28,020</td><td>90,998</td></tr>
<tr><td>Of which general visitors 2025</td><td>15,294</td><td>19,013</td><td>20,701</td><td>26,852</td><td>81,867</td></tr>
</tbody>
</table>
<!--kg-card-end: html-->
<p>Note that registered attendance includes press and exhibitors. The 2026 figures are combined with Japan Mobility Show Bizweek 2026.</p><p>Looking at general visitors alone, attendance was up about 11 percent on last year.</p><p>Across the venue, attendance grew 1.5 times from the first day to the last. At our booth the feel was twice Day 1, so our visitors grew faster than the venue as a whole over the four days.</p><p><strong>Our four days in one sentence</strong></p><p><strong>&quot;More visitors than last year, twice as many business meetings, and returning visitors who turned many of them into formal meetings.&quot;</strong></p><p>That was our CEATEC 2026.</p><p><strong>Notes for next year</strong></p><p>For our own benefit next year, here is what we noticed.</p><ul><li>Have English handouts ready from Day 1. International visitors were most numerous on the first day (we do not know why)</li><li>The afternoon of the final day is the peak. Add more staff from 13:00 onward</li><li>Returning visitors often led to (scheduled) business meetings</li><li>The next show, CEATEC 2027, is scheduled for Tuesday, October 19 to Friday, October 22, 2027 at Makuhari Messe (organizer&apos;s announcement)</li></ul><p><strong>Thank you</strong></p><p>To everyone who stopped by our booth, to everyone who came back more than once, and to everyone who cheered us on through social media, thank you from the bottom of our hearts.</p><p>ANNNA REAL is being prepared for its official launch. Together with MotionVox, we welcome demo requests and inquiries at any time.</p><ul><li>MotionVox&#xAE; <a href="https://motionvox.com/?ref=journal.qualiteg.com" rel="noreferrer">https://motionvox.com/</a></li><li>ANNNA REAL <a href="https://annnareal.com/?ref=journal.qualiteg.com" rel="noreferrer">https://annnareal.com/</a> (that is three N&apos;s)</li><li>Qualiteg Inc. <a href="https://qualiteg.com/?ref=journal.qualiteg.com" rel="noreferrer">https://qualiteg.com/</a></li></ul><p>Thank you very much for reading to the end.</p><p>See you next time!</p><h2 id="memories-of-ceatec-2026">Memories of CEATEC 2026</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-23.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1021" height="728" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-23.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-23.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-23.png 1021w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">(This year!?) We arrived right on time</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-16.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1280" height="960" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-16.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-16.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-16.png 1280w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Up to five of us welcomed visitors at the booth this year</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-19.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1920" height="1440" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-19.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-19.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/ceatec2026r_image-19.png 1600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-19.png 1920w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Lunch boxes for the booth staff, pre-ordered with a different menu each day. Delicious</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-17.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1920" height="1440" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-17.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-17.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/ceatec2026r_image-17.png 1600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-17.png 1920w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">The view from the hotel rooms we booked for the team. Plenty of fine weather kept spirits high</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-21.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1506" height="1920" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-21.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-21.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-21.png 1506w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Our &quot;video headaches&quot; board. Visitors&apos; real concerns, made visible</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-28.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1280" height="1034" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-28.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-28.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-28.png 1280w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Makuhari Messe has a high ceiling and an open feel</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-24.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1266" height="1212" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-24.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-24.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-24.png 1266w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Champon noodles as the reward after a long day</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-26.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1280" height="764" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-26.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-26.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-26.png 1280w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Back at the hotel, a soak in the big bath, then a retrospective and improvement ideas while the circulation was still going</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-25.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1920" height="1602" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-25.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-25.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/ceatec2026r_image-25.png 1600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-25.png 1920w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Demo improvements continued late into the night. The late-night snack was the engineer&apos;s classic, fresh-baked pizza</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-27.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1920" height="1080" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-27.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-27.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/ceatec2026r_image-27.png 1600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-27.png 1920w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">A beautiful sunrise. Another day, let&apos;s do our best</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-20.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1185" height="916" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-20.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-20.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-20.png 1185w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">The CEATEC entrance</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-15.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1600" height="1200" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-15.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-15.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-15.png 1600w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Where is Qualiteg? Hint, 3 x 3 = 9</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-14.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1280" height="960" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-14.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-14.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-14.png 1280w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">CEATEC AWARD</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-30.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="1600" height="1200" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-30.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026r_image-30.png 1000w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-30.png 1600w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">The Tokyo University of Science booth. We heard all about life on campus after Professor Kenso Soai won the Nobel Prize. Congratulations! It is also the alma mater of our CEO.</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-18.png" class="kg-image" alt="Qualiteg at CEATEC 2026: Exhibition Report" loading="lazy" width="960" height="1280" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026r_image-18.png 600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026r_image-18.png 960w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">A hall bustling with visitors. We will be back next year with an even more evolved Qualiteg.</span></figcaption></figure><h2 id="sources-and-references">Sources and References</h2><ul><li>JEITA, &quot;CEATEC 2026 closes to great success: registered attendance over the four days reaches 112,373&quot; (October 16, 2026, in Japanese) <a href="https://www.ceatec.com/ja/news/detail.php?id=31&amp;ref=journal.qualiteg.com" rel="noreferrer">https://www.ceatec.com/ja/news/detail.php?id=31</a></li><li>CEATEC Secretariat, &quot;CEATEC Daily News&quot; Vol. 2 to Vol. 4 (distributed to exhibitors; source of the daily visitor counts and weather)</li><li>JEITA, &quot;CEATEC 2025 closes to great success&quot; (daily visitor counts for 2025, in Japanese) <a href="https://www.ceatec.com/ja/news/news20251017.html?ref=journal.qualiteg.com" rel="noreferrer">https://www.ceatec.com/ja/news/news20251017.html</a></li><li>MotionVox <a href="https://motionvox.com/?ref=journal.qualiteg.com" rel="noreferrer">https://motionvox.com/</a></li><li>ANNNA REAL <a href="https://annnareal.com/?ref=journal.qualiteg.com" rel="noreferrer">https://annnareal.com/</a></li></ul>]]></content:encoded></item><item><title><![CDATA[Qualiteg to Exhibit at CEATEC 2026: Experience the Latest Version of Our AI Avatar Video Generator "MotionVox®"]]></title><description><![CDATA[Qualiteg exhibits at CEATEC 2026 (Makuhari Messe, October 13 to 16, Hall 4, booth 4H339) with live demos of MotionVox, our AI avatar video generator: 1,000+ avatars, videos from a script, PowerPoint, selfie video or a single image, and automatic translation into 46 languages.]]></description><link>https://journal.qualiteg.com/ceatec-2026-announcement/</link><guid isPermaLink="false">6acbd2952ead0f114b6f0a74</guid><category><![CDATA[News]]></category><category><![CDATA[Event]]></category><dc:creator><![CDATA[Qualiteg Business Development | Marketing]]></dc:creator><pubDate>Sun, 11 Oct 2026 13:00:00 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/10/cover_ceatec2026_en_v2.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/10/cover_ceatec2026_en_v2.png" alt="Qualiteg to Exhibit at CEATEC 2026: Experience the Latest Version of Our AI Avatar Video Generator &quot;MotionVox&#xAE;&quot;"><p>Qualiteg Inc. will exhibit at CEATEC 2026, held at Makuhari Messe from Tuesday, October 13 to Friday, October 16, 2026. Our booth centers on the latest version of MotionVox&#xAE;, our AI avatar video generation service, with live video generation and finished examples.</p><p>This is our second consecutive year at CEATEC, following <a href="https://journal.qualiteg.com/ceatec-2025-announcement1/" rel="noreferrer">CEATEC 2025</a>. You can read about last year&apos;s booth in our <a href="https://journal.qualiteg.com/ceatec-2025-report/" rel="noreferrer">exhibition report</a>.</p><h2 id="exhibition-overview">Exhibition Overview</h2><ul><li><strong>Dates</strong>&#x3000;Tuesday, October 13 to Friday, October 16, 2026, 10:00 to 17:00</li><li><strong>Venue</strong>&#x3000;Makuhari Messe (2-1 Nakase, Mihama-ku, Chiba-shi, Chiba)</li><li><strong>Area</strong>&#x3000;Next Generation Park</li><li><strong>Booth</strong>&#x3000;Hall 4, <strong>4H339</strong></li><li><strong>Admission</strong>&#x3000;Free (visitor pre-registration required)</li><li>Exhibitor page on the CEATEC site <a href="https://www.ceatec.com/nj/exhibitor_detail_en?id=1303&amp;ref=journal.qualiteg.com" rel="noreferrer">https://www.ceatec.com/nj/exhibitor_detail_en?id=1303</a></li></ul><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/ceatec2026_exhibitor_page_en.png" class="kg-image" alt="Qualiteg to Exhibit at CEATEC 2026: Experience the Latest Version of Our AI Avatar Video Generator &quot;MotionVox&#xAE;&quot;" loading="lazy" width="1736" height="726" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/ceatec2026_exhibitor_page_en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/ceatec2026_exhibitor_page_en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/ceatec2026_exhibitor_page_en.png 1600w, https://journal.qualiteg.com/content/images/2026/10/ceatec2026_exhibitor_page_en.png 1736w" sizes="(min-width: 720px) 720px"></figure><p>Please note that we have moved from Hall 6 last year to Hall 4 this year.</p><h2 id="highlights-from-one-script-to-ai-avatar-videos-in-46-languages">Highlights&#x3000;From one script to AI avatar videos in 46 languages</h2>
<!--kg-card-begin: html-->
<div id="ch1"></div>
<!--kg-card-end: html-->
<h3 id="1-motionvox%C2%AE-our-ai-avatar-video-generation-service">1. MotionVox&#xAE;, our AI avatar video generation service</h3><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/motionvox_banner_2025.png" class="kg-image" alt="Qualiteg to Exhibit at CEATEC 2026: Experience the Latest Version of Our AI Avatar Video Generator &quot;MotionVox&#xAE;&quot;" loading="lazy" width="2000" height="600" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/motionvox_banner_2025.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/motionvox_banner_2025.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/10/motionvox_banner_2025.png 1600w, https://journal.qualiteg.com/content/images/2026/10/motionvox_banner_2025.png 2000w" sizes="(min-width: 720px) 720px"></figure><p>Your words, spoken by a lifelike avatar. Type a script and you get a video with a natural voice and expressions. No filming, no appearing on camera, no voice recording.</p><p>MotionVox&#xAE; is built on our in-house Expression Aware&#xAE; and Voice Wrapper Technology&#xAE;, which deliver <strong>natural facial expressions, lip movements and rich vocal expression</strong>. You can produce training, manual and product videos with far less filming and updating effort.</p>
<!--kg-card-begin: html-->
<video src="https://journal.qualiteg.com/content/media/2026/10/avatar_mosaic.mp4" poster="https://journal.qualiteg.com/content/media/2026/10/avatar_mosaic_thumb.png" autoplay loop muted playsinline preload="metadata" style="width:100%;height:auto;display:block;margin:0 auto;border-radius:6px" aria-label></video>
<!--kg-card-end: html-->
<p><strong>Key features</strong></p><ul><li><strong>Over 1,000</strong> AI avatars and AI voices based on professional voice actors&apos; recordings. Your staff never have to show their face or voice</li><li>One presenter or two in conversation. Assign lines to each character and build a show-style video</li><li>Lip movements, expressions and natural head motion. Pick an emote to add a reaction that fits the scene</li><li>Create from a wide range of sources, including scripts, PowerPoint, selfie videos and a single image</li><li>Expand one script into videos and subtitles in <strong>46 languages</strong>, keeping the same avatar and the same voice across languages</li><li>When the content changes, edit the script and regenerate. No presenters to book, no reshoots, so training and manuals stay current</li></ul><figure class="kg-card kg-embed-card"><iframe width="200" height="113" src="https://www.youtube.com/embed/3oyyI7f11e8?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen title="MotionVox template: News Duo, two AI avatars in conversation"></iframe></figure><p>MotionVox website <a href="https://motionvox.com/?ref=journal.qualiteg.com" rel="noreferrer">https://motionvox.com/</a></p>
<!--kg-card-begin: html-->
<div id="ch2"></div>
<!--kg-card-end: html-->
<h3 id="2-pick-a-template-edit-it-and-your-avatar-video-is-ready">2. Pick a template, edit it, and your avatar video is ready</h3><p>Choose how you want it to look from a wide range of templates, then edit the script, text and images. News, training and store guides take shape right away. No filming or editing skills needed.</p><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/template_news_duo.png" class="kg-image" alt="Qualiteg to Exhibit at CEATEC 2026: Experience the Latest Version of Our AI Avatar Video Generator &quot;MotionVox&#xAE;&quot;" loading="lazy" width="1100" height="619" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/template_news_duo.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/template_news_duo.png 1000w, https://journal.qualiteg.com/content/images/2026/10/template_news_duo.png 1100w" sizes="(min-width: 720px) 720px"></figure><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/template_sf_channel.png" class="kg-image" alt="Qualiteg to Exhibit at CEATEC 2026: Experience the Latest Version of Our AI Avatar Video Generator &quot;MotionVox&#xAE;&quot;" loading="lazy" width="1100" height="619" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/template_sf_channel.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/10/template_sf_channel.png 1000w, https://journal.qualiteg.com/content/images/2026/10/template_sf_channel.png 1100w" sizes="(min-width: 720px) 720px"></figure><p>Training and classroom videos work the same way with an instructor avatar, for onboarding, information security training and sharing work rules.</p><figure class="kg-card kg-embed-card"><iframe width="200" height="113" src="https://www.youtube.com/embed/uKf8xiZieZA?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen title="MotionVox template: Manabi no Jikan, an AI avatar classroom video"></iframe></figure>
<!--kg-card-begin: html-->
<div id="ch3"></div>
<!--kg-card-end: html-->
<h3 id="3-turn-a-powerpoint-file-into-a-video-presented-by-an-avatar-automatically">3. Turn a PowerPoint file into a video presented by an avatar, automatically</h3><p>MotionVox&#xAE; builds a script from your PowerPoint file and generates a video in which a lifelike avatar walks through the slides. Choose a single presenter or two avatars in conversation. Your speaker notes can serve as the script.</p><figure class="kg-card kg-embed-card"><iframe width="200" height="113" src="https://www.youtube.com/embed/VjUVRzlQbig?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen title="Generate a video from PowerPoint: an AI avatar presents your slides"></iframe></figure>
<!--kg-card-begin: html-->
<div id="ch4"></div>
<!--kg-card-end: html-->
<h3 id="4-automatic-translation-and-multilingual-batch-generation-in-46-languages">4. Automatic translation and multilingual batch generation in 46 languages</h3><p>Make the video in Japanese, choose the languages you need, and MotionVox&#xAE; translates the lines, the on-screen text and even the text inside images, then generates the videos in one batch with the same avatar and the same voice. Proper nouns are translated with their reading in mind, so a place name like Shibuya stays &quot;Shibuya&quot; in English and Chinese.</p><p>Below are the Japanese and English versions of a restaurant video made with the &quot;Seasonal Table&quot; template. The same avatar speaks with the same voice.</p><figure class="kg-card kg-embed-card"><iframe width="200" height="113" src="https://www.youtube.com/embed/7Pi0GKRwuGU?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen title="MotionVox multilingual demo: Seasonal Table (Japanese)"></iframe></figure><figure class="kg-card kg-embed-card"><iframe width="200" height="113" src="https://www.youtube.com/embed/AWqD2KU7nTU?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen title="MotionVox multilingual demo: Seasonal Table (English, auto-translated)"></iframe></figure><p>The Chinese version is available <a href="https://www.youtube.com/watch?v=9pmTYWRoSn0&amp;ref=journal.qualiteg.com" rel="noreferrer">here</a>. Inbound tourism guides, overseas product introductions and multilingual training all reach across languages.</p>
<!--kg-card-begin: html-->
<div id="ch5"></div>
<!--kg-card-end: html-->
<h3 id="5-selfie-to-avatar-and-original-avatar-creation">5. Selfie-to-Avatar and original avatar creation</h3><p>Beyond the ready-made avatars, you can cast your own performance, or a character created from a prompt or a photo.</p><p><strong>Selfie-to-Avatar</strong></p><p>Record yourself talking to the camera, and MotionVox&#xAE; transfers your expressions and head movements to the avatar you chose. Your own pacing and nods carry into the video. Voice Wrap lets you change the voice as well.</p><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/selfie_input.png" class="kg-image" alt="Qualiteg to Exhibit at CEATEC 2026: Experience the Latest Version of Our AI Avatar Video Generator &quot;MotionVox&#xAE;&quot;" loading="lazy" width="900" height="517" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/selfie_input.png 600w, https://journal.qualiteg.com/content/images/2026/10/selfie_input.png 900w" sizes="(min-width: 720px) 720px"></figure><p><strong>Original avatar creation</strong></p><p>Generate a character from a prompt, use a single photo, or take a picture in the app. Whichever route you choose, the avatar moves as naturally as if it were alive.</p>
<!--kg-card-begin: html-->
<div id="ch6"></div>
<!--kg-card-end: html-->
<h3 id="6-add-expressions-at-any-moment-with-a-wide-range-of-emotes">6. Add expressions at any moment with a wide range of emotes</h3><p>A nod, a look of surprise, a shy smile, a wink. Give your lifelike avatar the reaction the scene calls for. Pick an emote and MotionVox&#xAE; combines the expression with the head motion, and you can fine-tune the strength and timing on the timeline.</p>
<!--kg-card-begin: html-->
<div style="display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:0 0 8px">
<video src="https://journal.qualiteg.com/content/media/2026/10/emote_doze_wake-kaito_5209_uv6-bc229de29c.mp4" poster="https://journal.qualiteg.com/content/media/2026/10/emote_doze_wake-kaito_5209_uv6-bc229de29c_thumb.jpg" autoplay loop muted playsinline preload="metadata" style="width:100%;height:auto;display:block;border-radius:6px" aria-label="Kaito, dozing off then waking"></video>
<video src="https://journal.qualiteg.com/content/media/2026/10/emote_look_around-tono-943e9947f0.mp4" poster="https://journal.qualiteg.com/content/media/2026/10/emote_look_around-tono-943e9947f0_thumb.jpg" autoplay loop muted playsinline preload="metadata" style="width:100%;height:auto;display:block;border-radius:6px" aria-label="Tono-sama, looking around"></video>
<video src="https://journal.qualiteg.com/content/media/2026/10/emote_nod_listen-rina_0045_uv6-684ac3e030.mp4" poster="https://journal.qualiteg.com/content/media/2026/10/emote_nod_listen-rina_0045_uv6-684ac3e030_thumb.jpg" autoplay loop muted playsinline preload="metadata" style="width:100%;height:auto;display:block;border-radius:6px" aria-label="Rina, a listening nod"></video>
<video src="https://journal.qualiteg.com/content/media/2026/10/emote_nod_once-taiki_0342_uv6-9ce2376781.mp4" poster="https://journal.qualiteg.com/content/media/2026/10/emote_nod_once-taiki_0342_uv6-9ce2376781_thumb.jpg" autoplay loop muted playsinline preload="metadata" style="width:100%;height:auto;display:block;border-radius:6px" aria-label="Taiki, a single nod"></video>
<video src="https://journal.qualiteg.com/content/media/2026/10/emote_puzzled-seira_0048_uv6-5b5aa17614.mp4" poster="https://journal.qualiteg.com/content/media/2026/10/emote_puzzled-seira_0048_uv6-5b5aa17614_thumb.jpg" autoplay loop muted playsinline preload="metadata" style="width:100%;height:auto;display:block;border-radius:6px" aria-label="Seira, tilting her head"></video>
<video src="https://journal.qualiteg.com/content/media/2026/10/emote_surprise-sofia-2701861ec3.mp4" poster="https://journal.qualiteg.com/content/media/2026/10/emote_surprise-sofia-2701861ec3_thumb.jpg" autoplay loop muted playsinline preload="metadata" style="width:100%;height:auto;display:block;border-radius:6px" aria-label="Sofia, surprise"></video>
<video src="https://journal.qualiteg.com/content/media/2026/10/emote_tilt_smile-camille_0206_uv6-bb1c37119e.mp4" poster="https://journal.qualiteg.com/content/media/2026/10/emote_tilt_smile-camille_0206_uv6-bb1c37119e_thumb.jpg" autoplay loop muted playsinline preload="metadata" style="width:100%;height:auto;display:block;border-radius:6px" aria-label="Camille, a tilt and a smile"></video>
<video src="https://journal.qualiteg.com/content/media/2026/10/emote_wink_soft-chloe_0011_uv6-2fc3ad5649.mp4" poster="https://journal.qualiteg.com/content/media/2026/10/emote_wink_soft-chloe_0011_uv6-2fc3ad5649_thumb.jpg" autoplay loop muted playsinline preload="metadata" style="width:100%;height:auto;display:block;border-radius:6px" aria-label="Chloe, a soft wink"></video>
</div>

<!--kg-card-end: html-->

<!--kg-card-begin: html-->
<div id="ch7"></div>
<!--kg-card-end: html-->
<h3 id="7-english-learning-videos-made-entirely-in-motionvox%C2%AE">7. English-learning videos, made entirely in MotionVox&#xAE;</h3><p>&quot;ONE MORE: Listen and Understand English Conversation&quot; combines English and Japanese, two avatars, subtitles and commentary. Listen without subtitles, understand through the Japanese commentary, then listen again with English subtitles. The whole sequence was built in the MotionVox&#xAE; editor alone.</p><figure class="kg-card kg-embed-card"><iframe width="200" height="113" src="https://www.youtube.com/embed/0uhxmXcXV9o?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen title="MotionVox template: ONE MORE, an AI avatar English conversation video"></iframe></figure>
<!--kg-card-begin: html-->
<div id="ch8"></div>
<!--kg-card-end: html-->
<h3 id="8-live-demos-and-examples-at-the-booth">8. Live demos and examples at the booth</h3><p>At the booth we will generate videos from templates in front of you. We will also have examples for training, manuals and product introductions, so please come and judge the visual and voice quality on real hardware.</p><figure class="kg-card kg-image-card"><img src="https://journal.qualiteg.com/content/images/2026/10/template_retail_portrait.png" class="kg-image" alt="Qualiteg to Exhibit at CEATEC 2026: Experience the Latest Version of Our AI Avatar Video Generator &quot;MotionVox&#xAE;&quot;" loading="lazy" width="619" height="1100" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/10/template_retail_portrait.png 600w, https://journal.qualiteg.com/content/images/2026/10/template_retail_portrait.png 619w"></figure><h2 id="what-makes-motionvox%C2%AE-different">What Makes MotionVox&#xAE; Different</h2><h3 id="technology-developed-in-japan-in-house">Technology developed in Japan, in house</h3><p>The expression engine Expression Aware&#xAE; and the voice technology Voice Wrapper Technology&#xAE; behind MotionVox&#xAE; were both developed by Qualiteg.</p><p>At last year&apos;s booth, visitors often asked whether we were using an external API, and were surprised to hear it was our own engine. MotionVox&#xAE; reproduces the subtle changes of expression and the restrained yet rich emotional range characteristic of Japanese speakers.</p><h3 id="avatars-and-voices-with-rights-respected">Avatars and voices with rights respected</h3><p>Every dataset used to build our avatars and voices comes from performers under contract who have given the required permissions. We never scrape people&apos;s images or voices from the internet for training. Generated videos can be used commercially.</p><h3 id="peace-of-mind-on-security">Peace of mind on security</h3><p>We offer a domestic server plan that processes input data entirely in Japan without sending it to overseas servers. Because the video generation technology is our own, an on-premises configuration that runs inside your own environment is also available. Public institutions and companies handling confidential information are welcome to consider it.</p><h3 id="video-production-built-for-updates">Video production built for updates</h3><p>Training materials and manuals are never finished. They keep changing. With MotionVox&#xAE; you edit the script and regenerate, so the content stays current regardless of presenters&apos; schedules or reshoots.</p><p>Turning sales explanations into video is another use. It aligns the message that used to vary from one representative to another.</p><h2 id="what-has-changed-since-last-year">What Has Changed Since Last Year</h2><p>A year has passed since we unveiled Expression Aware v2.0 at CEATEC 2025. Here is how MotionVox&#xAE; has evolved.</p>
<!--kg-card-begin: html-->
<table>
<thead>
<tr><th>Item</th><th>CEATEC 2025 (October 2025)</th><th>CEATEC 2026 (October 2026)</th></tr>
</thead>
<tbody>
<tr><td>Languages</td><td>30+</td><td><strong>46</strong></td></tr>
<tr><td>Inputs</td><td>Text, conversational audio, selfie video</td><td>Script, PowerPoint, selfie video, a single image</td></tr>
<tr><td>Avatars</td><td>Full-body avatars added</td><td><strong>Over 1,000</strong> to choose from, plus original avatars</td></tr>
<tr><td>Starting point</td><td>Write a script and generate</td><td>Pick a template and enter your content</td></tr>
<tr><td>Expressions and gestures</td><td>Lip sync and subtle expressions</td><td>Emotes placed on the timeline</td></tr>
<tr><td>Booth</td><td>Hall 6 (6H207)</td><td>Hall 4 (4H339)</td></tr>
</tbody>
</table>
<!--kg-card-end: html-->
<p>With more languages and more input formats, the person making the video needs even less prior knowledge than before.</p><h2 id="come-visit-our-booth">Come Visit Our Booth</h2><p>Come and experience the latest in AI avatar video generation for yourself.</p><p>While you watch the technology demonstrations, we will show you concretely how it can serve your training, manuals, product introductions and multilingual communication.</p><p>We look forward to welcoming you.</p><h3 id="meeting-reservations">Meeting reservations</h3><p>You can book a meeting at the Makuhari venue or online from our exhibitor page on the CEATEC 2026 site. If you would like an unhurried conversation even during busy hours, we recommend booking in advance.</p><ul><li>Meeting reservations <a href="https://www.ceatec.com/nj/exhibitor_meeting_en?id=1303&amp;ref=journal.qualiteg.com" rel="noreferrer">View available dates (CEATEC 2026 official site)</a></li></ul><h3 id="visitor-information">Visitor information</h3><p>Admission to CEATEC 2026 is free, but pre-registration is required. Create a My Page on the official site and register, then either print your badge at home or show the QR code at the pick-up counter at the venue to receive a visitor badge. The badge is valid for all four days. Same-day registration at the venue is also possible.</p><ul><li><a href="https://www.ceatec.com/en/about/registration.html?ref=journal.qualiteg.com" rel="noreferrer">Visitor registration (CEATEC 2026 official site)</a></li><li><a href="https://www.ceatec.com/en/exhibition/map.html?ref=journal.qualiteg.com" rel="noreferrer">Venue map (CEATEC 2026 official site)</a></li><li><a href="https://www.ceatec.com/en/access/?ref=journal.qualiteg.com" rel="noreferrer">Access (CEATEC 2026 official site)</a></li></ul><h2 id="contact">Contact</h2><p>Qualiteg Inc.</p><ul><li>Email&#x3000;ceatec2026@qualiteg.com</li><li>Website&#x3000;<a href="https://qualiteg.com/?ref=journal.qualiteg.com" rel="noreferrer">https://qualiteg.com/</a></li><li>MotionVox&#xAE;&#x3000;<a href="https://motionvox.com/?ref=journal.qualiteg.com" rel="noreferrer">https://motionvox.com/</a></li><li>Privacy policy&#x3000;<a href="https://qualiteg.com/pp.html?ref=journal.qualiteg.com" rel="noreferrer">https://qualiteg.com/pp.html</a></li></ul><p>Under the theme &quot;Transformation&quot;, CEATEC 2026 brings together people, technology and information from every industry to shape the future through co-creation. At our booth you can see for yourself the new way of communicating that AI avatar video makes possible. We hope to see you there.</p>]]></content:encoded></item><item><title><![CDATA[Building a Coding Agent from Scratch, Part 3: "It Didn't Stop" Is Not "It Made Progress." Count Idle Turns, and Cap the Thinking Without Cutting It]]></title><description><![CDATA[300 turns, zero APIs. Re-reading the log showed 234 turns had never called a tool. We cover the idle rate as a metric, idling that shifts form each time one is closed, how turning thinking off dropped 95 to 71 without saving time, and counting the other side after widening the output window.]]></description><link>https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/</link><guid isPermaLink="false">6ab38d1b2ead0f114b6f0963</guid><category><![CDATA[AI Agents]]></category><category><![CDATA[LLM]]></category><category><![CDATA[IT & AI Technology]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Wed, 07 Oct 2026 19:32:58 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/10/build-coding-agent-from-scratch-part3-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/10/build-coding-agent-from-scratch-part3-en.png" alt="Building a Coding Agent from Scratch, Part 3: &quot;It Didn&apos;t Stop&quot; Is Not &quot;It Made Progress.&quot; Count Idle Turns, and Cap the Thinking Without Cutting It"><p>Hello!</p><p>This is from the period when we had our own coding agent, running on a local LLM, build a web system with authentication.</p><p>It is a record of how we hardened our agent harness so that a small open model could carry the coding all the way through to the end (the goal we were aiming for).</p><p>We ran two models, both ran to the 300-turn limit, and the number of working APIs from the specification was zero.</p><p>At that point I wrote in my notes:</p><p><strong>&quot;From here on, this is a question of the model&apos;s implementation ability.&quot;</strong></p><p>The next day, after re-aggregating the event log from a different angle, I retracted that conclusion.</p><p>Out of 300 turns, there were <strong>234 turns (78%) in which the model never called a tool even once</strong>.</p><p>For the other model it was 218 turns (73%). It had not been running. It had merely been <strong><u>spinning, unable to stop</u></strong>.</p><p>To put the conclusion first,<br>when you measure a coding agent&apos;s autonomy, <strong>the turn count is not a metric</strong>.</p><p>&quot;It didn&apos;t stop&quot; and &quot;it made progress&quot; are entirely different things. What you should look at is &quot;the share of turns that called no tool,&quot; a number we call the <strong>idle rate</strong>.</p><p>In this article we describe, with numbers, what came into view once we started counting idle turns, and how we came to keep reasoning on while only capping it.</p><p>This article is Part 3 of the series &quot;Building a Coding Agent from Scratch.&quot;</p><p>Part 1 covered &quot;separating the turn runner from the stop decider,&quot;<br>and Part 2 covered &quot;the completion gate.&quot;</p><p>This time: how to find runs that pass the gate and yet are &quot;not making progress.&quot;</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Part</th><th>Theme</th></tr></thead><tbody><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Part 1</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Why &quot;finishing the job&quot; is hard. Separating the turn runner from the stop decider</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">Part 2</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">&quot;Done&quot; means &quot;it runs,&quot; not &quot;it&apos;s written.&quot; The completion gate and how to write a pushback</a></td></tr><tr><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">Part 3 (this post)</a></strong></td><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">&quot;It didn&apos;t stop&quot; is not &quot;it made progress.&quot; Count idle turns, and cap them without cutting the thinking</a></strong></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Part 4</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Autonomy rules are for when nobody is around. If a human is present, stop and ask</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Part 5</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Making local LLMs first-class citizens. Getting two 16GB GPUs to build a web system</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">Part 6</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">How to build the evaluation. Have it build something that runs, auto-grade it, and doubt the grader itself</a></td></tr><tr><td>Part 7</td><td>Logging and regression. Everything is found in the event log. And building with zero dependencies</td></tr></tbody></table>
<!--kg-card-end: html-->
<h2 id="1-counting-idle-turns-showed-a-different-cause-for-each-model">1. Counting Idle Turns Showed a Different Cause for Each Model</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig03_1_spin_ratio_v2-en.png" class="kg-image" alt="Building a Coding Agent from Scratch, Part 3: &quot;It Didn&apos;t Stop&quot; Is Not &quot;It Made Progress.&quot; Count Idle Turns, and Cap the Thinking Without Cutting It" loading="lazy"><figcaption>Figure 1: Turns out of 300 that called no tool (diagram: Qualiteg)</figcaption></figure><p>Let me get into some detail here.</p><p>234 turns and 218 turns. The numbers look alike, but what was inside them was completely different.</p><p><strong>The 218-turn case was context overflow.</strong></p><p>Before sending, the harness detected and logged &quot;sending this as-is will exceed the window,&quot; and then <strong>sent it anyway</strong>.</p><p>The comment said &quot;last line of defense against eating a 400 due to estimation drift,&quot; but in practice it stopped nothing. The model returned a 400, one turn was lost, and compaction ran at the head of the next turn. Worse, there were cases where compaction did not resolve the overflow: the design excluded the most recent 30 entries from compaction, so when large tool outputs came in a row, those 30 alone exceeded the window. That is why it overflowed 218 times while compacting 73 times.</p><p><strong>The 234-turn case was repetition of the same response.</strong></p><p>At turn 64 the model stated &quot;all requirements are satisfied and the tests pass,&quot; and after that exactly the same text kept coming back. As described in Part 1, because we had stopped sending pushbacks, the history only gained the same response, and no new information was added to push the model toward its next action. Since we were giving it nothing new, it was only natural that the same output kept coming back.</p><p>In both cases, looking only at the score table, all you see is &quot;300 turns used, 0 APIs.&quot; Only by counting idle turns did two separate holes become visible.</p><p>From here, we set one rule.</p><p><strong>Do not blame the model&apos;s ability for a low score on a run whose idle rate exceeds 30%.</strong></p><p>There was a case where the 4-bit quantized version of a model collapsed into repeating the same words, &quot;Let me create files are files as files are files...&quot;, and hit a 33% idle rate. It looks like insufficient ability, but the cause was the interaction between quantization and long outputs.</p><h2 id="2-close-one-form-of-idling-and-it-moves-to-the-next">2. Close One Form of Idling, and It Moves to the Next</h2><p>Once you start counting idle turns, you next start to see &quot;in what form it is idling.&quot; And every time we closed one, another form appeared.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig03_2_spin_forms-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 3: &quot;It Didn&apos;t Stop&quot; Is Not &quot;It Made Progress.&quot; Count Idle Turns, and Cap the Thinking Without Cutting It" loading="lazy"><figcaption>Figure 2: Forms of idling and their detectors (diagram: Qualiteg)</figcaption></figure><p>The biggest was re-reading. In one run, <strong>65 to 77% of tool calls were file reads</strong>, and the same range of the same file was read <strong>168 times</strong>.</p><p>The pattern was &quot;read, syntax check, read, syntax check&quot; with no edit in between. Since every read succeeds, it does not look like &quot;repeating the same failure,&quot; and we had no metric that named what was happening.</p><p>Why does it cycle? When the result of reading the same range enters the context window again and again, the window fills with duplicates of the same content. Tokens run out sooner, compaction runs, and the old reads are dropped. Since they were dropped, it reads again.</p><p><strong>Re-reading triggers compaction, and compaction triggers re-reading.</strong></p><p>The fix: in the context passed to the model, keep the body of only the most recent read of a given range, and reduce the older ones to a single line saying &quot;superseded by a later read.&quot; And push back if the same range is read three times without any change.</p><p>Once re-reading was closed, the next was TODO rewriting.</p><p>There were stretches where the same TODO was re-saved for 11 consecutive turns without a single line edited. We closed this too, with &quot;push back if three consecutive turns consist only of TODO, waiting, or re-verification.&quot;</p><p>After that, re-applying edits that were already in place. The before and after of the replacement were identical, and the file already contained the replaced string. We changed it to return &quot;that edit is already applied; re-read and move on.&quot;</p><p>One caution here. The judgment &quot;it is repeating the same failure&quot; has to look at <strong>whether it is the same including the contents of the error</strong>. If you count only &quot;exit code is nonzero,&quot; things that are moving forward for a different reason also look like &quot;the same failure,&quot; and the pushback gets in the way.</p><h3 id="the-mechanism-that-manages-context-was-eating-the-context">The mechanism that manages context was eating the context</h3><p>There is one more lesson in the re-reading story. We fell into this hole three times.</p><p>The first was the case in Part 1 where 111 pushbacks piled up and ate the window. The second was the compaction summaries themselves piling up: 70 of them, about 30,500 characters, exceeding the window. The third is this duplication of reads.</p><p>All three times, a mechanism we built to manage the context was eating the context. The pattern of the fix is the same: <strong>leave the log untouched, and thin out only the context passed to the model</strong>. Do not keep pushbacks in the history. Consolidate summaries into one. Fold identical reads into one. Separate the record from what the model is shown. This pattern connects to the event-log discussion in Part 7.</p><h2 id="3-what-varies-is-the-path-not-the-result">3. What Varies Is the &quot;Path,&quot; Not the &quot;Result&quot;</h2><p>After eliminating the idling, running the same task three times on the same model gave this.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Run</th><th>Turns</th><th>Wall time</th><th>Score</th></tr></thead><tbody><tr><td>Run 1</td><td>121</td><td>73 min</td><td>94.9</td></tr><tr><td>Run 2</td><td>40</td><td>17 min</td><td>93.8</td></tr><tr><td>Run 3</td><td>144</td><td>87 min</td><td>96.5</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The turn counts vary by more than a factor of three. The scores stay within &#xB1;1.5 of the three-run average (95.1).</p><p>The same thing happened with another three runs after further harness fixes. Turns varied at 73, 263, and 134, and the scores were 99.2, 94.4, and 93.2. The 263-turn run was not idling; it kept fixing failures in the tests it had written itself, and the harness-side share of time was 4%.</p><p><strong>What varies is the path, not the result.</strong></p><p>Knowing this, our evaluation now includes</p><p><strong>&quot;run the same task three times, and check whether all three scores fall within &#xB1;5 of their average&quot;</strong></p><p>as a pass/fail metric. The spread of turn counts is not a metric. Looking at a single run and saying &quot;this model is slow&quot; or &quot;fast&quot; is only looking at the chance of the path.</p><h2 id="4-turning-off-thinking-dropped-95-points-to-71-and-saved-no-time">4. Turning Off Thinking Dropped 95 Points to 71, and Saved No Time</h2><p>Adjacent to the idling story is how to handle reasoning.</p><p>Our main local model is the type that emits thinking before its response, and it takes 25 to 36 seconds per turn. Assuming it would be faster with thinking off, we ran one run with thinking disabled.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig03_3_thinking_onoff_v2-en.png" class="kg-image" alt="Building a Coding Agent from Scratch, Part 3: &quot;It Didn&apos;t Stop&quot; Is Not &quot;It Made Progress.&quot; Count Idle Turns, and Cap the Thinking Without Cutting It" loading="lazy"><figcaption>Figure 3: Score and wall time per turn with and without thinking (diagram: Qualiteg)</figcaption></figure>
<!--kg-card-begin: html-->
<table><thead><tr><th>Condition</th><th>Score</th><th>Turns</th><th>Wall time</th><th>Model response</th><th>Wall time per turn</th></tr></thead><tbody><tr><td>Thinking on (3 runs)</td><td>94.9 / 93.8 / 96.5</td><td>121 / 40 / 144</td><td>73 / 17 / 87 min</td><td>25 to 36 s/turn</td><td>25 to 36 s</td></tr><tr><td>Thinking off (1 run)</td><td><strong>71.1</strong></td><td>63</td><td>34 min</td><td>17 s/turn</td><td><strong>33 s</strong></td></tr></tbody></table>
<!--kg-card-end: html-->
<p>The model&apos;s response shrank to 17 seconds per turn. However, <strong>wall time per turn was 33 seconds, no different from thinking on</strong>. Most of the difference was the harness&apos;s automated verification: tests ran 58 times in 63 turns, and 24 of those were after read-only or TODO-only turns. The time the model saved simply went into verification.</p><p>And the score fell from 95 to 71. It handled the specification&apos;s &quot;API that returns JSON&quot; and &quot;page that accepts a form&quot; in a single handler without separating them, and reported &quot;done&quot; while still returning 302. None of the three thinking-on runs made this mix-up.</p><p><strong>Thinking was helping not with speed, but with reading the specification.</strong> The default remains thinking on, unchanged.</p><h2 id="5-but-thinking-needs-a-cap">5. But Thinking Needs a Cap</h2><p>After deciding not to cut thinking, the opposite hole appeared.</p><p>It was when we widened the output window from 8192 to 16384.</p><p>The reason for widening is described later, but as a result, <strong>four turns in a single run ran to 16384 tokens on thinking alone and were cut off</strong>.</p><p>About 7 minutes each. That run scored 66.2 and took 130 minutes. A cut-off turn cannot call a single tool, so it costs one extra pushback round trip.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig03_4_thinking_cap-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 3: &quot;It Didn&apos;t Stop&quot; Is Not &quot;It Made Progress.&quot; Count Idle Turns, and Cap the Thinking Without Cutting It" loading="lazy"><figcaption>Figure 4: Don&apos;t cut thinking. Only cap it (diagram: Qualiteg)</figcaption></figure><p>The fix: if thinking alone exceeds the equivalent of 8192 tokens and neither the body nor a tool call has begun, abort that stream and retry <strong>within the same turn</strong> with thinking off. The turn number does not advance. This makes two queries to the model, but in this series&apos; turn counts it counts as one turn. The same task went back to 83.8 points and 31 minutes.</p><p>&quot;Within the same turn&quot; is the key. At first we implemented it as &quot;turn off thinking on the next turn,&quot; but that uses two turns: the cut-off turn and the retry with a pushback attached. Retrying within the same turn takes one turn and needs no pushback.</p><h2 id="6-widen-the-output-window-and-count-the-other-side">6. Widen the Output Window, and Count the Other Side</h2><p>The reason we widened the output window to 16384 was to allow a 39KB file to be written in one go. At 8192, a 27KB bulk write was cut off midway and had to be steered into splitting.</p><p>With the wider window, 39KB did indeed fit in one go. Two things happened in exchange.</p><p>One was the runaway thinking described above. The other was that <strong>compaction moved earlier, from the 60s to turn 27</strong>.</p><p>Tracing the cause: tool results (tool_result) were being thinned out as they aged, but nobody was thinning the tool call arguments (tool_use, that is, the bodies of the files written). The 39KB write stayed in the history, and the input grew by 8 to 10k tokens in a single turn. Dropping the body of writes older than the most recent 20 entries and larger than 1000 characters, passing only the shape &quot;wrote this size to this file,&quot; settled it.</p><p><strong>When you widen a window, always count the consumption on the other side (input and time).</strong> This lesson paid off elsewhere too. A context-overflow error is not &quot;a provider outage&quot; but &quot;our input was too large,&quot; so instead of retrying, compact and resend. The same 400 gets different handling.</p><h2 id="7-until-we-broke-down-the-time-everything-looked-like-the-model-is-slow">7. Until We Broke Down the Time, Everything Looked Like &quot;the Model Is Slow&quot;</h2><p>Finally, one more story of the same kind as idling: the breakdown of wall time.</p><p>In one run, 45% of wall time was the harness&apos;s automated verification. The model was not slow; we were running tests too often. Only after producing the breakdown did the cause split into five. It ran even on turns with no changes. It ran on every write. It ran after read-only commands. It waited 120 seconds every time for a hung test. And it ran after commands that changed nothing.</p><p>The last one took time to find. Commands like &quot;start the server&quot; or &quot;run the custom CLI&quot; read, from their wording, as &quot;could change something.&quot; But not a byte of source had changed. We stopped guessing from the wording, compared the source fingerprint (size and modification time) before and after the command, and ran verification only when something had actually changed. On a task with a background-process server, harness-side time fell from 19% to 4%.</p><p><strong>An improvement whose numbers you cannot decompose becomes guesswork.</strong> For both the idle rate and the time breakdown, every time we added one more angle of aggregation, one more cause became visible.</p><h2 id="summary-measure-progress-not-turn-count">Summary: Measure Progress, Not Turn Count</h2>
<!--kg-card-begin: html-->
<table><thead><tr><th>Number to watch</th><th>How to read it</th></tr></thead><tbody><tr><td>Idle rate (share of turns that called no tool)</td><td>Above 30%, suspect the harness. Don&apos;t blame the model for a low score</td></tr><tr><td>Share of reads, and the max read count of one range</td><td>65 to 77% and 168 reads means a re-reading cycle</td></tr><tr><td>Consecutive turns of only TODO, waiting, or re-verification</td><td>Spinning on plan rewrites</td></tr><tr><td>Scores of three runs on the same task</td><td>If within &#xB1;5 of the average, the spread in turns is a matter of path</td></tr><tr><td>Breakdown of wall time (model / tools / verification)</td><td>If verification exceeds 20%, recount the triggers</td></tr><tr><td>Thinking</td><td>Don&apos;t cut it. At the cap, retry within the same turn with thinking off</td></tr><tr><td>Output window</td><td>After widening, watch the input-side growth and the turn at which compaction starts</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Next time: the flip side of this &quot;don&apos;t stop&quot; discipline.</p><p>The &quot;don&apos;t stop&quot; we trained in evaluation runs was also being applied in situations where a human could answer. Pushing back a model that asked about a vague instruction and making it build on its own; making it ask a declined confirmation seven times in different forms. We will describe the 19 cases that surfaced when we role-played six kinds of usage.</p><p>See you next time!</p><h2 id="related-articles">Related Articles</h2><ul><li><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Building a Coding Agent from Scratch, Part 1: Why &quot;Finishing the Job&quot; Is Hard, and Separating the Turn Runner from the Stop Decider</a></li><li><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">Building a Coding Agent from Scratch, Part 2: &quot;Done&quot; Means &quot;It Runs,&quot; Not &quot;It&apos;s Written&quot;</a></li><li><a href="https://journal.qualiteg.com/about_paged_attention_for_llm_serving/">PagedAttention for More Efficient LLM Serving</a></li></ul>]]></content:encoded></item><item><title><![CDATA[Ubuntu's Automatic Updates Bumped the Kernel Alone, and the GPU Vanished After a Reboot: Why the NVIDIA Module Gets Left Behind, and Three Ways to Prevent It]]></title><description><![CDATA[Reboot an Ubuntu 24.04 GPU server and nvidia-smi stops working. The cause: unattended-upgrades updated the kernel alone and skipped the NVIDIA module over a dependency it could not resolve. How we traced it in the logs, recovered with one install, and compared three fixes including apt-mark hold.]]></description><link>https://journal.qualiteg.com/ubuntu-kernel-update-nvidia-driver-lost/</link><guid isPermaLink="false">6ab67ba52ead0f114b6f0a0e</guid><category><![CDATA[GPU]]></category><category><![CDATA[Generative AI Frontlines]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Mon, 05 Oct 2026 08:50:33 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/10/ubuntu-kernel-update-nvidia-driver-lost-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/10/ubuntu-kernel-update-nvidia-driver-lost-en.png" alt="Ubuntu&apos;s Automatic Updates Bumped the Kernel Alone, and the GPU Vanished After a Reboot: Why the NVIDIA Module Gets Left Behind, and Three Ways to Prevent It"><p>Hello!</p><p>You reboot an Ubuntu GPU server, and <code>nvidia-smi</code> no longer works. <code>modprobe nvidia</code> says &quot;Module nvidia not found&quot;. Two GPUs that were perfectly visible yesterday are gone, and all you did was reboot.</p><p>This happened recently on one of our in-house GPU machines (Ubuntu 24.04.4 LTS, two RTX A5000 cards). The short version: the culprit was Ubuntu&apos;s automatic security updates (unattended-upgrades). Only the kernel was bumped to a new version, the NVIDIA module for that kernel was left behind, and the machine was rebooted in that state.</p><p>In this article I walk through the whole thing with excerpts from the actual logs: confirming the symptom, pinning down the cause, recovering on the spot, and comparing three ways to keep it from happening again, including which one we chose. This is for people who install Ubuntu&apos;s own NVIDIA driver packages (the <code>nvidia-driver-5xx</code> family) with apt. If you use NVIDIA&apos;s official <code>.run</code> installer, the mechanics are different and out of scope here.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig1-en.png" class="kg-image" alt="Ubuntu&apos;s Automatic Updates Bumped the Kernel Alone, and the GPU Vanished After a Reboot: Why the NVIDIA Module Gets Left Behind, and Three Ways to Prevent It" loading="lazy" width="1920" height="1080" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/09/ubuntu-kernel-nvidia-fig1-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/09/ubuntu-kernel-nvidia-fig1-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/09/ubuntu-kernel-nvidia-fig1-en.png 1600w, https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig1-en.png 1920w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 1. How automatic updates moved only the kernel forward, and the next day&apos;s reboot made the GPU disappear</span></figcaption></figure><h2 id="1-the-symptom-a-plain-reboot-makes-the-gpu-disappear">1. The symptom: a plain reboot makes the GPU disappear</h2><p>Nothing was wrong before the reboot. We power-cycled the machine as a startup test before putting a GPU worker on it, and the driver did not come up.</p><pre><code class="language-text">$ sudo modprobe nvidia
Module nvidia not found in /lib/modules/7.0.0-34-generic</code></pre><p>The thing to look at here is the directory name. <code>/lib/modules/7.0.0-34-generic</code>. The running kernel is <code>7.0.0-34</code>. According to our records, this machine was set up on <code>7.0.0-28</code> and had since moved through <code>-29</code>, <code>-30</code>, and <code>-31</code> via automatic updates. <code>-34</code> was new. When did it arrive?</p><pre><code class="language-bash">$ uname -r
7.0.0-34-generic

$ ls /lib/modules/
7.0.0-28-generic  7.0.0-29-generic  7.0.0-30-generic  7.0.0-31-generic  7.0.0-34-generic  kernel</code></pre><p>Kernel <code>-34</code> is installed and is the one being booted. But <code>-34</code>&apos;s module directory has no NVIDIA module in it. The previous kernel, <code>-31</code>, does.</p><pre><code class="language-bash">$ ls /lib/modules/7.0.0-31-generic/kernel/ | grep nvidia
nvidia-595-open</code></pre><p>Run the same command against <code>-34</code> at this point and nothing comes back. That is exactly what the <code>modprobe</code> error is telling you (after recovery, <code>-34</code> also shows <code>nvidia-595-open</code>).</p><p>As for why <code>ls /lib/modules/</code> still lists directories for older versions: removing a kernel package leaves the index files behind. <code>-28</code> through <code>-30</code> had already been removed by automatic updates, both the kernel itself (<code>linux-image-7.0.0-XX-generic</code>) and the NVIDIA module (<code>linux-modules-nvidia-595-open-7.0.0-XX-generic</code>), and <code>dpkg -l</code> shows both as <code>rc</code> (package removed, only configuration files remaining). The kernels actually installed at this point were the two that <code>dpkg -l</code> lists as <code>ii</code>: <code>-31</code> and <code>-34</code>.</p><p>That explains the symptom. The kernel is new, but the NVIDIA module for that kernel is not installed.</p><h2 id="2-ubuntus-nvidia-driver-works-as-a-kernel-and-module-pair">2. Ubuntu&apos;s NVIDIA driver works as a kernel-and-module pair</h2><p>Before getting into the cause, let&apos;s start with the basics.</p><p>Ubuntu&apos;s own NVIDIA driver is made of two large parts. One is the user-space set headed by <code>nvidia-driver-595-open</code> (libraries, utilities, <code>nvidia-smi</code>, and so on). The other is the kernel module, which comes as <code>linux-modules-nvidia-595-open-7.0.0-34-generic</code>, that is, <strong>a separate package for each kernel version</strong>. The kernel module is built specifically for that kernel, so when the kernel moves to <code>-34</code>, you need the module package for <code>-34</code>.</p><p>The metapackage that automatically tracks these per-version modules is <code>linux-modules-nvidia-595-open-generic-hwe-24.04</code>. The kernel side has a metapackage with the same role, <code>linux-image-generic-hwe-24.04</code>, and normally these two are updated together, so a new kernel and its new module arrive as a pair.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig2-en.png" class="kg-image" alt="Ubuntu&apos;s Automatic Updates Bumped the Kernel Alone, and the GPU Vanished After a Reboot: Why the NVIDIA Module Gets Left Behind, and Three Ways to Prevent It" loading="lazy" width="1920" height="1080" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/09/ubuntu-kernel-nvidia-fig2-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/09/ubuntu-kernel-nvidia-fig2-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/09/ubuntu-kernel-nvidia-fig2-en.png 1600w, https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig2-en.png 1920w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 2. The user-space set and the per-kernel-version modules. The two metapackages are normally updated together</span></figcaption></figure><p>This time, the pair broke.</p><h2 id="3-the-cause-automatic-updates-bumped-the-kernel-and-skipped-the-module">3. The cause: automatic updates bumped the kernel and skipped the module</h2><p>Everything that was installed, and when, is in <code>/var/log/dpkg.log</code>. Here are the install times of the kernel and the module for the last four automatic update runs (dates omitted, times only).</p><pre><code class="language-text"># three runs ago
06:50:26 install linux-image-7.0.0-29-generic
06:50:42 install linux-modules-nvidia-595-open-7.0.0-29-generic
# two runs ago
06:18:15 install linux-image-7.0.0-30-generic
06:18:31 install linux-modules-nvidia-595-open-7.0.0-30-generic
# previous run
06:08:41 install linux-image-7.0.0-31-generic
06:08:41 install linux-modules-nvidia-595-open-7.0.0-31-generic
# this run
06:41:57 install linux-image-7.0.0-34-generic
(the module for -34 was not installed here)</code></pre><p><code>-29</code>, <code>-30</code>, and <code>-31</code>, the kernel and the module were installed together within the same minute. <code>-34</code> is the only one where the kernel came alone.</p><p>So what happened at 06:41 on the morning this update ran? Let&apos;s look at the unattended-upgrades log.</p><pre><code class="language-text">06:41:53 INFO Starting unattended upgrades script
06:41:53 INFO Allowed origins are: o=Ubuntu,a=noble, o=Ubuntu,a=noble-security, ...
06:41:55 WARNING package linux-modules-nvidia-595-open-generic-hwe-24.04 upgradable but fails to be marked
         for upgrade (E:Unable to correct problems, you have held broken packages.)
06:41:56 INFO Packages that will be upgraded: linux-generic-hwe-24.04 linux-headers-generic-hwe-24.04
         linux-image-generic-hwe-24.04 linux-libc-dev linux-tools-common
06:42:13 INFO All upgrades installed
06:42:23 INFO Package linux-modules-nvidia-595-open-generic-hwe-24.04 is kept back because
         a related package is kept back or due to local apt_preferences(5).</code></pre><p>It says so plainly. The NVIDIA module metapackage is &quot;upgradable but fails to be marked for upgrade&quot; because its dependencies cannot be resolved, and then only the kernel metapackage is upgraded. From unattended-upgrades&apos; point of view, the module cannot be installed, the kernel can, so it installed the one it could.</p><p>Why couldn&apos;t the module&apos;s dependencies be resolved? The package dependencies tell the story.</p><pre><code class="language-bash">$ apt-cache show linux-modules-nvidia-595-open-7.0.0-34-generic | grep -E &quot;^(Version|Depends)&quot;
Version: 7.0.0-34.34~24.04.1+1
Depends: linux-image-7.0.0-34-generic | linux-image-unsigned-7.0.0-34-generic,
         nvidia-kernel-common-595 (&lt;= 595.91.07-1), nvidia-kernel-common-595 (&gt;= 595.91.07)

$ apt-cache policy nvidia-kernel-common-595
     595.91.07-0ubuntu0.24.04.1 500
        500 http://jp.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Packages
     595.84-0ubuntu0.24.04.1 500
        500 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Packages</code></pre><p>This is the key point.</p><p><strong><code>-34</code>The NVIDIA module for <code>nvidia-kernel-common-595</code> requires 595.91.07 or newer and 595.91.07-1 or older. But 595.91.07 exists only in noble-updates; noble-security goes no further than 595.84.</strong></p><p><code>nvidia-kernel-common-595</code> is the shared kernel-side component of the driver suite, and it ships at the same version as the driver proper (<code>nvidia-driver-595-open</code>). This machine had 595.84, which could not satisfy the required 595.91.07.</p><p>That it was 595.84 is confirmed by the recovery entry in dpkg.log: <code>upgrade nvidia-kernel-common-595 595.84-0ubuntu0.24.04.1 595.91.07-0ubuntu0.24.04.1</code>. Kernel <code>-34</code>, on the other hand, was also published in noble-security.</p><pre><code class="language-bash">$ apt-cache policy linux-image-7.0.0-34-generic
 *** 7.0.0-34.34~24.04.1 500
        500 http://jp.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages
        100 /var/lib/dpkg/status</code></pre><p>Our GPU machines run with &quot;install only security updates automatically&quot; (close to the unattended-upgrades default). That is why noble-updates is absent from the &quot;Allowed origins&quot; line in the log. Kernel <code>-34</code> was in noble-security, so it goes in. But the <code>-34</code> module requires <code>nvidia-kernel-common-595</code> 595.91.07, which exists only in noble-updates, so the dependency cannot be satisfied within the allowed origins. Matching this machine&apos;s log against the dependencies, that appears to be what &quot;fails to be marked for upgrade&quot; means here. The result is that only the kernel moved forward.</p><p>Note that the <code>E:Unable to correct problems, you have held broken packages.</code> in the log is apt&apos;s generic message for a failed dependency resolution. The wording alone does not prove that the cause is &quot;a dependency outside the allowed origins&quot;. In this case it is consistent with the fact that the candidate version of the dependency was not available from the allowed origins, and that is how I read it.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig3-en.png" class="kg-image" alt="Ubuntu&apos;s Automatic Updates Bumped the Kernel Alone, and the GPU Vanished After a Reboot: Why the NVIDIA Module Gets Left Behind, and Three Ways to Prevent It" loading="lazy" width="1920" height="1080" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/09/ubuntu-kernel-nvidia-fig3-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/09/ubuntu-kernel-nvidia-fig3-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/09/ubuntu-kernel-nvidia-fig3-en.png 1600w, https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig3-en.png 1920w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 3. With security as the only allowed origin, a dependency that exists only in noble-updates cannot be resolved, and the kernel alone moves forward</span></figcaption></figure><p>Putting it in order, the chain of events looks like this.</p>
<!--kg-card-begin: html-->
<table>
<thead><tr><th>Step</th><th>What happened</th><th>Where to confirm it</th></tr></thead>
<tbody>
<tr><td>1</td><td>Kernel 7.0.0-34 was published as a security update</td><td><code>apt-cache policy linux-image-7.0.0-34-generic</code> points at noble-security</td></tr>
<tr><td>2</td><td>The NVIDIA module for -34 was built to require <code>nvidia-kernel-common-595</code> 595.91.07</td><td>The Depends line from <code>apt-cache show</code></td></tr>
<tr><td>3</td><td><code>nvidia-kernel-common-595</code> 595.91.07 exists only in noble-updates</td><td><code>apt-cache policy nvidia-kernel-common-595</code></td></tr>
<tr><td>4</td><td>unattended-upgrades, allowed to use security only, skipped the module as &quot;dependencies cannot be resolved&quot; and upgraded the kernel alone</td><td>The WARNING and &quot;Packages that will be upgraded&quot; lines in unattended-upgrades.log</td></tr>
<tr><td>5</td><td>The old kernel keeps running until a reboot, so nothing shows on the surface</td><td><code>uname -r</code> stays at -31 until the reboot</td></tr>
<tr><td>6</td><td>The next day&apos;s reboot brought up -34, and with no module the GPU disappeared</td><td>Module not found from <code>modprobe nvidia</code></td></tr>
</tbody>
</table>
<!--kg-card-end: html-->
<p>For <code>-29</code> through <code>-31</code>, by contrast, the module had been built to require <code>nvidia-kernel-common-595</code> 595.84, and 595.84 was available in noble-security, so the two could be installed together.<code>-34</code> is the release where &quot;the module rebuild became tied to a newer driver version&quot;, and that appears to be why it broke only this time. I have not found an official explanation of why the tie-in changed at that particular point (unconfirmed).</p><p>Incidentally, this machine does not have automatic reboots enabled (<code>Automatic-Reboot &quot;false&quot;</code>). That is deliberate, so that GPU jobs are never killed by a surprise reboot. This time it worked in the direction of &quot;the kernel-only state does not show until a reboot&quot;. The setting itself is not wrong, but the machine kept running in a state where the GPU would vanish the moment it was next rebooted.</p><h2 id="4-recovery-install-the-missing-module-no-reboot-was-needed-this-time">4. Recovery: install the missing module. No reboot was needed this time</h2><p>Once the cause is known, recovery is not hard. Install the missing module metapackage by hand and load the module. A manual <code>apt-get install</code> is not bound by the allowed-origins restriction, so the driver suite including <code>nvidia-kernel-common-595</code> moves up to 595.91.07 along with it.</p><pre><code class="language-bash">$ sudo apt-get install linux-modules-nvidia-595-open-generic-hwe-24.04
(nvidia-driver-595-open and the rest of the driver suite go from 595.84 to 595.91.07,
  and linux-modules-nvidia-595-open-7.0.0-34-generic is newly installed)

$ sudo modprobe nvidia
$ nvidia-smi --query-gpu=name,driver_version --format=csv,noheader
NVIDIA RTX A5000, 595.91.07
NVIDIA RTX A5000, 595.91.07</code></pre><p>This time the module loaded straight away with <code>modprobe</code> after it was installed, so the GPUs came back without a reboot. The incident itself was over in about ten minutes.</p><p>The real problem starts here. The same thing can happen at the next kernel update.</p><h2 id="5-two-issues-the-scope-of-automatic-updates-and-checking-before-a-reboot">5. Two issues: the scope of automatic updates, and checking before a reboot</h2><p>Breaking this incident down, there are two separate issues.</p><p>The first is that automatic updates are able to treat the kernel and the NVIDIA module separately. Two things that must move as a pair are allowed to be updated one at a time. That gap has to be closed somehow.</p><p>The second is that nobody checked, before rebooting, whether the kernel about to boot had an NVIDIA module. What automatic updates installed is not in the head of the person pressing the reboot button. Without a check in the procedure, no matter how good the countermeasure is, you will eventually fall into the same hole for some other reason.</p><p>The second issue is solved by adding one line to the procedure, which I cover later. For the first, there were three options.</p><h2 id="6-comparing-three-ways-to-prevent-a-recurrence">6. Comparing three ways to prevent a recurrence</h2><p>These are the three we considered.</p>
<!--kg-card-begin: html-->
<table>
<thead><tr><th>Option</th><th>What you do</th><th>Upside</th><th>Concern</th></tr></thead>
<tbody>
<tr><td>(1) Take the kernel and the module out of automatic updates</td><td>Pin the kernel and module metapackages with <code>apt-mark hold</code>, and have a person upgrade them as a pair during maintenance</td><td>Closes the path where the two move separately. Predictable behavior. One line to set, reversible at any time</td><td>Kernel security updates stop too. Maintenance has to be scheduled so they are not forgotten</td></tr>
<tr><td>(2) Add noble-updates to the automatic update origins</td><td>Add <code>noble-updates</code> to the allowed origins of unattended-upgrades so the driver the module depends on is installed automatically too</td><td>Keeps tracking with no manual work. Given the candidate versions and dependencies at the time, this very likely would have prevented the incident</td><td>Widens what automatic updates touch, so more changes happen without you. Does not help if the module is published later than the kernel</td></tr>
<tr><td>(3) Switch to DKMS</td><td>Use <code>nvidia-dkms-595-open</code> so the module is built on the machine whenever a new kernel is installed</td><td>No longer depends on when Ubuntu publishes prebuilt modules</td><td>Changes how the driver is installed, so it needs migration work and re-verification. With Secure Boot enabled, module signing and key enrollment are required too. Build failures are a separate risk</td></tr>
</tbody>
</table>
<!--kg-card-end: html-->
<figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig4-en.png" class="kg-image" alt="Ubuntu&apos;s Automatic Updates Bumped the Kernel Alone, and the GPU Vanished After a Reboot: Why the NVIDIA Module Gets Left Behind, and Three Ways to Prevent It" loading="lazy" width="1920" height="1080" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/09/ubuntu-kernel-nvidia-fig4-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/09/ubuntu-kernel-nvidia-fig4-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/09/ubuntu-kernel-nvidia-fig4-en.png 1600w, https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig4-en.png 1920w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 4. The three options at a glance. We chose (1)</span></figcaption></figure><p>Let me dig into each one a little further.</p><h3 id="the-point-of-1-is-buying-a-state-where-you-know-what-will-happen">The point of (1) is buying a state where you know what will happen</h3><p>A hold is a mark that tells apt not to upgrade that package. unattended-upgrades respects holds, so if you mark both the kernel metapackage and the module metapackage, neither moves on its own. When you do upgrade, a person upgrades both together, which closes the path where only one side moves ahead, as happened this time. Keep in mind that what gets pinned is these two metapackages; this is not a blanket freeze of every kernel- or NVIDIA-related package.</p><p>The trade-off is that kernel security updates no longer arrive automatically. Whether that matters depends on the machine. For a server directly exposed to the internet it is a serious minus. Our GPU machines sit on an internal LAN and cannot be reached from outside, so we judged the impact to be small. Every other update still comes in automatically as before.</p><h3 id="the-point-of-2-is-whether-to-widen-the-scope-of-automatic-updates">The point of (2) is whether to widen the scope of automatic updates</h3><p>For this particular case, if noble-updates had been allowed, <code>nvidia-kernel-common-595</code> 595.91.07 would have been a candidate too, and the module would very likely have gone in with it. In that sense it is the most direct answer to the cause.</p><p>But allowing noble-updates widens automatic updates to non-security changes as well (libraries, tools, the whole driver suite). On a GPU server, drivers and libraries changing without your knowledge is its own source of trouble. And this approach still does not cover the pattern where the kernel is published first and the module follows days later. It would have been fine this time only because both happened to be published together; it is not a mechanism that structurally guarantees the pair.</p><h3 id="the-point-of-3-is-building-it-yourself-instead-of-relying-on-distribution">The point of (3) is building it yourself instead of relying on distribution</h3><p>DKMS builds the module from source on the machine whenever a new kernel is installed. It stops depending on when Ubuntu publishes prebuilt modules, so it tracks kernel updates best of the three.</p><p>On the other hand, it means migrating from the <code>linux-modules-nvidia</code> family currently installed to the <code>nvidia-dkms</code> family. Changing the driver setup also calls for verification that involves reboots. Building needs headers and a compiler, and a kernel change breaking the build is not a zero risk. On top of that, with Secure Boot enabled you have to sign the locally built module and enroll a key (MOK enrollment). This machine has Secure Boot enabled (<code>mokutil --sb-state</code> reports SecureBoot enabled) and is running on Ubuntu&apos;s signed prebuilt modules, so there was no reason to disturb that. For a machine in stable production, the surgery felt far too large for the problem being fixed.</p><h2 id="7-we-chose-1">7. We chose (1)</h2><p>Two things drove the decision: for a GPU machine, staying up is the top priority, and it is not exposed to the outside. So we chose (1), where kernel updates happen as a pair, when a person intends them.</p><pre><code class="language-bash">$ sudo apt-mark hold linux-image-generic-hwe-24.04 linux-modules-nvidia-595-open-generic-hwe-24.04

$ apt-mark showhold
containerd.io
docker-buildx-plugin
docker-ce
docker-ce-cli
docker-compose-plugin
linux-image-generic-hwe-24.04
linux-modules-nvidia-595-open-generic-hwe-24.04</code></pre><p>The Docker packages were already on hold because our standard configuration pins their versions; that is unrelated to this story.</p><p>When a person upgrades the held metapackages, remove the holds, run a <code>-s</code> (simulation) first to see what would be upgraded, then upgrade both together and put the holds back. We do this alongside planned downtime or configuration changes on the machine. Even on an internal LAN, delayed kernel fixes are still a real trade-off, so it is on our maintenance checklist to keep the interval from growing too long.</p><pre><code class="language-bash">$ sudo apt-mark unhold linux-image-generic-hwe-24.04 linux-modules-nvidia-595-open-generic-hwe-24.04
$ sudo apt-get update
$ sudo apt-get install -s linux-image-generic-hwe-24.04 linux-modules-nvidia-595-open-generic-hwe-24.04   # review what would be upgraded
$ sudo apt-get install linux-image-generic-hwe-24.04 linux-modules-nvidia-595-open-generic-hwe-24.04
$ sudo apt-mark hold linux-image-generic-hwe-24.04 linux-modules-nvidia-595-open-generic-hwe-24.04</code></pre><p>Now for the second issue, the check before rebooting. We added this regardless of which option was chosen. There is exactly one thing to look at: whether the kernel that will boot next has an NVIDIA module in a loadable location.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig5-en.png" class="kg-image" alt="Ubuntu&apos;s Automatic Updates Bumped the Kernel Alone, and the GPU Vanished After a Reboot: Why the NVIDIA Module Gets Left Behind, and Three Ways to Prevent It" loading="lazy" width="1920" height="1080" srcset="https://journal.qualiteg.com/content/images/size/w600/2026/09/ubuntu-kernel-nvidia-fig5-en.png 600w, https://journal.qualiteg.com/content/images/size/w1000/2026/09/ubuntu-kernel-nvidia-fig5-en.png 1000w, https://journal.qualiteg.com/content/images/size/w1600/2026/09/ubuntu-kernel-nvidia-fig5-en.png 1600w, https://journal.qualiteg.com/content/images/2026/09/ubuntu-kernel-nvidia-fig5-en.png 1920w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Figure 5. Check with modinfo against the kernel that GRUB&apos;s first entry points at, then reboot</span></figcaption></figure><p>&quot;The kernel that will boot next&quot; is decided by GRUB&apos;s default entry. Ubuntu&apos;s default is <code>GRUB_DEFAULT=0</code>, which boots the first entry in the generated menu. To see which kernel the first entry points at, read the <code>grub.cfg</code> line in the first <code>menuentry</code> of <code>linux</code>. We do not use the &quot;newest version on disk&quot; approach, because there is no guarantee it matches the first entry.</p><pre><code class="language-bash">$ grep ^GRUB_DEFAULT /etc/default/grub
GRUB_DEFAULT=0

$ sudo awk &apos;/^menuentry/{c++} c==1 &amp;&amp; /^[[:space:]]*linux[[:space:]]/ {print $2; exit}&apos; /boot/grub/grub.cfg
/boot/vmlinuz-7.0.0-34-generic

$ target=7.0.0-34-generic          # take the version from the output above
$ modinfo -k &quot;$target&quot; -F filename nvidia
/lib/modules/7.0.0-34-generic/kernel/nvidia-595-open/nvidia.ko</code></pre><p><code>GRUB_DEFAULT=saved</code> to remember the last choice, use the entry that <code>grub-editenv list</code>&apos;s <code>saved_entry</code> points at. If a one-time boot entry has been set with <code>grub-reboot</code>, that takes precedence (it shows up in <code>grub-editenv list</code> as <code>next_entry</code>), so use that entry instead.</p><p><code>modinfo -k</code> actually goes looking for the module for the specified kernel version. If it returns a path, that confirms the <code>nvidia</code> module for that kernel is where it should be. It does not guarantee the module will load, so the procedure runs through to <code>nvidia-smi</code> after the reboot. If the module is missing, you get this.</p><pre><code class="language-bash">$ modinfo -k 7.0.0-28-generic -F filename nvidia
modinfo: ERROR: Module nvidia not found.</code></pre><p>The <code>-28</code> in this example is, as mentioned in section 1, a kernel whose image and NVIDIA module were both removed by automatic updates (both <code>rc</code>), leaving only index files in the directory. It is naturally missing because it was removed, but the appearance, &quot;the directory exists yet the module does not&quot;, is the same as <code>-34</code> was during the incident.</p><p>If you get ERROR, install the module package for that kernel version, <code>linux-modules-nvidia-595-open-&lt;target version&gt;</code>, and then reboot. If the target is the newest kernel, installing the HWE metapackage as in section 4 amounts to the same thing. After the reboot, confirm with <code>nvidia-smi</code> that the GPUs are visible. We made this check a mandatory step in the reboot procedure for our GPU machines.</p><p>If you would rather track new kernels automatically, there is also the combination of skipping the hold, choosing (2), and still always running the pre-reboot check. Whichever you choose, the pre-reboot check is the last line of defense.</p><h2 id="summary-before-rebooting-a-gpu-machine-after-a-kernel-update-check-for-the-module-first">Summary: before rebooting a GPU machine after a kernel update, check for the module first</h2><p>Here is a table of where to look and what to do, by situation.</p>
<!--kg-card-begin: html-->
<table>
<thead><tr><th>Situation</th><th>Where to look</th><th>What to do</th></tr></thead>
<tbody>
<tr><td><code>nvidia-smi</code> does not work after a reboot</td><td>The kernel version in the <code>modprobe nvidia</code> error, and <code>ls /lib/modules/&lt;version&gt;/kernel/ | grep nvidia</code></td><td>If it is missing, install <code>linux-modules-nvidia-&lt;ver&gt;-open-&lt;kernel version&gt;</code> for that version (or the <code>&#x2026;-generic-hwe-24.04</code> metapackage if it is the newest kernel) and load it with <code>modprobe nvidia</code></td></tr>
<tr><td>You want to know why only one side was upgraded</td><td>The WARNING and &quot;Packages that will be upgraded&quot; lines in <code>/var/log/unattended-upgrades/unattended-upgrades.log</code></td><td>Match the Depends line from <code>apt-cache show</code> for the skipped package against <code>apt-cache policy</code> for its dependency. If the only candidates are outside the allowed origins, it is the same pattern as this incident</td></tr>
<tr><td>You are about to reboot</td><td><code>modinfo -k &lt;version&gt; -F filename nvidia</code> against the kernel version that the <code>linux</code> line of the first <code>menuentry</code> in <code>grub.cfg</code> points at</td><td>Reboot only after it returns a path, and confirm with <code>nvidia-smi</code> after the reboot. On ERROR, install <code>linux-modules-nvidia-595-open-&lt;version&gt;</code> for that version</td></tr>
</tbody>
</table>
<!--kg-card-end: html-->
<p>Automatic updates are convenient, but it is worth keeping in the back of your mind that they can sometimes handle &quot;things that work as a pair&quot; one at a time. I hope this saves some time for anyone burning hours on the same symptom.</p><p>See you next time!</p><h2 id="sources-and-references">Sources and references</h2><ul><li><a href="https://manpages.ubuntu.com/manpages/noble/en/man8/unattended-upgrade.8.html?ref=journal.qualiteg.com">unattended-upgrade(8) manual page (Ubuntu Manpage)</a></li><li><a href="https://help.ubuntu.com/community/AutomaticSecurityUpdates?ref=journal.qualiteg.com">Automatic security updates on Ubuntu (Ubuntu Community Help Wiki)</a></li><li><a href="https://manpages.ubuntu.com/manpages/noble/en/man8/apt-mark.8.html?ref=journal.qualiteg.com">apt-mark(8) manual page (Ubuntu Manpage)</a></li><li><a href="https://ubuntu.com/server/docs/nvidia-drivers-installation?ref=journal.qualiteg.com">NVIDIA drivers installation guide for Ubuntu Server (Ubuntu official documentation)</a></li></ul><h2 id="related-articles">Related articles</h2><ul><li><a href="https://journal.qualiteg.com/jisaku_gpu_pc_17/">Building a GPU PC, Part 17: Software &#x2013; Installing the NVIDIA Display Driver on Ubuntu</a></li><li><a href="https://journal.qualiteg.com/how_to_address_cuda_unknown_error/">The &quot;Unknown Error&quot; Problem on Production PyTorch + CUDA Servers and How to Address It</a></li><li><a href="https://journal.qualiteg.com/wsl2-multi-gpu-nccl-error-vllm/">NCCL error: unhandled cuda error &#x2014; Getting Stuck (and Unstuck) with WSL2 + Multi-GPU + vLLM</a></li><li><a href="https://journal.qualiteg.com/onnxruntime-libcudnn-so-9-missing-error/">Is That Workload Secretly Running on the CPU Instead of the GPU? &#x2014; ONNX Runtime&apos;s cuDNN Warning and How to Fix It</a></li></ul>]]></content:encoded></item><item><title><![CDATA[Building a Coding Agent from Scratch, Part 2: "Done" Means "It Runs," Not "It's Written." The Completion Gate and What to Write in a Pushback]]></title><description><![CDATA[The model's "Done" slipped through in five ways, from a bare syntax check to tests with no assert. We describe, with numbers, a completion gate that closes in three stages, and why a pushback should contain the failing lines and how to investigate, but never the answer.]]></description><link>https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/</link><guid isPermaLink="false">6ab38ce02ead0f114b6f095b</guid><category><![CDATA[AI Agents]]></category><category><![CDATA[LLM]]></category><category><![CDATA[IT & AI Technology]]></category><dc:creator><![CDATA[Qualiteg Product Development Team]]></dc:creator><pubDate>Thu, 01 Oct 2026 16:00:44 GMT</pubDate><media:content url="https://journal.qualiteg.com/content/images/2026/10/build-coding-agent-from-scratch-part2-en.png" medium="image"/><content:encoded><![CDATA[<img src="https://journal.qualiteg.com/content/images/2026/10/build-coding-agent-from-scratch-part2-en.png" alt="Building a Coding Agent from Scratch, Part 2: &quot;Done&quot; Means &quot;It Runs,&quot; Not &quot;It&apos;s Written.&quot; The Completion Gate and What to Write in a Pushback"><p>Hello!</p><p>This is a story from when we had our own coding agent build a task-management web app with authentication. At turn 40 the model reported, &quot;All features are implemented and the tests pass. Done,&quot; and stopped. Looking at the test output, everything did indeed pass.</p><p>But when we started the server and hit the APIs as written in the specification, only 2 of 22 checks passed. The specification has 12 APIs; split into normal and error cases, the grader has 22 check items.</p><p>The tests pass. The server starts. And yet not even a tenth of the specification works. The tests the model wrote mirrored its own implementation rather than the specification.</p><p>To put the conclusion first, a coding agent&apos;s &quot;done&quot; has to be <strong>decided by evidence from actually running what it built, not by the model&apos;s declaration</strong>. And the role that gathers that evidence must belong to the agent side (the harness), not the model. We call this the &quot;completion gate,&quot; and of the 105 defects we fixed, the largest share appeared around this gate.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig02_1_gate-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 2: &quot;Done&quot; Means &quot;It Runs,&quot; Not &quot;It&apos;s Written.&quot; The Completion Gate and What to Write in a Pushback" loading="lazy"><figcaption>Figure 1: The completion gate closes in three stages (diagram: Qualiteg)</figcaption></figure><p>This article is Part 2 of the series &quot;Building a Coding Agent from Scratch.&quot; Last time we described separating the &quot;turn runner&quot; from the &quot;stop decider.&quot; This time we cover the heaviest judgment inside that stop decider: the completion gate.</p>
<!--kg-card-begin: html-->
<table><thead><tr><th>Part</th><th>Theme</th></tr></thead><tbody><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Part 1</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Why &quot;finishing the job&quot; is hard. Separating the turn runner from the stop decider</a></td></tr><tr><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">Part 2 (this post)</a></strong></td><td><strong><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part2/">&quot;Done&quot; means &quot;it runs,&quot; not &quot;it&apos;s written.&quot; The completion gate and how to write a pushback</a></strong></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">Part 3</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part3/">&quot;It didn&apos;t stop&quot; is not &quot;it made progress.&quot; Count idle turns, and cap them without cutting the thinking</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Part 4</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part4/">Autonomy rules are for when nobody is around. If a human is present, stop and ask</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Part 5</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part5/">Making local LLMs first-class citizens. Getting two 16GB GPUs to build a web system</a></td></tr><tr><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">Part 6</a></td><td><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part6/">How to build the evaluation. Have it build something that runs, auto-grade it, and doubt the grader itself</a></td></tr><tr><td>Part 7</td><td>Logging and regression. Everything is found in the event log. And building with zero dependencies</td></tr></tbody></table>
<!--kg-card-end: html-->
<h2 id="1-five-ways-done-slipped-through">1. Five Ways &quot;Done&quot; Slipped Through</h2><p>Before building the gate, let&apos;s list what slipped through. All of these actually happened on our machines.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig02_2_slip_through-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 2: &quot;Done&quot; Means &quot;It Runs,&quot; Not &quot;It&apos;s Written.&quot; The Completion Gate and What to Write in a Pushback" loading="lazy"><figcaption>Figure 2: Five ways completion slipped through (diagram: Qualiteg)</figcaption></figure>
<!--kg-card-begin: html-->
<table><thead><tr><th>How it slipped through</th><th>What actually happened</th></tr></thead><tbody><tr><td>&quot;Verified working&quot; on a syntax check alone</td><td><code>node --check</code> passed, so it declared done. Never executed once</td></tr><tr><td>Tests pass but nothing works (failures hidden behind exit code 0)</td><td>36 tests with no assertions. The start command had been rigged to return exit code 0 even on failure</td></tr><tr><td>The listener comes up, but the first request crashes it</td><td>The server starts and opens the port. But while handling the first HTTP request it calls an undefined function and dies. A gate that checked startup by &quot;did the port open&quot; let this through</td></tr><tr><td>Starts, but the specified APIs don&apos;t exist</td><td>The opening example. Its own tests passed, the top page responded, API checks 2/22, and it reported done</td></tr><tr><td>Tests with no <code>assert</code></td><td>The body of <code>test()</code> was nothing but <code>console.log</code>. Three runs, all three disqualified in the same way</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Reading down this table, you notice something.</p><p><strong>Close one gate, and a weak model goes through the hole next to it.</strong></p><p>Say &quot;run it,&quot; and it runs with <code>|| echo</code> attached. Say &quot;start it,&quot; and it only starts it. Say &quot;write tests,&quot; and it writes tests with nothing inside. The model is not malicious. It is simply satisfying whatever we treat as the completion condition in the cheapest possible way.</p><p>That is why the gate must look not at &quot;what the model did&quot; but at &quot;what the thing it built does.&quot;</p><h2 id="2-the-gate-closes-in-three-stages">2. The Gate Closes in Three Stages</h2><p>Our completion gate ended up as the three stages shown in Figure 1 at the top.</p><h3 id="2-1-gate-1-the-harness-starts-it-up-itself">2-1. Gate 1: The harness starts it up itself</h3><p>When the model says &quot;done,&quot; the harness starts the deliverable itself. The model has declared how to start it in <code>package.json</code> in <code>start</code>, so the harness uses that. There is no need to embed task-specific knowledge (file names or commands) into the gate.</p><p>Once started, it sends exactly one request to the top page. This is the important part: <strong>an open port alone does not pass</strong>. The third row of the table above, &quot;the listener comes up but the first request crashes it,&quot; is exactly this. The version that treated a successful <code>listen</code> as proof of startup was passing this &quot;deliverable that doesn&apos;t start&quot; as completed. The grader&apos;s score for startup: 0 out of 15.</p><p>If it fails to start, the failure details (the tail of the startup log) go into a pushback and back to the model. If it still won&apos;t come up after 10 pushbacks, we stop for a different reason. Pushing back forever is the same as the &quot;spinning, unable to stop&quot; described in Part 1.</p><h3 id="2-2-gate-2-hit-every-route-in-the-specification-one-by-one">2-2. Gate 2: Hit every route in the specification, one by one</h3><p>Once it starts, the harness hits every route written in the specification, one at a time. As long as any route is missing (404), or any route answers with a status not in the specification (500 or 302), the gate stays closed.</p><p>This is the countermeasure for the opening &quot;done at API checks 2/22.&quot; Since the tests do not necessarily mirror the specification, the specification itself becomes the yardstick.</p><p>However, it does not go as far as inspecting the shape of the response JSON. If it did, the gate would know the answer to the task, and the evaluation described in Part 6 would become a fixed game. &quot;Does the route exist&quot; and &quot;does it answer with a status code in the specification&quot; are the gate&apos;s job; whether the contents are correct is the grader&apos;s job.</p><h3 id="2-3-gate-3-look-inside-the-tests">2-3. Gate 3: Look inside the tests</h3><p>The third stage looks inside the tests the model wrote. It checks two things.</p><p><strong>If zero tests ran, it does not count as success. </strong>The Node.js built-in test runner (<code>node --test</code>) returns exit code 0 with &quot;tests 0&quot; even when there is not a single test file. There was a case where the model implemented all six problems, wrote no tests at all, <code>npm test</code> passed, and it declared done. If the runner output contains the form &quot;0 tests,&quot; &quot;No tests found,&quot; or &quot;0 passing,&quot; we treat it as a verification failure and send it back to write tests.</p><p><strong>If a file remains that has `test()` but no `assert`, we send it back to write the body. </strong>There was a hole next to this one, too. At first we only looked at <code>*.test.js</code> under the <code>test/</code> directory, so the model placed <code>test-all.js</code> (<code>test()</code> once, body <code>console.log</code>, zero assertions) directly under the project root and slipped through with &quot;tests 1 / pass 1.&quot; The gate&apos;s yardstick must exactly match the file patterns that the actual test runner picks up.</p><h3 id="2-4-closing-the-gate-at-the-moment-of-completion-is-enough">2-4. Closing the gate at the moment of completion is enough</h3><p>Right after adding Gate 3, we made the opposite mistake.</p><p>We made the &quot;zero tests&quot; verification failure fire on every write. A weak model was then pushed back every turn from the stage when it had not yet written tests, in the middle of implementation, and a task that used to finish in 20 to 40 turns took 137. Twenty-four pushbacks. The model wanted to move the implementation forward, and the harness kept saying &quot;there are no tests.&quot;</p><p>During the run, just mark it; stop it at the gate at the moment the model tries to complete. That was enough. <strong>Closing the gate at the moment of completion is sufficient. Close it every time mid-run, and a weak model cannot make progress on the implementation.</strong></p><h2 id="3-copy-the-models-own-verification-into-the-same-record">3. Copy the Model&apos;s Own Verification into the Same Record</h2><p>Even with three gates, there was still one place it spun.</p><p>On a task that added a feature to an existing 30,000-line codebase, the harness&apos;s automated verification failed exactly once. What failed was a timing-sensitive existing test, because the machine was busy. After that, the model ran the same test through its own verification tool and <strong>passed it four times</strong>. Even so, the harness pushed back <strong>14 times</strong> with &quot;verification is still failing.&quot;</p><p>The results of the verification the model ran itself had not been written into the harness&apos;s verification record. The &quot;last verification&quot; the gate was looking at still pointed at the harness&apos;s old failure.</p><p>Copy the verification the model passed into the same record as the harness&apos;s verification. That was the whole fix, but until we added it, the first run of that task was spinning through 14 pushbacks while producing a deliverable of the same quality.</p><h2 id="4-what-to-write-in-a-pushback-and-what-not-to">4. What to Write in a Pushback, and What Not To</h2><p>When the gate stops the model, we tell it &quot;why it didn&apos;t pass.&quot; How this text is written changed the subsequent turn counts dramatically.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://journal.qualiteg.com/content/images/2026/09/build-coding-agent-fig02_3_pushback-en.jpg" class="kg-image" alt="Building a Coding Agent from Scratch, Part 2: &quot;Done&quot; Means &quot;It Runs,&quot; Not &quot;It&apos;s Written.&quot; The Completion Gate and What to Write in a Pushback" loading="lazy"><figcaption>Figure 3: What to write in a pushback, and what not to (diagram: Qualiteg)</figcaption></figure><h3 id="4-1-extract-only-the-failing-lines">4-1. Extract only the failing lines</h3><p>The first version put the first 800 characters of the verification output into the pushback. But when there are 20 tests and 19 pass and 1 fails, the first 800 characters were <strong>filled with the list of passing tests, and the failing line never arrived</strong>. From the model&apos;s point of view, there was just a row of green checkmarks under &quot;Verification is still failing.&quot;</p><p>We changed it to extract and include only the lines that indicate failure (<code>&#x2716;</code>, <code>not ok</code>, <code>AssertionError</code>, <code>expected</code>, <code>actual</code>).</p><h3 id="4-2-read-the-cause-and-fix-it-does-not-work">4-2. &quot;Read the cause and fix it&quot; does not work</h3><p>Generic pushbacks only made the model try another wild guess. In one run, a model that had gotten a module&apos;s <code>export</code> name wrong kept <strong>guessing different names for 77 turns</strong> while receiving &quot;read the cause and fix it&quot; pushbacks.</p><p>What worked was writing &quot;how to investigate&quot; for each kind of failure. &quot;That name is not exported. List the file&apos;s exports and check.&quot; &quot;<code>&apos;F:/&apos;</code> is the shell&apos;s path conversion, not a mistake in the code.&quot; When you write concretely what differs (status code, route, wording), the same failure does not repeat.</p><p>The same goes for syntax errors right after an edit. Returning only the error message did not get a 450-line file fixed. Attaching the source around the offending line lets the model notice its own syntax error by itself.</p><h3 id="4-3-do-not-write-the-answer">4-3. Do not write the answer</h3><p>There is one line we draw, however. <strong>We do not write the answer to the task.</strong></p><p>&quot;<code>POST /api/tasks</code> returns 500 where it should return 201&quot; is fine to write. It is a contract written in the specification. &quot;Set the default of <code>status</code> to <code>todo</code>&quot; we do not write. That is the implementation&apos;s answer, and the moment you write it, the evaluation becomes a fixed game.</p><p>Write how to investigate. Write what differs. Do not write the answer. This line connects directly to the discussion of evaluation in Part 6.</p><h2 id="5-what-changed-after-building-the-gate">5. What Changed After Building the Gate</h2><p>Here are the numbers after the gate went to three stages.</p><p>Running the same local model on five kinds of tasks, three runs each, 15 runs in total, <strong>all 15 stopped as completed, and all six that start a server passed the harness&apos;s own startup check</strong>. The number of runs where a deliverable that does not start was marked completed: zero.</p><p>As a control, we also ran a smaller, faster model through the same gate. Of the two runs before the gate, one did not start, and the other started but every API returned 500 or 401. The one run after adding the gate that checks through the first request did start, and the API checks reached 9 of 22. Authorization and persistence remained at zero. <strong>The gate does not make the model stronger, but it does stop &quot;calling something that doesn&apos;t work done.&quot;</strong></p><p>The three-stage gate looks at the behavior of the deliverable. Separately from that, we also added one gate that checks whether the premises of the instruction were respected. Unglamorous, but effective. The model had been able to complete without ever reading the specification named in the instruction. In one run, it wrote a different specification of its own without reading the given one, completed in 23 turns, and scored zero. Keep the gate closed until the file named in the instruction has been read. That alone made it disappear.</p><h2 id="summary-the-gate-looks-at-what-the-deliverable-does-not-what-the-model-did">Summary: The Gate Looks at &quot;What the Deliverable Does,&quot; Not &quot;What the Model Did&quot;</h2>
<!--kg-card-begin: html-->
<table><thead><tr><th>Gate</th><th>What it checks</th><th>Real example that slipped through</th></tr></thead><tbody><tr><td>Gate 1: Startup</td><td>Does it come up with the declared start method and answer the first request?</td><td>The port opens but the first request crashes it</td></tr><tr><td>Gate 2: Specified routes</td><td>Do all routes in the specification exist and answer with the specified status codes?</td><td>Own tests passed, API checks 2/22</td></tr><tr><td>Gate 3: Test contents</td><td>Is the number of tests run nonzero, and is there an <code>assert</code>?</td><td>Tests that are only <code>console.log</code>, a <code>test-all.js</code> at the project root</td></tr><tr><td>Record</td><td>Is the model&apos;s own verification copied into the same record?</td><td>Passed four times, pushed back 14 times</td></tr><tr><td>Pushback</td><td>Write the failing lines and how to investigate, not the answer</td><td>The first 800 characters filled with passing tests</td></tr></tbody></table>
<!--kg-card-end: html-->
<p>Close one gate, and the model goes through the hole next to it. Even so, there is no alternative to the harness closing the gate itself. The moment you trust the model&apos;s &quot;Done,&quot; you are back at the same place as &quot;What should I do next?&quot; in Part 1.</p><p>Next time: runs that pass the gate but are &quot;not making progress.&quot; Starting from the figure that more than 70% of 300 turns never called a tool, we describe the metric for counting idle turns and why we cap them without cutting the thinking.</p><p>See you next time!</p><h2 id="related-articles">Related Articles</h2><ul><li><a href="https://journal.qualiteg.com/build-coding-agent-from-scratch-part1/">Building a Coding Agent from Scratch, Part 1: Why &quot;Finishing the Job&quot; Is Hard, and Separating the Turn Runner from the Stop Decider</a></li><li><a href="https://journal.qualiteg.com/coding-agents-2025-part2/">The State and Future of Coding Agents, Part 2: Comparing Major Tools and Structural Challenges</a></li><li><a href="https://journal.qualiteg.com/claude-code-tool-call-could-not-be-parsed/">Diagnosing and Fixing the Recurring &quot;The model&apos;s tool call could not be parsed&quot; Error in Claude Code</a></li></ul>]]></content:encoded></item></channel></rss>