AI Agents

Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation, and Doubt the Grader Itself

AI Agents

Building a Coding Agent from Scratch, Part 6: How to Build the Evaluation. Have It Build Something That Runs, Auto-Grade It, and Doubt the Grader Itself

The grader's 10 points for tests were 0/0 for three whole days, invisible in the score table. How we built a rig that has the agent build a web system and grades it by running it, the five-kinds-times-three pass/fail criteria, running the control first, and contracts decided only by the tests.

By Qualiteg Product Development Team
Building a Coding Agent from Scratch, Part 3: Count Idle Turns, and Cap the Thinking Without Cutting It

AI Agents

Building a Coding Agent from Scratch, Part 3: "It Didn't Stop" Is Not "It Made Progress." Count Idle Turns, and Cap the Thinking Without Cutting It

300 turns, zero APIs. Re-reading the log showed 234 turns had never called a tool. We cover the idle rate as a metric, idling that shifts form each time one is closed, how turning thinking off dropped 95 to 71 without saving time, and counting the other side after widening the output window.

By Qualiteg Product Development Team