Testing an Agent Runtime with Weak Models
A few days ago, Factory published “Droid 1.0: Building a Frontier Harness”, on how they refined their Agent Runtime. There is one point in the post I strongly agree with: if you tune against only one model, it is hard to tell which changes are genuine improvements and which are just workarounds for that model’s quirks. So they test across many models at once and keep only the improvements that hold up across different models.
Reading that reminded me of a practice I have been using myself.
When people talk about Claude Code, it is easy to treat it as the same thing as the Claude model because Anthropic built both. In fact, it can be wired up to other models just fine, and the same goes for Codex. Products like Claude Code and Codex, as well as what I am building myself, are Agent Runtimes. The model inside can be swapped out at any time; what we actually build by hand is the runtime system around the model, which supplies prompts, defines tools, organizes context, and drives the execution loop step by step.
To make sure the Agent Runtime itself is solid, I first set an evaluation bar and then deliberately plug in a weaker model to run the tasks. If even the weak model can meet that bar, I know the Agent Runtime is in good shape.
Next, I run an A/B test: I keep the Agent Runtime unchanged and swap in a strong model. Under the same criteria, the results are usually much better.
Why start with a weak model?
If the goal is simply to get a task working, reaching for the strongest model on hand is obviously the easiest path. The problem is that a strong model will mask flaws in the Agent Runtime. Write a vague tool description, and it can still guess what you meant; pass in messy context, and it can sort things out on its own. The task gets done in the end, but it is hard to tell whether the Agent Runtime is genuinely solid or the model is just carrying it through.
A weak model cannot do that. Wherever the instructions are unclear or the information is incomplete, it simply fails. When I trace back from where it got stuck, what I find are the gaps in the Agent Runtime itself. In that sense, it works more like a test: what it tests is not the model, but the Agent Runtime I built.
At heart, this is the same idea as Factory’s: do not let the traits of a single model hijack your judgment. The only difference is that they switch horizontally across models from different providers, while I switch vertically across models of different capability levels.
There is a trap in this approach, though. To get a weak model across the finish line, it is easy to start adding crutches to the Agent Runtime: breaking tasks into tiny pieces, piling on checks, and restricting what the model is allowed to do. Those guardrails help a weak model, but for a strong model they can become a ceiling. Steps a strong model could have judged for itself end up tied down by overly detailed rules. So after switching to a strong model, I do not just look at whether the score went up; I also look at which crutches can be removed.
A good Agent Runtime should look like this: solid enough in its own right to support a weak model, without getting in the way of a strong one. As models keep getting stronger, it does not need to be rewritten; it simply gets stronger along with them.
Reference: Factory, “Droid 1.0: Building a Frontier Harness” (2026-10-29)