A Second Model Should Review the Plan
When I cannot judge a plan myself, the model that wrote it should not be its only reviewer. A model from a different vendor can critique it in a bounded loop, and any factual claim the reviewer makes still has to be checked.
Planning skills have made one thing easy: getting an agent and me to agree on what to build. A grilling session asks questions until the plan matches what is in my head. Then the plan gets written up and split into tasks.
Agreement is a lower bar than quality. If the plan covers ground I cannot judge on my own, the only reviewer left is the model that wrote it. My own chain has exactly this shape. The grilling, the requirements doc and the task breakdown all run inside one model family.
Chase AI's recent short video frames the gap as "who watches the watchman" and proposes a structural fix. Hand the plan to a model from a different vendor and let the two argue it out.
The loop
Chase's skill is called Grill Me Codex and sits on top of Matt Pocock's grill-me. When the grilling ends, a second act begins. Codex critiques the plan, Claude Code defends or revises it, and the exchange repeats for up to five cycles until the two reach an approved verdict. He notes the reviewer is swappable, including for a local model to keep cost down.
grill -> plan
plan -> reviewer from a different vendor -> critique
critique -> planner -> defend or revise
repeat, at most 5 cycles
verdict -> approved plan
What the video does not say is what happens when cycle five ends without agreement. I would want the open disagreements handed to me as a list, not forced into a verdict. A dispute two models cannot settle marks the spot where a human decision is needed.
Why a different vendor
Chase's argument is that Claude Code grades its own work generously. Ask it to review its own output, he says, and it gives itself an "A plus". He adds that Anthropic has acknowledged the weakness.
Both statements are asserted. The video is two minutes long and shows neither a test nor the acknowledgment. I still find the direction plausible, for a reason that does not depend on his evidence. A model rereading its own reasoning brings the same assumptions that produced it. Whatever it missed the first time, it is well placed to miss again.
That is also why spawning subagents from the same model does not close the gap. Same-family agents are good for splitting work. They share training and habits, so they tend to share blind spots.
The principle is older than any of this. Verification should be independent of generation, which is why authors do not approve their own pull requests. The new part is applying it to the plan, before any code exists, when a wrong assumption costs the least to fix.
What it costs
Chase does not price the loop, and it is not free. In every cycle both models read the plan plus the argument so far. He also sells Claude Code content and this skill, which is worth remembering when a teaser skips the bill.
Cole Medin shows a related arrangement in a live stream on the Pi coding harness, for a different reason. There a cheap working model asks a stronger model for a second opinion before it acts. In his setup that was Kimi K2.6 as the worker and Opus as the reviewer. The split is about economics: pay frontier prices only for judgment. The two ideas stack. A different vendor addresses shared bias, and a different tier addresses cost.
The same stream surfaced a practical snag. Switching models in the middle of a conversation lost the history. So the reviewer needs the plan and its context handed over explicitly, as a file.
If I were setting the dial, I would not start at five. One critique and one response covers most plans. I would save the longer debate for plans that are expensive to get wrong.
The reviewer can be wrong too
The risk Chase's video does not touch is that two language models in a debate can settle on a confident answer that is false. Their agreement tells me the plan survived one more reader. It does not tell me the plan is correct.
I recently started using a multi-agent debate for hard design calls. Several personas argue, and one consensus per point gets written down. It came with a standing rule: any factual claim the debate leans on gets checked independently before I act on it. The same rule belongs in this loop.
| Reviewer objection | Settled by |
|---|---|
| The plan skips a failure case | Debate. The planner revises or explains |
| There is a simpler design | Debate, then my call |
| That API, flag or limit does not exist | Reading the docs or running it |
| Both models agree | Nothing yet. Agreement is not a check |
Models are useful for arguing about structure and tradeoffs. Neither one is a source of truth about facts. If the reviewer says a library lacks a feature, that goes to the documentation, not back to the planner for a rebuttal.
The pilot I have in mind is small. After the requirements doc is written and before it becomes tasks, send it once to Codex or Gemini CLI, both of which I already have installed, and see whether the critique changes anything I would have shipped.
Key Takeaways
- Planning skills produce agreement between me and the agent. They do not verify that the plan is good.
- The reviewer should come from a different vendor, because a model rereading its own plan shares the assumptions that shaped it. The source asserts that bias and does not demonstrate it.
- Bound the loop. One round is a sensible default, and the reviewer gets the plan as a file.
- Debate can settle design objections. Factual claims from either model get checked against a primary source.
Sources
- Chase AI, "/grill-me and plan mode aren't enough"
- Cole Medin, "Pi is INCREDIBLE - Building a Custom Coding Agent Live"
Next: run one cross-vendor critique on my next requirements doc and write down what it caught, what it got wrong, and what it cost.