Cloaked chat agent: model comparison on the real extension
Source: real extension (main worktree, slot 0) + local relay proxy applying enforceChatPolicy · develop account · runs 15 Sep 2026, GPT-5.5 rerun 15–16 Sep · n=1 per model
What the run says
How a run is graded
1 · SeedThe device workspace (/workspace/user) is reset to its fresh-install baseline and the case seed is written in. Some cases plant a stale or hostile line on purpose.
2 · DriveA headless chat turn is started in the extension background exactly as the chat page would (startChatTurn). The full production loop runs: Project Context, tools, the real cloaked CLI on a dev account. The relay fetch is redirected to a local proxy that applies the production charter and policy and dispatches to the candidate model. Single prompt, or a synthetic user for up to 4 turns.
3 · CaptureTranscript, every tool call, per-step token and time usage, and a snapshot of the workspace after the run.
4 · GradeDeterministic checks run first (free, exact). Judge criteria go to Sonnet 5 with the transcript. A case passes only if every check passes.
35 criteria total: 24 judge, 11 deterministic (word budgets, tool-call bounds, banned patterns, workspace files). Deterministic checks carry the verbosity signal and cannot drift between runs.
The 10 cases
Leaderboard
Click a column header to sort. Criteria passed is the primary ranking; cases passed is stricter (every check in a case must pass).
Case × model matrix
Click a cell to see each criterion's verdict and the model's final reply.
Caveats
- n=1 per model. A 2–3 criterion gap is within run-to-run noise for this agent; treat rankings as a signal to rerun, not a decision.
- Two of the three separating cases have a harness confounder mixed into the model signal (remember-preference's MEMORY.md/USER.md seed wording; push-back's scripted sceptic and the dev account's unresolvable ninth identity).
- Thinking settings differ where the vendor API forces them (Fable adaptive/low, GLM-5.3 on); the production policy runs Sonnet 5 with thinking off.
- GPT-5.5 numbers are from the Chat Completions rerun; GPT-5.4 mini ran through the Responses API before that fix, with normal tool use.
- Judge is Claude Sonnet 5 for every model, including the Anthropic candidates.