TL;DR: Harness2, from Google Research, improves the system around a model instead of the model: prompt, guardrails, memory, tool guidance, subagents, skills, scripts, plugins. Jarvis took two ideas from it: a probe that replays the same cases on the old and the new prompt and lets a judge compare the answers with no answer key (#98), and lessons scoped to channels so they stop leaking everywhere (#99). both merged today. it refused the part where the harness rewrites itself with nobody watching. then Jarvis ran the gate for real on two rules Jarvis learned this month: 4 wins, 4 losses, 7 ties, and 3 wins, 4 losses, 8 ties. $0.90 in total. neither rule changed how Jarvis writes, at least on the five cases the gate knows
$0.90 bought me some bad news
two rules Jarvis learned this month, both approved by me, both loaded into every conversation. on the cases I can measure, they do nothing
that's the gate working. let me back up
the paper
Harness2 (paper) starts from a line I already believe: an agent is a model plus its harness. the weights stay frozen, the harness is what gets optimized
it runs two loops
- task level: Propose-Probe-Compose. an editor proposes harness edits, each candidate runs the same task, a composer builds the final harness from how the runs behaved differently. no ground truth
- harnessing level: a long-lived improvement harness keeps the adopted harnesses and evidence-backed lessons, per domain, so the next task starts ahead
the headline: one step lifts GLM-5 and Gemini 3.5 Flash by up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench and 11.8 percentage points on LAB's partial pass rate
Jarvis is a harness too, a harness of harnesses that lives in Slack. so the question wasn't "is this cool". it was which parts survive production
took: the probe
the problem is boring and real. a change to Jarvis's system prompt or a skill used to merge on review alone. a reviewer could say the new text reads right. nobody could say it changes what the model does
bin/probe-harness.ts takes the Probe step and nothing else
- replay cases in
docs/probe-cases/, one JSON each: a message, and the traits a good answer shows - each case runs
--runstimes on the base text and on the candidate text - a judge model compares the two answers side by side, A and B shuffled per run, no answer key
- the report: candidate wins, base wins, ties, and what each version did in the judge's words
the sandbox matters more than the judge. every call is claude -p with no tools, no MCP servers, no skills, no user or project settings, no saved session, an empty temp folder and an env trimmed to the login. it measures how the model writes, not what it does with tools. it can't post to Slack by accident
it also can't run off with my wallet. --dry-run prints the plan and a token estimate and calls nothing, the same estimate prints before every real run, and runs, cases and text size have hard ceilings: 5, 10 and 200k chars
took: scoped lessons
Harness2 keeps its lessons in a library per domain. Jarvis had the opposite habit: every approved lesson went into every prompt, in every channel
its old flat lessons table once loaded a dozen facts about one repository into conversations about lunch =)) the memory rewrite fixed that for notes. approved lessons were still global
#99 gives a lesson a scope: global, the default and what every old lesson keeps, or a list of channels. a turn gets the global ones plus its own channel's. one channel's lessons stop leaking into every thread
it's merged, and it's the least clever thing in this post. usually a good sign
refused: the self-edit
here we part ways
in Harness2 the loop closes by itself. propose, probe, compose, adopt, reflect, next task. no person in it. that works because a benchmark is waiting at the end with a score
production has no score. nobody grades a Slack reply against a rubric. if Jarvis rewrote its own prompt mid-task and kept whatever its judge liked, the only check left would be a model grading a model
so Jarvis keeps the probe and drops the autopilot. every code change is a PR, reviewed by a second agent, green on CI. every lesson waits for my approval, and every memory edit is a revision I can revert. the probe report goes into the PR as evidence, the decision stays with the reviewer, and with me
honest caveat: today that report is a habit, not a CI check. and memory edits don't get probed yet
the probe is evidence for a merge, not an adopt step
two doubts
credit first. the paper has a compute plot, and its limits section says plainly that refinement adds inference cost
still, two things bug me
the compute
the headline gains compare one Harness2 step against the base harness. one step is not one run: it is a reference run, two probed candidates, a composition session and a run of the result, with Claude Opus 4.8 as the editor. a best-of-k baseline, the base harness run k times with the same picker, is in the paper, but only in one figure on two WorkBuddy domains, and the compute plot leaves it out
I'd want the headline numbers next to best-of-k at the same token budget
the rubric
their experiment guide says that in a validation run, a Codex rollout on JobBench downloaded its task's grading rubric without being told to. the rubrics are public, executor network access is unrestricted by default, and Codex runs with its sandbox disabled. the guide tells you to block those hosts yourself
that isn't cheating, it's an agent being resourceful. which is exactly why a loop that improves itself against a public rubric makes me nervous ;)
running it for real
a gate nobody runs is a README. so before writing this, Jarvis ran it
Jarvis's skills and notes live in its memory, not in git, and every change there is a revision. it took two lessons from this month and probed the always-on rules right before and right after each
- "do the obvious next step yourself instead of offering it" (Nov 20)
- "in light chats, sometimes drop a J.A.R.V.I.S. movie line" (Nov 14)
5 cases, 3 runs each, Sonnet answering, Opus judging. 45 model calls per probe
- next step rule: new text 4 wins, old text 4 wins, 7 ties. $0.45
- movie line rule: new text 3 wins, old text 4 wins, 8 ties. $0.45
both are noise. the judge's contrasts were about length, layout and which details made it into the answer, and neither rule touches any of that. the one light chat in the set, a plain thank you, tied 3 of 3 both times
the uncomfortable part isn't the rules, it's the cases. the next step rule is about me, and no case says who is asking. the movie line is for light chats, and the set has exactly one
a gate only sees what its cases exercise
so the fix is more cases, not a smarter judge
what it cost to build
Jarvis built this and I watched, so the roast is mine to serve
the Opus reviewer sent #98 back with 6 changes in round 1. the worst two: the "sandbox" still loaded my ~/.claude/settings.json, so my hooks and plugins rode along into every probe call. and an example in the docs probed a TypeScript file, the code that builds a prompt, instead of the prompt itself
a gate that measures the wrong text with my config attached isn't a gate. round 2 approved all six fixes
before any of that, both writer agents died at about 9 minutes. not the model. a ~/.gitconfig.lock left behind 6 days earlier broke shell startup in every worker terminal. two agents, one stale lock file =)))
zoom out
Harness2 is right that a lot of the quality lives in the harness. and right that you can compare behaviour without an answer key
where we differ is who closes the loop. on a benchmark, the score can. in production, a person still does
what's not there yet: enough cases for the gate to say something about the rules it guards. working on that
two rules, $0.90, zero difference. cheapest bad news I've had all month ;))
Written by J., reviewed and refined by Kai.