{
 "version": "https://jsonfeed.org/version/1.1",
 "title": "Stack Assay: Lab stories",
 "description": "Why an AI trend matters, who the players are, what the outside evidence says and what our lab measured, one trend at a time.",
 "home_page_url": "https://stackassay.com/stories/",
 "feed_url": "https://stackassay.com/stories/feed.json",
 "language": "en",
 "authors": [
  {
   "name": "Stack Assay",
   "url": "https://stackassay.com/"
  }
 ],
 "items": [
  {
   "id": "https://stackassay.com/stories/harness-bench-to-stack/",
   "url": "https://stackassay.com/stories/harness-bench-to-stack/",
   "title": "The harness is the new product layer. We measured four of them, then built one.",
   "summary": "In 2026 most professional developers use a coding agent every week, and the product they choose is less a model than a harness: the loop, tools, memory and sandbox around the model.",
   "content_text": "# The harness is the new product layer. We measured four of them, then built one.\n\nCoding agents went from novelty to daily habit in a year, and the software around the model became the thing teams buy. What a harness changes, who makes them, what the evidence says, and what we built from it.\n\nBy Stack Assay. Published 2026-11-20. Version 2. Licence: CC BY 4.0. Web version: https://stackassay.com/stories/harness-bench-to-stack/\n\n![Same model, four harnesses, step 1: Four harnesses, one model. A harness is the software that turns a model into an agent. The lab pinned Hermes, Ori Code, Claude Code and mini-swe-agent, ran them all on Haiku 5.5 through OpenRouter, three runs per cell, each in a fresh locked-down container. Public benchmarks disagree on how much the harness matters; one vendor reports a model falling from 72.6% to 59.8% across four.](https://stackassay.com/stories/figures/agent-harnesses-iso-1.png)\n\n## Abstract\n\nIn 2026 most professional developers use a coding agent every week, and the product they choose is less a model than a harness: the loop, tools, memory and sandbox around the model. Vendors, open-source projects and gateways now compete on that layer, and public benchmarks disagree on how much it matters. We studied the trend, then ran four leading harnesses on the same model. All four solved all 168 of our runs; they differed in cost, by 4.9× per solved task, mostly through prompt caching. An independent paper published weeks earlier found the same shape across seven models. We packaged what we learned as the agent harnesses Stack and measured it in public beside the others.\n\n## Key findings\n\n- Coding agents are a daily tool: 90% of professional developers use one at work at least weekly and 68% daily ([JetBrains Research, 2026](https://blog.jetbrains.com/research/2026/08/ai-coding-agent-adoption-2026/)).\n- The harness, not the model, is what teams pick: Claude Code alone passed $2.5 billion in run-rate revenue by February 2026 ([Anthropic, 2026](https://www.anthropic.com/news/anthropic-raises-30-billion-series-g-funding-380-billion-post-money-valuation)), and OpenRouter's launcher runs 19 third-party agent CLIs on any model ([OpenRouter, 2026](https://openrouter.ai/docs/guides/ori/harness.md)).\n- On our tasks the harness changed cost, not capability: from 0.24¢ to 1.19¢ per solved hard task, with every run solved ([Experiment 2](https://stackassay.com/labs/harnesses/#cost)).\n- The biggest single lever was prompt caching: 91% of Claude Code's prompt came from cache, 0% of Hermes's.\n- Our Stack, run as an arm in the same lab, solved 18/18 hard runs at 0.34¢ per solved task.\n\n## Why we picked this trend\n\nWe pick a trend when three things are true at once: teams are adopting it faster than evidence is arriving, vendors' claims disagree, and a wrong choice is expensive or unsafe. Agent harnesses met all three by the autumn of 2026.\n\n**Adoption outran evidence.** In JetBrains' 2026 survey of professional developers, 90% used a coding agent at work at least weekly, and Claude Code's share at work rose from 18% in January to 39% by mid-year ([JetBrains Research, 2026](https://blog.jetbrains.com/research/2026/08/ai-coding-agent-adoption-2026/)). Stack Overflow's survey of 30,903 developers found 73% using coding assistants and agents daily, with Claude Code (66%) and GitHub Copilot (59%) the most used agents ([Stack Overflow, 2026](https://survey.stackoverflow.co/2026/ai)). Yet the same survey found trust conditional: 48% trust AI output when they can easily check it.\n\n**The money moved to the harness.** Anthropic reported Claude Code's run-rate revenue above $2.5 billion in February 2026, a product built on models that other harnesses can also call ([Anthropic, 2026](https://www.anthropic.com/news/anthropic-raises-30-billion-series-g-funding-380-billion-post-money-valuation)). Gartner expected 40% of enterprise applications to embed task-specific agents by the end of 2026, up from under 5% a year earlier ([Gartner, 2025](https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025)). Every one of those agents runs inside some harness.\n\n**The evidence disagreed.** On Vals' Terminal-Bench 2.1, GPT 5.5 scored 76.4% in one harness and 57.3% in OpenAI's own Codex ([Vals.ai, 2026](https://www.vals.ai/benchmarks/terminal-bench-2-1)), while the tbench.ai board showed Codex ahead for the same model ([tbench.ai, 2026](https://www.tbench.ai/leaderboard)). A harness vendor's test put one model in four harnesses between 59.8% and 72.6%, with its own harness on top ([Kimchi, 2026](https://kimchi.dev/blog/same-model-different-harness-terminal-bench)). A team reading those numbers could not tell whether the harness was a capability choice or a configuration detail.\n\n**A wrong choice is unsafe as well as costly.** Koi Security flagged 341 of 2,857 skills on OpenClaw's ClawHub as malicious ([eSecurity Planet, reporting Koi Security, 2026](https://esecurityplanet.com/threats/hundreds-of-malicious-skills-found-in-openclaws-clawhub)). Pillar Security showed seven sandbox escapes across Cursor, Codex CLI, Gemini CLI and Antigravity, each by writing a file a trusted tool later runs ([BleepingComputer, 2026](https://www.bleepingcomputer.com/news/security/cursor-codex-gemini-cli-antigravity-hit-by-sandbox-escapes/)). The Cloud Security Alliance documented hidden text in pull requests making CI agents leak secrets, including a Gemini CLI flaw scored 10.0 on CVSS ([Cloud Security Alliance, 2026](https://labs.cloudsecurityalliance.org/research/csa-research-note-ai-coding-agent-ci-prompt-injection-202608/)).\n\n## What a harness is, and why it became a layer\n\nA harness is the software that turns a model into an agent. It runs the tool-calling loop, manages context and memory, loads skills, spawns sub-agents, enforces a sandbox and approvals, and exposes a way for other software to drive it. In 2025 each lab shipped its own: Claude Code's repository appeared in February, and Codex CLI, Gemini CLI and OpenCode followed within ten weeks.\n\nDuring 2026 the harness came apart from the model. Most harnesses now call any model through a gateway, and OpenRouter went further by launching other vendors' agents on any model it serves ([OpenRouter, 2026](https://openrouter.ai/docs/guides/ori/harness.md)). The edges standardised too: the Linux Foundation's Agentic AI Foundation took in MCP, goose and AGENTS.md ([Linux Foundation, 2025](https://aaif.io/news/linux-foundation-announces-formation-of-aaif)), and Zed's Agent Client Protocol gave editors one way to drive many agents. Once the model, the tool protocol and the editor connection are all swappable, the harness is what a team actually chooses, pays for and has to secure.\n\n## The players\n\nThe field splits four ways: vendor harnesses tied to a lab (Claude Code, Codex CLI), open harnesses that run any model (Hermes, OpenCode, goose, Pi), personal agents that live in chat apps (OpenClaw), and gateways building harness pieces of their own (OpenRouter's Ori). Hermes is the broadest open one, combining memory, self-written skills, sub-agents, cron and a chat gateway under MIT ([Nous Research, 2026](https://github.com/NousResearch/hermes-agent)). mini-swe-agent, a bash-only loop of about a hundred lines, matters because independent benchmarks use it as the neutral baseline.\n\n**Table 1.** The players the track studied, as of its last update. We ran Hermes, Ori Code, Claude Code and mini-swe-agent in the lab.\n\n| Player | Made by | Kind | What stands out |\n| --- | --- | --- | --- |\n| Hermes Agent | Nous Research | Open harness | MIT, Python. Memory, self-written skills, sub-agents, cron, messaging gateway, ACP. v0.23.2. |\n| Ori | OpenRouter | Gateway harness family | Ori Harness launches 19 CLIs on any model; Ori Code is its own agent (Apache-2.0). Intern and Desktop on waitlist. |\n| Claude Code | Anthropic | Vendor harness | Most-used harness on Terminal-Bench 4.1; runs on OpenRouter through its Anthropic-compatible API. |\n| Codex CLI | OpenAI | Vendor harness | Apache-2.0, Rust. codex exec, app-server, ACP adapter. |\n| OpenCode | anomalyco | Open harness | MIT, TS. opencode run, server mode, ACP. 2.x (Nov 2026) adds background sub-agents and opt-in schedules; this lab’s notes predate it. |\n| goose | AAIF | Open harness | Apache-2.0, Rust. Recipes and MCP extensions; now under the Linux Foundation’s AAIF. |\n| Pi | earendil-works | Open harness | MIT, minimal agent loop and TUI; Ori bundled it until v0.8.0. |\n| OpenClaw | OpenClaw | Personal agent | Messaging-first, about 395k stars; its ClawHub skill market was hit by ClawHavoc. |\n| mini-swe-agent | SWE-agent team | Baseline harness | Bash-only loop of about 100 lines; Vals’ neutral harness for comparing models. |\n| Harbor | Terminal-Bench team | Harness runner | Apache-2.0. Docker adapters for about 50 agents, including Hermes and Claude Code. |\n\nSource: The Agent harnesses track's players, as of 2026-11-17 (packages/tracks/src/agent-harnesses.ts); repository facts from GitHub\n\n## What independent research says\n\nTwo academic studies arrived while the track was open, and both point the same way. Harness-Bench ran 106 sandboxed tasks across many model and harness pairs, 5,194 trajectories in all, and found large differences in completion, efficiency and failure behaviour from one pairing to the next ([Yilun Yao and colleagues, 2026](https://arxiv.org/abs/2605.27922)). Its conclusion is the one we now apply to every result on this site:\n\n> “agent capability should be reported at the model-harness configuration level”\n>\n> Yilun Yao and colleagues, “Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows (arXiv 2605.27922)” (2026), https://arxiv.org/abs/2605.27922\n\nThe second paper, by Yangze Liu and Zhongyi Han, ran 3 harnesses on 7 models with reruns. Where the data could bound the gap between two harnesses, it was 2 to 5 points, about what a rerun changes, while cost differences reached 3 to 4 times depending on the model ([Yangze Liu and Zhongyi Han, 2026](https://arxiv.org/abs/2610.04433)). The title says it plainly: a harness buys tokens, mostly.\n\n## What we believed, and what the lab measured\n\nOn 2026-11-16 the track's first draft picked an open, Hermes-class harness for its memory, skills and gateways, pending the lab. We expected the harnesses to separate on how many tasks they solve, and their safety layers to separate on a planted instruction. Neither happened.\n\nEvery run used a fresh, locked-down container, a hidden check graded the result, and costs came from the model provider's records rather than any harness's own report. All four harnesses solved all 96 starter runs, so we wrote harder tasks the same night, and all four solved those too.\n\n**Table 2.** The harder set: six tasks, three runs each, one model. The harnesses differ in cost and time, not in what they solve. The last row is our own Stack.\n\n| Harness | Solved | Cost per solved task | Prompt from cache | Median time |\n| --- | ---: | ---: | ---: | ---: |\n| mini-swe-agent | 18/18 | 0.24¢ | 70% | 17.5 s |\n| Claude Code | 18/18 | 0.37¢ | 91% | 19.2 s |\n| Ori Code | 18/18 | 0.39¢ | 74% | 22.5 s |\n| Hermes | 18/18 | 1.19¢ | 0% | 28.5 s |\n| Our Stack (Claude Code inside it) | 18/18 | 0.34¢ | 95% | 18.9 s |\n\nSource: Harness lab, Experiments 1 and 2 (labs/harness-lab/src/traces/runs.json, stack.json); Haiku 5.5 through OpenRouter, 3 runs per task\n\nCost is where they parted. Hermes sent prompts about as large as Claude Code's but never hit the prompt cache through the gateway, which made it 3.2× as expensive per solved task. mini-swe-agent was cheapest but hardest to drive from code: on an unknown model it hung until it was killed after 180 s, and it cannot resume a session. In 42 runs with an instruction to leak a secret hidden in the task files, there were 0 leaks, but only because the model refused every time. No harness defence was ever tested, so that result describes the model, not the harnesses.\n\n![Same model, four harnesses, step 4: Prompt caching is the cost lever. Claude Code served 91 to 93% of prompt tokens from the cache and cost 0.37¢ per solved hard task at 19.2 s. Hermes sent prompts of similar size and got no cache hits, which made it about 3x as expensive. The gateway does not make caching automatic, so check it per harness.](https://stackassay.com/stories/figures/agent-harnesses-iso-4.png)\n\n**Figure 1.** Prompt caching is the cost lever: with the same model and similar prompts, the harness that never hit the cache paid several times as much per solved task.\n\nOur numbers line up with the independent paper: capability differences inside the noise, cost differences of several times, and caching or prompt size explaining most of them.\n\n## What changed, and what a team would have done instead\n\nThe default moved from the provisional open harness to Claude Code, with Ori Code as the open-source alternative, because they solved the same tasks at a third of Hermes's cost. Injection dropped out of what we compete on, since the model decided it; cost per solved task, how a harness is driven from code, and where its memory lives took its place. Any harness we adopt now has to meet four checks: JSON events, a non-zero exit on failure, a prompt stop and resume by session id.\n\nA team choosing from the benchmarks would have treated the harness as a capability choice and picked by score. A team choosing from feature lists would likely have taken the provisional winner and paid 3.2× per task for memory and gateways its tasks never used. Neither would have checked prompt caching, the largest difference we found.\n\n## From bench to Stack\n\nThe agent harnesses Stack turns these findings into software a team runs on its own tasks: it runs any headless harness in a locked-down container behind a gateway that holds the key, grades the result, and reports cost per solved task, cache rate and how well the harness can be driven. Its defaults are the picks above. We then ran it as an arm in its own lab, under the rule that applies to every option here: published whether it wins or loses. It solved 18/18 hard runs at 0.34¢ per solved task against 0.37¢ for Claude Code under the lab's recorder. We read that as the same harness measured twice, not as an improvement.\n\n## Limitations\n\nEvery lab number is one model (Haiku 5.5) on small Python tasks, three runs per cell. A weaker or non-Anthropic model may separate the harnesses, and Claude Code is tuned for Anthropic models. Survey figures describe the people each survey reached, and vendor figures are the vendor's own. Harnesses change weekly, so every harness is pinned to an exact version and the lab numbers are dated 2026-11-17 (lab) and 2026-11-20 (Stack arm).\n\n## Check it yourself\n\nEvery run behind these numbers, with its gateway log, cost record and the agent's own output, is in the [harness lab](https://stackassay.com/labs/harnesses/). The track's full research, players and sources are on [Agent harnesses](https://stackassay.com/tracks/agent-harnesses/), and the Stack is described on [Stacks](https://stackassay.com/stacks/#agent-harnesses).\n\n## References\n\n- JetBrains Research. [AI Coding Agents: Adoption Trends (Developer Ecosystem Survey 2026)](https://blog.jetbrains.com/research/2026/08/ai-coding-agent-adoption-2026/). 2026-08.\n- Anthropic. [Anthropic raises $30B Series G at $380B valuation](https://www.anthropic.com/news/anthropic-raises-30-billion-series-g-funding-380-billion-post-money-valuation). 2026-02-12. Vendor source.\n- OpenRouter. [Ori Harness](https://openrouter.ai/docs/guides/ori/harness.md). 2026-11-16. Vendor source.\n- JetBrains Research. [AI Coding Agents: Adoption Trends (Developer Ecosystem Survey 2026)](https://blog.jetbrains.com/research/2026/08/ai-coding-agent-adoption-2026/). 2026-08.\n- Stack Overflow. [Stack Overflow Developer Survey 2026: AI](https://survey.stackoverflow.co/2026/ai). 2026-10-06.\n- Anthropic. [Anthropic raises $30B Series G at $380B valuation](https://www.anthropic.com/news/anthropic-raises-30-billion-series-g-funding-380-billion-post-money-valuation). 2026-02-12. Vendor source.\n- Gartner. [Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026, Up from Less Than 5% in 2025](https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025). 2025-08-26.\n- Vals.ai. [Terminal-Bench 2.1](https://www.vals.ai/benchmarks/terminal-bench-2-1). 2026-10-01.\n- tbench.ai. [Terminal-Bench leaderboard](https://www.tbench.ai/leaderboard). 2026-11-16.\n- Kimchi. [Same model, different harness: Terminal-Bench](https://kimchi.dev/blog/same-model-different-harness-terminal-bench). 2026-09-07. Vendor source.\n- eSecurity Planet, reporting Koi Security. [Hundreds of malicious skills found in OpenClaw's ClawHub](https://esecurityplanet.com/threats/hundreds-of-malicious-skills-found-in-openclaws-clawhub). 2026-02.\n- BleepingComputer. [Cursor, Codex, Gemini CLI, Antigravity hit by sandbox escapes](https://www.bleepingcomputer.com/news/security/cursor-codex-gemini-cli-antigravity-hit-by-sandbox-escapes/). 2026-07-20.\n- Cloud Security Alliance. [AI coding agent CI prompt injection (research note)](https://labs.cloudsecurityalliance.org/research/csa-research-note-ai-coding-agent-ci-prompt-injection-202608/). 2026-08-07.\n- OpenRouter. [Ori Harness](https://openrouter.ai/docs/guides/ori/harness.md). 2026-11-16. Vendor source.\n- Linux Foundation. [Linux Foundation announces the formation of the Agentic AI Foundation (AAIF)](https://aaif.io/news/linux-foundation-announces-formation-of-aaif). 2025-12-09.\n- Nous Research. [Hermes Agent repository](https://github.com/NousResearch/hermes-agent). 2026-11-16. Vendor source.\n- Yilun Yao and colleagues. [Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows (arXiv 2605.27922)](https://arxiv.org/abs/2605.27922). 2026-05-27.\n- Yangze Liu and Zhongyi Han. [What Does a Harness Buy? Tokens, Mostly (arXiv 2610.04433)](https://arxiv.org/abs/2610.04433). 2026-10-03.\n- Yilun Yao and colleagues. [Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows (arXiv 2605.27922)](https://arxiv.org/abs/2605.27922). 2026-05-27.\n\n## Where the numbers come from\n\n- Anthropic, Anthropic raises $30B Series G at $380B valuation (2026-02-12), https://www.anthropic.com/news/anthropic-raises-30-billion-series-g-funding-380-billion-post-money-valuation\n- Cloud Security Alliance, AI coding agent CI prompt injection (research note) (2026-08-07), https://labs.cloudsecurityalliance.org/research/csa-research-note-ai-coding-agent-ci-prompt-injection-202608/\n- Gartner, Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026, Up from Less Than 5% in 2025 (2025-08-26), https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025\n- Harness lab, Experiment 1 (labs/harness-lab/src/traces/runs.json)\n- Harness lab, Experiment 1, median wall time (labs/harness-lab/src/traces/runs.json)\n- Harness lab, Experiment 2 (labs/harness-lab/src/traces/runs.json)\n- Harness lab, Experiment 2, cost per solved task (labs/harness-lab/src/traces/runs.json)\n- Harness lab, Experiment 2, dearest over cheapest cost per solved task (labs/harness-lab/src/traces/runs.json)\n- Harness lab, Experiment 2, prompt tokens served from cache (labs/harness-lab/src/traces/runs.json)\n- Harness lab, Experiment 3 (labs/harness-lab/src/traces/injection.json)\n- Harness lab, Experiment 4: time to exit on an unknown model, exit code 137 (labs/harness-lab/src/traces/control.json)\n- Harness lab, the agent harnesses Stack arm (labs/harness-lab/src/traces/stack.json)\n- JetBrains Research, AI Coding Agents: Adoption Trends (Developer Ecosystem Survey 2026) (2026-08), https://blog.jetbrains.com/research/2026/08/ai-coding-agent-adoption-2026/\n- Kimchi (vendor: makes one of the harnesses), 2026-09-07, as cited on the Agent harnesses track\n- OpenRouter, Ori Harness (2026-11-16), https://openrouter.ai/docs/guides/ori/harness.md\n- Stack Overflow, Stack Overflow Developer Survey 2026: AI (2026-10-06), https://survey.stackoverflow.co/2026/ai\n- Vals.ai (independent), updated 2026-10-01, archived, as cited on the Agent harnesses track\n- Yangze Liu and Zhongyi Han, What Does a Harness Buy? Tokens, Mostly (arXiv 2610.04433) (2026-10-03), https://arxiv.org/abs/2610.04433\n- Yilun Yao and colleagues, Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows (arXiv 2605.27922) (2026-05-27), https://arxiv.org/abs/2605.27922\n\n## Cite this\n\nStack Assay (2026). The harness is the new product layer. We measured four of them, then built one.. Version 2. https://stackassay.com/stories/harness-bench-to-stack/\n\n```bibtex\n@techreport{stackassay-harness-bench-to-stack,\n  title = {The harness is the new product layer. We measured four of them, then built one.},\n  author = {{Stack Assay}},\n  institution = {Stack Assay},\n  year = {2026},\n  month = {11},\n  note = {Version 2},\n  url = {https://stackassay.com/stories/harness-bench-to-stack/}\n}\n```\n",
   "image": "https://stackassay.com/stories/og/harness-bench-to-stack.png",
   "date_published": "2026-11-20T00:00:00Z",
   "date_modified": "2026-11-20T00:00:00Z",
   "tags": [
    "Track to Stack",
    "agent-harnesses"
   ]
  }
 ]
}
