What the July 2026 benchmark map is actually telling procurement
Three numbers hit the enterprise buying desk this month, and each one is being read against the wrong artifact. Anthropic's Claude Opus 4.8 posts 88.6% on SWE-bench Verified — the top published number on the coding-agent leaderboard. OpenAI's Codex + GPT-5.5 posts 83.4% on Terminal-Bench 2.1, ahead of Claude Code + Fable 5 at 83.1%. And per the Q2 2026 aggregate across enterprise agentic AI systems, the lab-benchmark-score to production-deployment-performance gap is 37 percentage points on average, with a 50× per-agent cost variance for the same accuracy tier.
The operationally important reads:
- The 88.6% SWE-bench Verified score is the vendor-marketing headline, and it is the wrong artifact for the enterprise buying decision. SWE-bench Verified grades against single-repo, single-issue tasks with the test contract already written. The enterprise coding-agent workload is multi-file refactors against a partial test contract, dependency upgrades against a private-registry pin set, structured extraction against a schema whose invariants live in a legacy code path. The 88.6% number does not price against the enterprise workload shape; the per-workload-class replay against the team's own repository does.
- The Terminal-Bench 2.1 leaderboard closes to a rounding-error gap at the top, and the ranking-order flip week-over-week is not a per-workload signal. 83.4% versus 83.1% is a 0.3-point delta on a 200-task benchmark — the ranking flips as the benchmark refreshes. The team routing its FY27 shortlist against last week's Terminal-Bench ranking is chasing the leaderboard as a moving target, not writing the routing policy against the per-workload-class portability envelope the standing contract underwrites.
- The 37-point lab-to-production gap is the artifact the vendor pitch deck does not show, and the 50× cost variance is the artifact the CFO signs against. The gap is where the vendor's benchmark-tuned scaffolding stops carrying, the team's actual repo topology stops matching the benchmark harness, and the per-workload verifier stops firing at the rate the benchmark harness fires against. The 50× cost variance is what happens when the team writes the model-routing policy against the accuracy number and does not price the per-token cost, the per-agent turn budget, and the per-task retry-multiplier that ship with the same accuracy tier.
The structural read isn't the benchmarks are broken. It is that the leaderboard grades against a shape the enterprise coding-agent workload does not match, the per-workload-class replay against the team's own repository is the load-bearing signal for the routing policy, and the domain-eval artifact — not the vendor benchmark — is what the FY27 standing contract negotiation should grade against.
What the leaderboard is measuring versus what the buying decision needs
SWE-bench Verified measures single-issue, single-repo task closure against a written test contract. The benchmark is a rigorous artifact for the research question of coding-agent capability against the SWE-bench shape. It is not the artifact for the procurement question of which agent survives contact with the team's private monorepo. The two questions grade against different distributions of code base age, dependency graph complexity, test-coverage density, and repository-conventions drift. The team that ships the FY27 shortlist against SWE-bench Verified score is grading its buying decision against a distribution that does not include its own code.
Terminal-Bench 2.1 measures end-to-end task completion in a terminal sandbox against a curated task set. The benchmark grades scaffolding-plus-model as one integrated system — the same model in a different harness returns a different score. The 83.4% Codex + GPT-5.5 number is not portable to the Claude Code harness; the 83.1% Claude Code + Fable 5 number is not portable to the Codex harness. The team that reads the leaderboard as a per-model ranking is treating a per-model × per-harness signal as if the harness variable had been controlled out.
SWE-bench Pro is where the leaderboard shape moves closest to the enterprise workload shape. Scale AI's SEAL lab built SWE-bench Pro with real multi-file changes across enterprise-scale repositories, standardised scaffolding with a 250-turn limit, and per-agent identical tooling — the harness variable is controlled. The load-bearing read for the FY27 shortlist is the SWE-bench Pro rank, not the SWE-bench Verified rank; Pro grades against the shape closer to the enterprise workload, and the harness controlled-variable means the score compares agents rather than agent-harness pairs.
The per-workload-class replay against the team's own repository is the domain eval that closes the 37-point gap. The domain eval is not a benchmark — it is a per-team artifact: the 30 most-representative tickets from the team's last two sprints, replayed against each candidate agent, graded against the team's own test suite, and scored against per-workload-class success rate, per-task per-token cost, per-task 95th-percentile latency, per-task turn budget consumed, and per-task human-intervention-required rate. The domain eval is where the vendor's marketing rank meets the team's actual code, and the artifact is what closes the buying-decision gap the vendor leaderboard cannot close.
Why the 50× cost variance shows up on the CFO desk
Same accuracy tier does not mean same per-successful-task cost. Two agents that both post 78% on the team's domain eval can carry a 50× per-successful-task cost delta driven by three variables the benchmark score does not surface: the per-token pricing tier (Fable 5 API-pricing versus Sonnet 5 promotional versus Gemini 3.5 Flash), the per-task turn-budget the agent's scaffolding consumes (an agent that solves the task in 12 turns costs 5× less than one that solves it in 60 turns at the same per-token cost), and the per-task retry multiplier the harness carries when the first attempt fails (the harness that spins up two parallel workers on retry doubles the per-successful-task cost against a harness that retries in a single worker).
The per-successful-task cost is the metric the FY27 budget grades against, not the per-token price the vendor publishes. The per-token price is one input into the per-successful-task cost; the per-task turn budget, per-task retry multiplier, and per-task human-intervention rate are the other three. The team that grades the routing policy against per-token price is under-pricing the tail of the FY27 budget by the factor of the ignored variables.
The per-task 95th-percentile latency is the SLO variable that drives the per-feature UX, and the leaderboard rank does not surface it. An agent that completes 78% of tasks at a 40-second p50 and a 12-minute p95 latency ships the p95 into the customer-facing feature envelope; an agent that completes 78% of tasks at a 90-second p50 and a 3-minute p95 ships a tighter tail. The customer-facing feature grades against the p95, not the p50 and not the mean. The team's per-feature SLO artifact grades against the p95, and the p95 does not show up on the leaderboard.
What the shortlist workflow looks like when it grades against the domain eval
Step 1: cut the shortlist from four to two on published benchmark rank, not on vendor loyalty. The four-vendor frontier map (Anthropic, OpenAI, Google, and one open-weight substrate) is the honest starting shortlist. The SWE-bench Pro rank and Terminal-Bench 2.1 rank cut the shortlist to the two agents whose leaderboard shape is closest to the enterprise workload shape — for most teams, the top two are Claude Code + Fable 5 (or Opus 4.8) and Codex + GPT-5.5 (or GPT-5.6 where the team has access). The other two frontier entries stay on the portability envelope as the second-source replacement path, not on the shortlist.
Step 2: run the domain eval against the two shortlist agents on the team's actual code. The domain eval takes 30 representative tickets from the last two sprints, wraps them in the same prompt harness the production feature will ship with, runs each ticket against each agent on identical scaffolding, and captures per-workload-class success rate, per-task per-token cost, per-task turn budget, per-task 95th-percentile latency, and per-task human-intervention rate. The domain eval is a two-week engineering artifact — not a six-month benchmark — and it is the load-bearing signal for the routing-policy artifact the team ships into the repo.
Step 3: write the per-prompt routing policy against the domain-eval matrix, not against the leaderboard rank. The policy artifact codifies the per-workload-class default-route decision, the per-workload-class escalation-path decision, the per-workload-class turn-budget cap, and the per-workload-class retry rule. The artifact is versioned in the team's repo, referenced from the per-feature client library, and re-graded per quarter against the refreshed domain eval.
Step 4: negotiate the FY27 standing contract against the per-workload-class portability envelope the domain eval surfaces. The envelope specifies the per-workload-class fallback substrate the team can route to on 30-days notice of a vendor capability slip, the per-workload-class quarterly re-shootout cadence the team runs against the substrate, and the per-vendor commitment against the aggregate spend the standing contract carries. The envelope is what makes the standing contract portable; without it, the contract is a per-vendor spend commitment with no exit ramp against the substrate-shift risk the FY28 frontier will keep shipping.
Where the 37-point gap is signal and where it is noise
Signal: the gap is not shrinking, and the FY28 frontier will not close it. Every leaderboard refresh moves the top-of-leaderboard number up; the enterprise workload shape does not change at the same rate. The team that defers the domain-eval artifact until the leaderboard cracks 95% is deferring against a gap that is a durable feature of the benchmark-versus-workload shape mismatch, not a transient consequence of the current models' maturity.
Signal: the 50× per-agent cost variance concentrates in the harness variables, not the model variables. The domain-eval matrix is what surfaces the concentration; the per-token pricing table is what the vendor pitch deck surfaces. The team that grades against the vendor pitch deck is under-pricing the harness variables that drive the per-successful-task cost.
Noise: the leaderboard is broken, so all benchmarks are useless is the wrong overcorrection. The leaderboard is a signal for the shortlist cut — SWE-bench Pro and Terminal-Bench 2.1 cut the four-vendor field to two candidates. The mistake is treating the leaderboard as the buying decision signal; the domain eval is the buying-decision signal, and the leaderboard is the shortlist-cut signal.
Noise: the domain eval is too expensive to run per quarter under-scopes the per-cycle re-grading cost. The 30-ticket domain eval runs in two engineer-weeks against the team's own repository. The per-quarter re-grading against the refreshed domain eval is the artifact that keeps the routing policy honest against the substrate-shift cadence the frontier ships against. The team that skips the per-quarter re-grade is running the FY27 policy against the FY26 domain eval by Q4 and reads the KPI regression against the substrate-shift the domain eval would have surfaced.
What the engineering team should do inside the next four weeks
Ship the 30-ticket domain-eval harness against the top two shortlist agents. Pull the 30 most-representative tickets from the last two sprints, wrap them in the production prompt harness, and run each ticket against each shortlist agent on identical scaffolding. The output is the per-workload-class matrix the routing policy grades against.
Instrument the per-task turn budget, per-task 95th-percentile latency, and per-task human-intervention rate on the production coding-agent surface. The three metrics are what close the 37-point lab-to-production gap in the team's own observability surface. Without them, the team reads the KPI regression as noise; with them, the team grades the per-workload-class routing decision against the substrate the agent actually runs on.
Write the per-workload-class portability envelope against the two-agent shortlist and the second-source replacement path. The envelope codifies the per-workload-class fallback substrate, the per-quarter re-shootout cadence, and the per-vendor commitment against the standing contract. The envelope is the artifact against which the FY27 standing-contract negotiation grades.
Re-grade the domain-eval matrix per quarter and re-write the routing policy against the refresh. The domain eval is the load-bearing signal; the routing policy is the artifact the team ships into the repo; the per-quarter re-grade is what keeps the two in sync against the substrate-shift cadence the frontier ships against.
At SONNET CODE we run the AI Development engagement against the per-workload-class domain-eval matrix — 30-ticket domain-eval harnesses on the team's own repository, per-workload-class routing policies against the four-vendor frontier map, per-vendor portability envelopes on the FY27 standing contract, and per-quarter re-shootouts against the refreshed substrate. If your team is buying against the SWE-bench Verified rank, schedule a call — we'll walk you through the domain-eval artifact we ship inside four weeks.
