Ox Alpha Benchmarks: What the Numbers Actually Say

Short answer: ox-alpha did not beat GPT-5.6 Sol. The viral 80% came from a 10-task sample its own author flagged as high-variance. Full 113-task runs put ox-alpha at 58.4% (95% CI: 49.2–67.1%), with larger evaluations converging around 62.8–63% — DeepSeek-V4-Pro territory, a few steps below the frontier leaders. At $0 per token, that's still the best value in coding models right now. Here's the full picture.

How the 80% headline happened

Days after ox-alpha appeared on OpenRouter, a developer running as @davis7 posted a DeepSWE comparison: ox-alpha above 80%, against Claude Fable 5 at 65% and GPT-5.6-sol at 52%. Screenshots traveled; the "stealth model beats the frontier" take wrote itself.

To @davis7's credit, the post itself warned that a 10-task sample carries huge variance. The full runs proved the warning right.

What full runs measured

An independent complete pass over all 113 DeepSWE tasks (MatchaOnMuffins/oxalpha) measured:

• 58.4% resolve rate (66/113), 95% CI [49.2%, 67.1%]
• 80% of tasks landed ≥90% of their fail-to-pass tests — when it solves a task, it usually really solves it
• 9.7% of the benchmark was lost to tool-call format failures — not reasoning failures

Larger third-party evaluations converged around ~62.8–63%. The honest read: ox-alpha sits in the low 60s, with real uncertainty at the edges of that confidence interval.

Where that lands on the leaderboard

Two caveats belong under that table. First, ox-alpha has no official leaderboard entry — every number is third-party, on someone else's harness. Second, the early "level with GPT-5.6-sol" comparisons were against mid-tier Sol configurations; Sol at max reasoning is still ~10 points clear.

The most interesting number isn't the score

It's the 9.7% lost to tool-call format failures. Those are tasks where the model's reasoning was fine but its tool calls came out malformed and the harness scored them as failures. Two implications:

1. The ceiling is higher than the score. Nearly 10 points of headroom is plumbing, not intelligence — and plumbing improves fast, on both sides (model updates and harness fixes).
2. Your harness matters. A coding agent that streams real tool calls, validates them, and retries cleanly recovers points a bare API loop drops. This is exactly where a full agentic environment earns its keep — you can watch every tool call land in real time instead of guessing what the model "really" meant.

The verdict for practitioners

• Not "beats GPT-5.6 Sol." Anyone still repeating the 80% line is quoting a 10-task sample.
• Yes to "DeepSeek-V4-Pro-class performance at $0" — which, per point of DeepSWE, makes it the best-priced coding model available today.
• Best suited to volume agent work: long sessions, parallel panes, fan-out subagents — the workloads where cost dominates and a low-60s model with a 1M context window is more than enough.

Want the mystery side? Who made ox alpha collects every attribution clue, ranked. And to actually run it — four panes in parallel, $0 per token — see ox-alpha in MeshCode.

Benchmark figures are third-party measurements as of August 26, 2026, not official leaderboard entries, and may shift as harnesses and model versions change. Ox Alpha is distributed anonymously via OpenRouter; MeshCode is not affiliated with its developer.