A Chartnaut benchmark
AI models, ranked on real trading research. We give AI models 77 closed research tasks in Chartnaut, then check every number they report against an answer we can prove. Here is how each model performed.
A model solves a task when its saved script, re-run on hidden windows, matches the certified answer and its report matches its own run. The range under each score is its 95% interval.
| # | Model | Run | ||||||
|---|---|---|---|---|---|---|---|---|
| 1tied | Opus 5.5Anthropic | 96%91 to 100 | 94.3 | 82.3 | $0.45$0.43/task · $33.18 run | 494k | 3.3 min | 27 Sep 2026 |
| 1tied | GPT-6 SolOpenAI | 96%91 to 100 | 95.0 | 84.0 | $0.15$0.15/task · $11.41 run | 318k | 3.9 min | 27 to 28 Sep 2026 |
| 1tied | GPT-6 AstraOpenAI | 96%91 to 100 | 95.6 | 91.1 | $0.53$0.51/task · $39.29 run | 235k | 2.2 min | 27 to 29 Sep 2026 |
| 2tied | GPT-5.6 SolOpenAI | 95%90 to 99 | 94.3 | 77.1 | $0.32$0.30/task · $23.19 run | 300k | 3.9 min | 28 Sep 2026 |
| 2tied | GPT-5.6 TerraOpenAI | 95%88 to 100 | 93.7 | 90.8 | $0.17$0.16/task · $12.42 run | 294k | 2.2 min | 28 Sep 2026 |
| 3 | Sonnet 5Anthropic | 90%81 to 96 | 86.8 | 85.1 | $0.53$0.47/task · $37.69 run | 753k | 5.5 min | 26 to 27 Sep 2026 |
| 4 | GPT-5.6 LunaOpenAI | 84%77 to 92 | 83.6 | 68.2 | $0.021$0.018/task · $1.39 run | 299k | 3.2 min | 28 Sep 2026 |
| 5 | DeepSeek V4.1 FlashDeepSeek | 81%71 to 88 | 74.2 | 77.8 | $0.050$0.040/task · $3.07 run | 1.20M | 9.4 min | 28 Sep 2026 |
| 6 | Fable 5.1Anthropic | 79%70 to 88 | 74.2 | 70.0 | $1.63$1.29/task · $99.69 run | 889k | 4.0 min | 27 Sep 2026 |
| 7 | GLM 5.3 FlashZhipu AI | 73%62 to 81 | 64.8 | 69.4 | $0.12$0.088/task · $6.80 run | 1.73M | 9.7 min | 28 to 29 Sep 2026 |
| 8 | GPT-6 LunaOpenAI | 65%53 to 75 | 57.2 | 51.5 | $0.008$0.005/task · $0.42 run | 236k | 2.3 min | 28 to 29 Sep 2026 |
| 9 | Haiku 4.5Anthropic | 48%38 to 58 | 40.3 | 45.5 | $0.38$0.18/task · $13.99 run | 908k | 2.6 min | 27 Sep 2026 |
The range under each score is a 95% bootstrap interval over tasks. Tier-weighted counts Foundation tasks once, Hard twice and Expert three times, so easy tasks cannot carry a score. Weighted also counts code efficiency on solved tasks (see How well the code runs). Tokens include input, output and cache. Cost is the model's recorded tokens priced at its lab's public API list price (standard tier, cache reads and writes at their own rates): cost per solve is the whole run's cost divided by the tasks solved, so a model pays for its failures too. Every model runs in the same harness, with the same tools and limits. 77 tasks in bench v1; an entry is scored on the tasks it was graded on.
Share of the 77 tasks each model solved. The line across each bar is its 95% interval over tasks.
Set a budget per task and see the best model within it, or set the solve rate you need and see the cheapest model that gets there. Cost is the list price of one task, failed attempts included.
Up is more tasks solved, left is cheaper. The line joins the entries nothing beats on both.
Opus 5.5 solves 96% of tasks. It uses 494k tokens and 3.3 min per task on average, and each solved task costs $0.45.
GPT-6 Luna solves 65% at $0.008 per solved task.
A task is a working session, not one question. The model reads Chartnaut's docs, writes scripts, runs them, reads the results and fixes what is wrong. Most of the tokens are usually cached context, read again on each turn.
Share of tasks solved by the kind of work the task asks for. Numbers are percentages.
| Indicators11 tasks | Definitions19 tasks | Research28 tasks | Debugging6 tasks | Reuse2 tasks | Performance2 tasks | Correctness1 tasks | Honesty8 tasks | |
|---|---|---|---|---|---|---|---|---|
| Opus 5.5 · Anthropic | 100 | 95 | 93 | 100 | 100 | 100 | 100 | 100 |
| GPT-6 Sol · OpenAI | 100 | 100 | 89 | 100 | 100 | 100 | 100 | 100 |
| GPT-6 Astra · OpenAI | 100 | 100 | 93 | 100 | 100 | 100 | 100 | 88 |
| GPT-5.6 Sol · OpenAI | 100 | 95 | 93 | 100 | 100 | 100 | 100 | 88 |
| GPT-5.6 Terra · OpenAI | 91 | 95 | 93 | 100 | 100 | 100 | 100 | 100 |
| Sonnet 5 · Anthropic | 82 | 89 | 86 | 100 | 100 | 100 | 100 | 100 |
| GPT-5.6 Luna · OpenAI | 82 | 95 | 82 | 100 | 50 | 100 | 100 | 63 |
| DeepSeek V4.1 Flash · DeepSeek | 100 | 95 | 57 | 83 | 100 | 100 | 100 | 88 |
| Fable 5.1 · Anthropic | 91 | 100 | 57 | 83 | 50 | 100 | 0 | 100 |
| GLM 5.3 Flash · Zhipu AI | 100 | 74 | 46 | 100 | 100 | 100 | 100 | 88 |
| GPT-6 Luna · OpenAI | 73 | 79 | 43 | 67 | 50 | 100 | 100 | 88 |
| Haiku 4.5 · Anthropic | 73 | 63 | 21 | 50 | 50 | 100 | 0 | 63 |
| Average | 91 | 90 | 71 | 90 | 83 | 100 | 83 | 89 |
Every failed attempt gets one cause. Most fall into five broad causes, with a small remainder we keep as Other. The causes are the same for every model. What changes is the mix. In this run, causes are assigned from which grading check the attempt failed.
The script runs, but it computes something other than what was asked.
Why it is common. Nothing errors. The number is plausible, often better than the truth, so nothing prompts a second look.
The logic is right, but it runs on the wrong slice of the market.
Why it is common. These pass on the window the model tested and fail on the hidden ones, which start on different dates.
The data cannot fully answer the question, and the model answered anyway.
Why it is common. The run succeeds. The reason not to trust it is one warning line in the summary.
The model ran out of time or budget, or never produced a script that runs.
Why it happens. Every task has the same time, turn and usage limits. Long chains of work leave less room for a wrong turn early on.
The final answer is not what Chartnaut returned.
Why it matters most. This is the failure that would mislead a trader, because the report sounds confident and specific.
Failures that fit none of the above.
Why we show it. A list of causes with no remainder hides the cases that don't fit. We keep it.
A right answer from a script that takes a minute per chart is not much use to a trader. So we run every script the model saved, not just check its answer, and measure the code itself the same way on every model.
Fast code that gives the wrong answer is worth nothing, so efficiency only counts on solved tasks. A wrong answer scores 0 on that task however quickly it ran. A right answer scores between 0.5 and 1, depending on how efficient the code is.
task score = solved ? efficiency : 0efficiency is 1.0 at or under the reference's run time, falls to 0.5 at 8 times slower, and loses 0.1 for each budget warning. It never goes below 0.5.weighted is the average task score, shown in the leaderboard's Weighted column.Solving still matters most: the weighting can never lift a wrong answer.
Up is more tasks solved; left is faster code. The dashed line at 1.0× is our own reference scripts. The line joins the entries nothing beats on both.
| Model | Solved | Speed vs reference | Efficiency | Weighted |
|---|---|---|---|---|
| GPT-6 Astra · OpenAI | 96% | 1.04× | 0.94 | 91.1▲2 |
| GPT-5.6 Terra · OpenAI | 95% | 1.05× | 0.95 | 90.8▲3 |
| Sonnet 5 · Anthropic | 90% | 1.08× | 0.95 | 85.1▲3 |
| GPT-6 Sol · OpenAI | 96% | 1.51× | 0.86 | 84.0▼2 |
| Opus 5.5 · Anthropic | 96% | 1.62× | 0.84 | 82.3▼4 |
| DeepSeek V4.1 Flash · DeepSeek | 81% | 1.09× | 0.96 | 77.8▲2 |
| GPT-5.6 Sol · OpenAI | 95% | 1.87× | 0.80 | 77.1▼3 |
| Fable 5.1 · Anthropic | 79% | 1.47× | 0.87 | 70.0▲1 |
| GLM 5.3 Flash · Zhipu AI | 73% | 1.11× | 0.95 | 69.4▲1 |
| GPT-5.6 Luna · OpenAI | 84% | 1.96× | 0.79 | 68.2▼3 |
| GPT-6 Luna · OpenAI | 65% | 2.72× | 0.76 | 51.5· |
| Haiku 4.5 · Anthropic | 48% | 1.06× | 0.94 | 45.5· |
Speed is the median over solved attempts; above 1.0× is slower than the reference. Efficiency is the average over solved attempts only, so a model is not rewarded for fast code on answers it got wrong. The arrow shows places gained or lost against the Solved ranking.
const all = ctx.accum("h", [], (p) => [...p, ctx.high]);
const hi = Math.max(...all.slice(-21, -1)); // copies every bar so far, every barconst last = ctx.accum("h", [], (p) => [...p, ctx.high].slice(-21));
const hi = Math.max(...last.slice(0, -1)); // 21 values, whatever the chart lengthAn illustration, not a task. Both give the same answer, so both pass the correctness checks. The first gets slower the longer the chart, and on long histories of minute bars it runs into the processing limits.
Every task sits in a tier, and every task above Foundation in a capability area. The tasks themselves stay closed. Below is what each tier and area asks of a model, with a generic example of the kind of work involved.
Separates working models from broken ones. One script, a precise spec, one answer.
Separates good models from average ones. Each task hits at least two things that make trading code go wrong quietly.
Separates the best models from each other. Several artifacts, long chains, and answers that must hold on data the model never saw.
Bar times arrive in UTC. Exchange clocks, daylight saving changes, weekends and session boundaries are the model's job.
A generic illustration. Real tasks give exact instruments, windows and output shapes, and are re-run on windows the model never sees.
Also in the framework, with no graded tasks in this run: layers and rendering, apps, long-horizon builds.
Chartnaut is a trading platform where AI writes the research. That only works if the numbers are right.
In Chartnaut, a trader describes an idea and an AI model builds it: the indicator that draws it, the definition that marks every time it happened, the study that measures what came next. The trader then makes decisions from those numbers. A model that writes code which runs but counts the wrong thing is worse than no model at all, because the answer looks trustworthy.
General coding benchmarks do not tell us how often that happens. A model can be excellent at writing software and still read tomorrow's high into today's bar. So we built a benchmark out of the work Chartnaut asks models to do, graded against answers we can prove.
We use these results when deciding which models Chartnaut offers for each kind of work. A model has to do well on the areas a job needs, not just on the headline number.
Every entry runs on the same bench version, with its run date on the table. As new models come out we run them the same way, so the table shows how model capability on trading research changes over time.
Every failure gets a cause. When many attempts stumble in the same place, we make that part of Chartnaut clearer, in the docs, the error messages or the tools, then measure again on the next version.
What this means for traders. You can choose a model on evidence from work like yours, and you know where to double-check: a result that looks too clean, a count of zero, an answer about history Chartnaut does not have.
Think of it as an exam where we mark the working as well as the answer, then hand the student a fresh set of numbers to see if the working still holds.
A task in plain English, the kind a trader would type, with the exact result shape to hand back. It works in our harness: an empty Chartnaut project, the Chartnaut CLI, tools to read and edit files, 20 minutes and a fixed usage budget. Nothing else.
The model writes indicators, definitions and studies, and runs them on Chartnaut's servers. It never sees raw prices, so it cannot work the answer out in its head or look it up. The only way to the number is a script that is right.
A short result: a number, a count, a yes or no, or "this can't be answered, because...". It also names the run the answer came from.
Does the named run exist, from the model's own saved script, on the window we asked for? Does the answer match what that run returned? And when we re-run the script on hidden windows it never saw, does it still match the certified answer? All three must pass.
Each task's answer is worked out twice: once by a Chartnaut script we wrote, once by a separate program written from scratch from raw bars, sharing no code with Chartnaut. A task enters the set only when the two agree on every window, public and hidden.
A script can be right on the dates it was tested on and wrong everywhere else: a warmup that is too short, a session hardcoded for one season. Re-running the model's own script on dates it never saw catches that. The result must match the model's run, and the run must hold up on the hidden windows.
The range under each score is a 95% bootstrap interval: we resample the tasks thousands of times and see how far the score moves. When two entries' ranges overlap, we cannot honestly say one is better, so they share a rank.
Every model runs in one harness we control, driven through the Chartnaut CLI. Same system prompt, same tools, same time, turn and usage limits. No lab's own coding app is involved, so a score reflects the model, not the wrapper around it.
Tier-weighted counts Foundation 1×, Hard 2×, Expert 3×, so a model cannot reach the top on easy tasks alone. Solved is the plain share of tasks solved, and it is what the ranks and ranges use.
Whether a strategy makes money. How nice the charts look. It measures one thing: given a research question, does the model get the right number and tell you the truth about it.
Once tasks are public, models get trained on them and the scores stop meaning anything. The task set stays closed. We describe what each area and tier tests, and add new tasks on market data that did not exist when the listed models were trained.
Because then you are comparing the apps as much as the models. One app retries more, another reads more files first. Every model here runs in the same harness, with the same tools and limits, so the only thing that changes between rows is the model.
No. The table ranks models from AI labs, run the same way. Ranking our own product here would mean marking our own homework.
No. It means the model is more likely to give a correct number about the past, and to say when it cannot. It does not measure whether a strategy makes money.
This is the first published run. More models go on the table as they are run on the current bench version, each with its run date.