80 completion-selected outcomes across two tool cohorts; 83 observed paid primary attempts and 52 prerequisites. This is a descriptive snapshot, not one randomized five-CLI experiment. Full experiment cost is unknown. Three partial native captures retain valid required grade artifacts. Native tool timing remains partial. Read result interpretation and native workflow findings.
Pass rate, cost, cache and tokens. Compare the harnesses on the same model.
All 8 tasks · 16 runs per harness
Higher means more passes. Further left means lower cost.
Select a point to see its pass rate, cost and distance from the observed best.
Up = more passes
Left = lower cost
Best = 100% for each metric · no combined score
Select a column to sort ↕| cline @ 3.0.61 | 37.5%estimated · 6/16 | $0.308 | $4.931 | 96.4% | 12.67M | 163.4K |
|---|---|---|---|---|---|---|
| codex @ codex-cli 0.149.1 | 81.3% | $0.077 | $1.235 | 98.6% | 6.83M | 73.6K |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | 56.3% | $0.103 | $1.645 | 98.9% | 9.50M | 74.9K |
| pi @ 0.73.1 | 81.3% | $0.062 | $0.997 | 98.4% | 4.44M | 58.3K |
| qwen @ 0.22.2 | 56.3% | $0.080 | $1.280 | 99.0% | 8.04M | 70.1K |
Average cost / taskAverage each task’s runs, then average across the selected tasks.
Whole benchmarkSum all selected runs per harness, including verification failures.
Relative to bestHigher pass/cache rates and lower costs/token counts score higher. No combined score.
Use the task scope and reference harness above. Each metric answers a separate question; no combined score or universal winner.
$0.15 per million uncached input tokens, $0.003 per million cached input tokens, and $0.6 per million output tokens. Reference estimates, not billing. Reference book: deepseek-v41-1017-reference-2026-10-18. Original books: deepseek-v41-1017-reference-2026-10-18.
8 distinct tasks; 2 repetitions per task and harness version. Pass differences and cost means use equal task weights; pass counts include every selected attempt. Recorded verifier image/environment identities match within each task. Recorded configuration details are unavailable. Observed differences are not causal and do not establish future superiority. Exploratory 95% task-cluster intervals use 10,000 deterministic resamples; repetitions are not independent tasks or paired random seeds. No correction for multiple comparisons or winner selection. Secondary cost/token intervals describe means, not headline token medians. Original grades remain unchanged; corrected verification is a separate audit with limited replay coverage.
Time is agent end-to-end elapsed time, excluding setup and verification. All-outcome values include fast failures; successful timing uses only passes. Reference cost includes all selected outcomes and weights tasks equally. Unmeasured workflows: not measured.
0 completed · 16 verification failures · 0 timeouts · 0 adapter errors
Mean $0.3082; exploratory 95% interval: $0.1730 to $0.4683n=16; missing=0; 8 task clusters
13 completed · 3 verification failures · 0 timeouts · 0 adapter errors
Mean $0.07716; exploratory 95% interval: $0.06797 to $0.08644n=16; missing=0; 8 task clusters
9 completed · 7 verification failures · 0 timeouts · 0 adapter errors
Mean $0.1028; exploratory 95% interval: $0.07257 to $0.1396n=16; missing=0; 8 task clusters
13 completed · 3 verification failures · 0 timeouts · 0 adapter errors
Mean $0.06233; exploratory 95% interval: $0.05310 to $0.07186n=16; missing=0; 8 task clusters
9 completed · 7 verification failures · 0 timeouts · 0 adapter errors
Mean $0.08001; exploratory 95% interval: $0.06430 to $0.09732n=16; missing=0; 8 task clusters
| Harness version | Pass difference | Mean cost difference | All-outcome median time difference | Coverage |
|---|---|---|---|---|
| cline @ 3.0.61 | −81.25 ppExploratory 95%: −93.75 pp to −62.5 pp | +$0.2310+299.4% | +12.18 min+149.8% | Exploratory task-cluster pass difference; equal task weights. |
| codex @ codex-cli 0.149.1 (reference) | 0 ppExploratory 95%: Reference | $00% | 0 s0% | Reference row; no comparison interval. |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | −25 ppExploratory 95%: −62.5 pp to 0 pp | +$0.02565+33.25% | +1.264 min+15.55% | Exploratory task-cluster pass difference; equal task weights. |
| pi @ 0.73.1 | 0 ppExploratory 95%: −31.25 pp to +25 pp | −$0.01484−19.23% | −57.32 s−11.75% | Exploratory task-cluster pass difference; equal task weights. |
| qwen @ 0.22.2 | −25 ppExploratory 95%: −62.5 pp to +12.5 pp | +$0.002850+3.694% | +37.7 s+7.727% | Exploratory task-cluster pass difference; equal task weights. |
API-active union (model_time) is proxy-observed API-active time including transport, not provider compute. Non-model residual (non_model_time) includes unobserved tools and harness work. Timing correlation is a hypothesis, not a diagnosed cause; these observations do not establish a full tool/harness split.
These are observed median, quartile and min–max distributions, not confidence intervals. Secondary token mean intervals do not replace the headline token medians. No winner-selection or multiple-comparison correction is applied.
Secondary input Mean 19,820,000; exploratory 95% interval: 11,380,000 to 29,380,000n=16; missing=0; 8 task clusters
Secondary output Mean 227,500; exploratory 95% interval: 138,900 to 329,100n=16; missing=0; 8 task clusters
Visibility: 0 none, 16 partial, 0 full. A visibility flag alone is not native timing qualification.
Evidence labels: 16 verification unknown. Verification-unknown does not diagnose an agent or verifier cause.
Selected run IDs (image/environment identities and request timelines in the linked evidence):
pinned:cline:cattrs-partial-structuring-recovery:0pinned:cline:cattrs-partial-structuring-recovery:1pinned:cline:happy-dom-deterministic-intersectionobserver:0pinned:cline:happy-dom-deterministic-intersectionobserver:1pinned:cline:ink-grid-box-layout:0pinned:cline:ink-grid-box-layout:1pinned:cline:psd-tools-blend-range-api:0pinned:cline:psd-tools-blend-range-api:1pinned:cline:superjson-error-stack-serialization:0pinned:cline:superjson-error-stack-serialization:1pinned:cline:textual-richlog-follow-state:0pinned:cline:textual-richlog-follow-state:1pinned:cline:tomlkit-toml-table-converters:0pinned:cline:tomlkit-toml-table-converters:1pinned:cline:true-myth-iterable-collection-combinators:0pinned:cline:true-myth-iterable-collection-combinators:1Inspect run identities and measurements · Inspect bound failure evidence
Secondary input Mean 6,668,000; exploratory 95% interval: 5,344,000 to 7,972,000n=16; missing=0; 8 task clusters
Secondary output Mean 73,880; exploratory 95% interval: 65,160 to 83,250n=16; missing=0; 8 task clusters
Visibility: 0 none, 16 partial, 0 full. A visibility flag alone is not native timing qualification.
Evidence labels: 3 verification unknown; 13 pass. Verification-unknown does not diagnose an agent or verifier cause.
Selected run IDs (image/environment identities and request timelines in the linked evidence):
pinned:codex:cattrs-partial-structuring-recovery:0pinned:codex:cattrs-partial-structuring-recovery:1pinned:codex:happy-dom-deterministic-intersectionobserver:0pinned:codex:happy-dom-deterministic-intersectionobserver:1pinned:codex:ink-grid-box-layout:0pinned:codex:ink-grid-box-layout:1pinned:codex:psd-tools-blend-range-api:0pinned:codex:psd-tools-blend-range-api:1pinned:codex:superjson-error-stack-serialization:0pinned:codex:superjson-error-stack-serialization:1pinned:codex:textual-richlog-follow-state:0pinned:codex:textual-richlog-follow-state:1pinned:codex:tomlkit-toml-table-converters:0pinned:codex:tomlkit-toml-table-converters:1pinned:codex:true-myth-iterable-collection-combinators:0pinned:codex:true-myth-iterable-collection-combinators:1Inspect run identities and measurements · Inspect bound failure evidence
Secondary input Mean 12,450,000; exploratory 95% interval: 7,016,000 to 18,980,000n=16; missing=0; 8 task clusters
Secondary output Mean 84,660; exploratory 95% interval: 65,720 to 108,500n=16; missing=0; 8 task clusters
Visibility: 0 none, 16 partial, 0 full. A visibility flag alone is not native timing qualification.
Evidence labels: 9 pass; 7 verification unknown. Verification-unknown does not diagnose an agent or verifier cause.
Selected run IDs (image/environment identities and request timelines in the linked evidence):
pinned:hermes:cattrs-partial-structuring-recovery:0pinned:hermes:cattrs-partial-structuring-recovery:1pinned:hermes:happy-dom-deterministic-intersectionobserver:0pinned:hermes:happy-dom-deterministic-intersectionobserver:1pinned:hermes:ink-grid-box-layout:0pinned:hermes:ink-grid-box-layout:1pinned:hermes:psd-tools-blend-range-api:0pinned:hermes:psd-tools-blend-range-api:1pinned:hermes:superjson-error-stack-serialization:0pinned:hermes:superjson-error-stack-serialization:1pinned:hermes:textual-richlog-follow-state:0pinned:hermes:textual-richlog-follow-state:1pinned:hermes:tomlkit-toml-table-converters:0pinned:hermes:tomlkit-toml-table-converters:1pinned:hermes:true-myth-iterable-collection-combinators:0pinned:hermes:true-myth-iterable-collection-combinators:1Inspect run identities and measurements · Inspect bound failure evidence
Secondary input Mean 4,687,000; exploratory 95% interval: 3,501,000 to 5,868,000n=16; missing=0; 8 task clusters
Secondary output Mean 63,140; exploratory 95% interval: 53,430 to 74,160n=16; missing=0; 8 task clusters
Visibility: 0 none, 16 partial, 0 full. A visibility flag alone is not native timing qualification.
Evidence labels: 13 pass; 3 verification unknown. Verification-unknown does not diagnose an agent or verifier cause.
Selected run IDs (image/environment identities and request timelines in the linked evidence):
pinned:pi:cattrs-partial-structuring-recovery:0pinned:pi:cattrs-partial-structuring-recovery:1pinned:pi:happy-dom-deterministic-intersectionobserver:0pinned:pi:happy-dom-deterministic-intersectionobserver:1pinned:pi:ink-grid-box-layout:0pinned:pi:ink-grid-box-layout:1pinned:pi:psd-tools-blend-range-api:0pinned:pi:psd-tools-blend-range-api:1pinned:pi:superjson-error-stack-serialization:0pinned:pi:superjson-error-stack-serialization:1pinned:pi:textual-richlog-follow-state:0pinned:pi:textual-richlog-follow-state:1pinned:pi:tomlkit-toml-table-converters:0pinned:pi:tomlkit-toml-table-converters:1pinned:pi:true-myth-iterable-collection-combinators:0pinned:pi:true-myth-iterable-collection-combinators:1Inspect run identities and measurements · Inspect bound failure evidence
Secondary input Mean 8,597,000; exploratory 95% interval: 5,996,000 to 11,440,000n=16; missing=0; 8 task clusters
Secondary output Mean 71,310; exploratory 95% interval: 58,890 to 85,080n=16; missing=0; 8 task clusters
Visibility: 0 none, 16 partial, 0 full. A visibility flag alone is not native timing qualification.
Evidence labels: 9 pass; 7 verification unknown. Verification-unknown does not diagnose an agent or verifier cause.
Selected run IDs (image/environment identities and request timelines in the linked evidence):
pinned:qwen:cattrs-partial-structuring-recovery:0pinned:qwen:cattrs-partial-structuring-recovery:1pinned:qwen:happy-dom-deterministic-intersectionobserver:0pinned:qwen:happy-dom-deterministic-intersectionobserver:1pinned:qwen:ink-grid-box-layout:0pinned:qwen:ink-grid-box-layout:1pinned:qwen:psd-tools-blend-range-api:0pinned:qwen:psd-tools-blend-range-api:1pinned:qwen:superjson-error-stack-serialization:0pinned:qwen:superjson-error-stack-serialization:1pinned:qwen:textual-richlog-follow-state:0pinned:qwen:textual-richlog-follow-state:1pinned:qwen:tomlkit-toml-table-converters:0pinned:qwen:tomlkit-toml-table-converters:1pinned:qwen:true-myth-iterable-collection-combinators:0pinned:qwen:true-myth-iterable-collection-combinators:1Inspect run identities and measurements · Inspect bound failure evidence
Detailed timing report and request timelines · Download diagnostics
Canonical analysis · Download diagnostics · Detailed report and request timelines
Compare pass rates with a reference harness. A range crossing zero means no clear separation. Differences use percentage points (pp).
8 task clusters · original verification · exploratory intervals
| Harness | Observed difference | Exploratory 95% interval |
|---|---|---|
| pi @ 0.73.1 | 0.0 pp | -31.3 pp to +25.0 ppNo clear separation |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | -25.0 pp | -62.5 pp to 0.0 ppNo clear separation |
| qwen @ 0.22.2 | -25.0 pp | -62.5 pp to +12.5 ppNo clear separation |
| cline @ 3.0.61 | -81.3 pp | -93.8 pp to -62.5 ppDirectional in this sample |
Exploratory 95% percentile task-cluster bootstrap, 10,000 deterministic resamples. Each task and all its repetitions move together; harnesses are paired by task, not by random seed. Marginal intervals are not corrected for multiple comparisons or winner selection. 8 task clusters provide limited evidence about other coding tasks. These intervals do not remove configuration, environment or verifier confounding.
A narrow interval does not prove equal performance. “Best = 100%” is an observed relative index. Corrected audit results are separate and do not enter these intervals.
Pass counts and repetitions; labeled entries are estimated. Select a task to explore its costs and token usage above.
| Task | cline @ 3.0.61 | codex @ codex-cli 0.149.1 | hermes @ Hermes Agent v0.20.5 (2026.8.19) | pi @ 0.73.1 | qwen @ 0.22.2 |
|---|---|---|---|---|---|
| cattrs-partial-structuring-recovery | 0 / 2 est. | 1 / 2 | 1 / 2 | 2 / 2 | 2 / 2 |
| happy-dom-deterministic-intersectionobserver | 0 / 2 est. | 1 / 2 | 1 / 2 | 1 / 2 | 0 / 2 |
| ink-grid-box-layout | 0 / 2 est. | 1 / 2 | 1 / 2 | 2 / 2 | 2 / 2 |
| psd-tools-blend-range-api | 2 / 2 est. | 2 / 2 | 2 / 2 | 2 / 2 | 2 / 2 |
| superjson-error-stack-serialization | 2 / 2 est. | 2 / 2 | 0 / 2 | 0 / 2 | 0 / 2 |
| textual-richlog-follow-state | 1 / 2 est. | 2 / 2 | 2 / 2 | 2 / 2 | 1 / 2 |
| tomlkit-toml-table-converters | 0 / 2 est. | 2 / 2 | 0 / 2 | 2 / 2 | 0 / 2 |
| true-myth-iterable-collection-combinators | 1 / 2 est. | 2 / 2 | 2 / 2 | 2 / 2 | 2 / 2 |
Every view is generated from the same validated report. You can inspect the selected runs, reproduce the tables, and trace the evidence.
Browse snapshot provenance ↗Model deepseek/deepseek-v4.1-flash-1017. 5 harnesses, 8 tasks; repetition counts are shown for each task. Routing, configuration and collection conditions are specific to the selected population and retained in its evidence. Pass rate uses recorded grades or explicitly labeled estimates. A verification failure alone does not distinguish a task failure from verifier infrastructure failure.
Lower token usage alone does not establish better task performance. Relative scores use all measured harness versions within the selected population and task scope, even when some are hidden. A ≥ cost is a known lower bound; incomplete metrics are excluded from relative scoring.
$0.15 per million uncached input tokens, $0.003 per million cached input tokens, and $0.6 per million output tokens. Reference estimates, not billing. Reference book: deepseek-v41-1017-reference-2026-10-18. Original books: deepseek-v41-1017-reference-2026-10-18. Exact request counters feed the existing report calculation. These are reference estimates, not actual billing.
Passes and failures both contribute to costs and token/cache summaries. Cost averages weight tasks equally. Filtering to one task shows its average across repetitions and sum across all selected runs.
This is a completion-selected descriptive snapshot across original tool cohorts. It is not the frozen v1 dataset. The selected tasks do not establish broad capability estimates; repeated runs do not create independent tasks. Recorded image, environment and collection differences limit interpretation; inspect the selected population’s evidence before treating it as a controlled comparison. Detailed reports retain environment identities.
Non-model time includes unobserved tools and harness work. Pass-rate estimates and recorded results are identified in the report.
The current-campaign pointer selects the sanitized analysis export. The README, detailed report and this site share its validated measurements and calculation functions. CI checks generated output without API keys or model calls.
Snapshot: linux-results-2026-10-24
Exported: 2026-10-24T08:12:53.079Z
Canonical analysis JSON · Artifact hashes · Full methodology
Estimated pass rates use the retained per-run audit. Other pass rates use recorded grades. Original results remain in the source data.
Three extra attempts are outside the selected 80. Prefix estimates do not give complete charges.
| Attempt | Outcome | Adapter time | Known prefix reference estimate | Full cost |
|---|---|---|---|---|
| pinned:cline:textual-richlog-follow-state:0 | timeout | 10,800.372 s | $1.836203754 | unknown |
| pinned:cline:tomlkit-toml-table-converters:1 | budget-interrupted | unknown | $0.444080070 | unknown |
| pinned:pi:textual-richlog-follow-state:0 | interrupted | unknown | $0.003784620 | unknown |
All extra attempt facts and captured request series · Original cohorts and collection windows · Collection ledger · Historical snapshot