80 completion-selected outcomes across two tool cohorts; 83 observed paid primary attempts and 52 prerequisites. This is a descriptive snapshot, not one randomized five-CLI experiment. Full experiment cost is unknown. Three partial native captures retain valid required grade artifacts. Native tool timing remains partial. Read result interpretation and native workflow findings.
Cline: 37.5% estimated (6/16). Method notes.
AGENT OVERHEAD BENCH / CAMPAIGN SNAPSHOT
Pass rate. Cost. Cache. Tokens. Explore the measurements behind each harness.
Selected attempts: 80. Comparison groups: 1.
All selected outcomes · task coverage can differ · task-average and whole-benchmark costs · each metric's best = 100%
All selected outcomes. Task coverage can differ; these are descriptive summaries, not a ranking. Cost per task is the mean across repetitions, then equally averaged across tasks. Whole benchmark cost sums every selected run, including failures. Cache and tokens are medians per measured attempt. Partial costs are known lower bounds. Cost is a static estimate, not billing. Cache rate is the median attempt cached-input percentage, not a pooled token ratio. Input includes cached tokens; token counters cover successful model responses. Missing measurements are not zero.
Percent of observed best: higher pass/cache values score higher; lower cost/token values score higher. Each metric has its own 100% baseline. Partial measurement coverage is unscored. These descriptive indices are not a composite quality score; task coverage can differ.
K = 1,000; M = 1,000,000. Focus a value for its exact accessible value. Overview values include every selected outcome and do not change with the detail outcome filter.
All harnesses combined: $10.088 for 80 selected runs at the shared reference prices.
Average across each task’s repetitions; all-repetition totals sum to the whole benchmark. ≥ marks a known lower bound where accounting is incomplete.
| Harness | Task | Runs | Average cost / task | All repetitions cost | Complete cost coverage |
|---|---|---|---|---|---|
| cline | cattrs-partial-structuring-recovery | 2 | $0.352 | $0.704 | 2/2 |
| cline | happy-dom-deterministic-intersectionobserver | 2 | $0.181 | $0.363 | 2/2 |
| cline | ink-grid-box-layout | 2 | $0.203 | $0.407 | 2/2 |
| cline | psd-tools-blend-range-api | 2 | $0.254 | $0.508 | 2/2 |
| cline | superjson-error-stack-serialization | 2 | $0.098 | $0.196 | 2/2 |
| cline | textual-richlog-follow-state | 2 | $0.780 | $1.560 | 2/2 |
| cline | tomlkit-toml-table-converters | 2 | $0.502 | $1.004 | 2/2 |
| cline | true-myth-iterable-collection-combinators | 2 | $0.094 | $0.189 | 2/2 |
| codex | cattrs-partial-structuring-recovery | 2 | $0.069 | $0.137 | 2/2 |
| codex | happy-dom-deterministic-intersectionobserver | 2 | $0.066 | $0.132 | 2/2 |
| codex | ink-grid-box-layout | 2 | $0.090 | $0.181 | 2/2 |
| codex | psd-tools-blend-range-api | 2 | $0.080 | $0.161 | 2/2 |
| codex | superjson-error-stack-serialization | 2 | $0.061 | $0.122 | 2/2 |
| codex | textual-richlog-follow-state | 2 | $0.096 | $0.192 | 2/2 |
| codex | tomlkit-toml-table-converters | 2 | $0.093 | $0.186 | 2/2 |
| codex | true-myth-iterable-collection-combinators | 2 | $0.062 | $0.123 | 2/2 |
| hermes | cattrs-partial-structuring-recovery | 2 | $0.089 | $0.177 | 2/2 |
| hermes | happy-dom-deterministic-intersectionobserver | 2 | $0.077 | $0.154 | 2/2 |
| hermes | ink-grid-box-layout | 2 | $0.074 | $0.149 | 2/2 |
| hermes | psd-tools-blend-range-api | 2 | $0.079 | $0.157 | 2/2 |
| hermes | superjson-error-stack-serialization | 2 | $0.044 | $0.089 | 2/2 |
| hermes | textual-richlog-follow-state | 2 | $0.157 | $0.315 | 2/2 |
| hermes | tomlkit-toml-table-converters | 2 | $0.205 | $0.411 | 2/2 |
| hermes | true-myth-iterable-collection-combinators | 2 | $0.097 | $0.194 | 2/2 |
| pi | cattrs-partial-structuring-recovery | 2 | $0.045 | $0.089 | 2/2 |
| pi | happy-dom-deterministic-intersectionobserver | 2 | $0.059 | $0.118 | 2/2 |
| pi | ink-grid-box-layout | 2 | $0.076 | $0.153 | 2/2 |
| pi | psd-tools-blend-range-api | 2 | $0.047 | $0.095 | 2/2 |
| pi | superjson-error-stack-serialization | 2 | $0.053 | $0.107 | 2/2 |
| pi | textual-richlog-follow-state | 2 | $0.064 | $0.128 | 2/2 |
| pi | tomlkit-toml-table-converters | 2 | $0.087 | $0.174 | 2/2 |
| pi | true-myth-iterable-collection-combinators | 2 | $0.067 | $0.134 | 2/2 |
| qwen | cattrs-partial-structuring-recovery | 2 | $0.074 | $0.149 | 2/2 |
| qwen | happy-dom-deterministic-intersectionobserver | 2 | $0.076 | $0.152 | 2/2 |
| qwen | ink-grid-box-layout | 2 | $0.109 | $0.218 | 2/2 |
| qwen | psd-tools-blend-range-api | 2 | $0.053 | $0.107 | 2/2 |
| qwen | superjson-error-stack-serialization | 2 | $0.049 | $0.098 | 2/2 |
| qwen | textual-richlog-follow-state | 2 | $0.087 | $0.175 | 2/2 |
| qwen | tomlkit-toml-table-converters | 2 | $0.123 | $0.247 | 2/2 |
| qwen | true-myth-iterable-collection-combinators | 2 | $0.068 | $0.136 | 2/2 |
Reference cost at fixed prices from deepseek-v41-1017-reference-2026-10-18: $0.15/M uncached input + $0.003/M cached input + $0.6/M output tokens. Calculated from exact request counters, averaged across repetitions per task and summed for the whole benchmark; not actual billing. Original price books remain in the evidence below.
All selected outcomes; these denominators remain visible when outcome filters are applied.
| Harness @ version | Selected | Pass | Verify fail | Other | Tasks | Timing n | Timing tasks | Turns n | Input n | Output n | Cache n | Cost n | IQR tasks / successful timing tasks |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| cline @ 3.0.61 | 16 | 0 | 16 | 0 | 8 | 16 | 8 | 16 | 16 | 16 | 16 | 16 | 0 / 0 |
| codex @ codex-cli 0.149.1 | 16 | 13 | 3 | 0 | 8 | 16 | 8 | 16 | 16 | 16 | 16 | 16 | 5 / 8 |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | 16 | 9 | 7 | 0 | 8 | 16 | 8 | 16 | 16 | 16 | 16 | 16 | 3 / 6 |
| pi @ 0.73.1 | 16 | 13 | 3 | 0 | 8 | 16 | 8 | 16 | 16 | 16 | 16 | 16 | 6 / 7 |
| qwen @ 0.22.2 | 16 | 9 | 7 | 0 | 8 | 16 | 8 | 16 | 16 | 16 | 16 | 16 | 4 / 5 |
| Harness @ version | Outcome | Selected | Priced n | Missing n | Selected static spend (USD) |
|---|---|---|---|---|---|
| cline @ 3.0.61 | Verification failure (verify_error) | 16 | 16 | 0 | Total: 4.93125 |
| codex @ codex-cli 0.149.1 | Verification failure (verify_error) | 3 | 3 | 0 | Total: 0.2066405 |
| codex @ codex-cli 0.149.1 | Pass (completed) | 13 | 13 | 0 | Total: 1.027939 |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | Pass (completed) | 9 | 9 | 0 | Total: 0.9623701 |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | Verification failure (verify_error) | 7 | 7 | 0 | Total: 0.6826842 |
| pi @ 0.73.1 | Pass (completed) | 13 | 13 | 0 | Total: 0.8236724 |
| pi @ 0.73.1 | Verification failure (verify_error) | 3 | 3 | 0 | Total: 0.1735329 |
| qwen @ 0.22.2 | Pass (completed) | 9 | 9 | 0 | Total: 0.6966012 |
| qwen @ 0.22.2 | Verification failure (verify_error) | 7 | 7 | 0 | Total: 0.5835826 |
| Harness @ version | Outcome | Runs | Median calls | Median of run median durations | Median of run median waits | Median of run median transfers | Median of run median gaps | Median time after last |
|---|---|---|---|---|---|---|---|---|
| cline @ 3.0.61 | Verification failure (verify_error) | 16 | 252.5 (available n=16; missing n=0) | 2.33s (available n=16; missing n=0) | 1.15s (available n=16; missing n=0) | 1.14s (available n=16; missing n=0) | 48.36447 ms (available n=16; missing n=0) | 1.48s (available n=16; missing n=0) |
| codex @ codex-cli 0.149.1 | Verification failure (verify_error) | 3 | 49 (available n=3; missing n=0) | 3.62s (available n=3; missing n=0) | 1.28s (available n=3; missing n=0) | 2.35s (available n=3; missing n=0) | 60.34602 ms (available n=3; missing n=0) | 5.16s (available n=3; missing n=0) |
| codex @ codex-cli 0.149.1 | Pass (completed) | 13 | 64 (available n=13; missing n=0) | 3.62s (available n=13; missing n=0) | 1.33s (available n=13; missing n=0) | 2.25s (available n=13; missing n=0) | 57.6276 ms (available n=13; missing n=0) | 5.15s (available n=13; missing n=0) |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | Pass (completed) | 9 | 102 (available n=9; missing n=0) | 2.65s (available n=9; missing n=0) | 1.32s (available n=9; missing n=0) | 1.31s (available n=9; missing n=0) | 166.4325 ms (available n=9; missing n=0) | 250.4241 ms (available n=9; missing n=0) |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | Verification failure (verify_error) | 7 | 65 (available n=7; missing n=0) | 3.43s (available n=7; missing n=0) | 1.25s (available n=7; missing n=0) | 1.98s (available n=7; missing n=0) | 162.8287 ms (available n=7; missing n=0) | 244.1931 ms (available n=7; missing n=0) |
| pi @ 0.73.1 | Pass (completed) | 13 | 59 (available n=13; missing n=0) | 2.81s (available n=13; missing n=0) | 1.24s (available n=13; missing n=0) | 1.61s (available n=13; missing n=0) | 38.59564 ms (available n=13; missing n=0) | 174.7571 ms (available n=13; missing n=0) |
| pi @ 0.73.1 | Verification failure (verify_error) | 3 | 39 (available n=3; missing n=0) | 2.93s (available n=3; missing n=0) | 1.2s (available n=3; missing n=0) | 1.78s (available n=3; missing n=0) | 142.0054 ms (available n=3; missing n=0) | 186.4018 ms (available n=3; missing n=0) |
| qwen @ 0.22.2 | Pass (completed) | 9 | 78 (available n=9; missing n=0) | 2.66s (available n=9; missing n=0) | 1.28s (available n=9; missing n=0) | 1.32s (available n=9; missing n=0) | 46.9397 ms (available n=9; missing n=0) | 180.8887 ms (available n=9; missing n=0) |
| qwen @ 0.22.2 | Verification failure (verify_error) | 7 | 94 (available n=7; missing n=0) | 2.76s (available n=7; missing n=0) | 1.29s (available n=7; missing n=0) | 1.42s (available n=7; missing n=0) | 47.65603 ms (available n=7; missing n=0) | 187.145 ms (available n=7; missing n=0) |
| Left | Right | Common tasks | Left median of task medians | Right median of task medians | Median task ratio (right / left) |
|---|---|---|---|---|---|
| cline @ 3.0.61 | codex @ codex-cli 0.149.1 | 0 | unavailable | unavailable | unavailable |
| cline @ 3.0.61 | hermes @ Hermes Agent v0.20.5 (2026.8.19) | 0 | unavailable | unavailable | unavailable |
| cline @ 3.0.61 | pi @ 0.73.1 | 0 | unavailable | unavailable | unavailable |
| cline @ 3.0.61 | qwen @ 0.22.2 | 0 | unavailable | unavailable | unavailable |
| codex @ codex-cli 0.149.1 | hermes @ Hermes Agent v0.20.5 (2026.8.19) | 6 | 8m 19.59s | 11m 25.53s | 1.415092 |
| codex @ codex-cli 0.149.1 | pi @ 0.73.1 | 7 | 8m 24.93s | 6m 47.48s | 0.8070086 |
| codex @ codex-cli 0.149.1 | qwen @ 0.22.2 | 5 | 8m 24.93s | 8m 23.53s | 0.8777608 |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | pi @ 0.73.1 | 6 | 11m 25.53s | 6m 42.23s | 0.6076702 |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | qwen @ 0.22.2 | 5 | 13m 25.58s | 8m 23.53s | 0.6250561 |
| pi @ 0.73.1 | qwen @ 0.22.2 | 5 | 6m 47.48s | 8m 23.53s | 1.235712 |
Descriptive common-success-task comparisons. Open a pair for task names and per-side sample counts. A zero common-task count means no comparison is available.
Common successful tasks: 0. Left median of task medians: unavailable. Right median of task medians: unavailable. Median task ratio (right / left): unavailable.
No common successful tasks; comparison unavailable.
Common successful tasks: 0. Left median of task medians: unavailable. Right median of task medians: unavailable. Median task ratio (right / left): unavailable.
No common successful tasks; comparison unavailable.
Common successful tasks: 0. Left median of task medians: unavailable. Right median of task medians: unavailable. Median task ratio (right / left): unavailable.
No common successful tasks; comparison unavailable.
Common successful tasks: 0. Left median of task medians: unavailable. Right median of task medians: unavailable. Median task ratio (right / left): unavailable.
No common successful tasks; comparison unavailable.
Common successful tasks: 6. Left median of task medians: 8m 19.59s. Right median of task medians: 11m 25.53s. Median task ratio (right / left): 1.415092.
| Common successful task | Left n | Left median | Right n | Right median | Right / left |
|---|---|---|---|---|---|
| cattrs-partial-structuring-recovery | 1 | 8m 24.93s | 1 | 13m 25.58s | 1.595425 |
| happy-dom-deterministic-intersectionobserver | 1 | 8m 14.26s | 1 | 9m 25.48s | 1.144099 |
| ink-grid-box-layout | 1 | 10m 43.54s | 1 | 16m 44.31s | 1.560606 |
| psd-tools-blend-range-api | 2 | 7m 0.91s | 2 | 8m 7.38s | 1.157929 |
| textual-richlog-follow-state | 2 | 12m 51.73s | 2 | 18m 25.74s | 1.432811 |
| true-myth-iterable-collection-combinators | 2 | 6m 5.5s | 2 | 8m 30.74s | 1.397372 |
Common successful tasks: 7. Left median of task medians: 8m 24.93s. Right median of task medians: 6m 47.48s. Median task ratio (right / left): 0.8070086.
| Common successful task | Left n | Left median | Right n | Right median | Right / left |
|---|---|---|---|---|---|
| cattrs-partial-structuring-recovery | 1 | 8m 24.93s | 2 | 6m 47.48s | 0.8070086 |
| happy-dom-deterministic-intersectionobserver | 1 | 8m 14.26s | 1 | 6m 36.97s | 0.8031637 |
| ink-grid-box-layout | 1 | 10m 43.54s | 2 | 11m 4.31s | 1.032266 |
| psd-tools-blend-range-api | 2 | 7m 0.91s | 2 | 4m 29.96s | 0.641363 |
| textual-richlog-follow-state | 2 | 12m 51.73s | 2 | 8m 39.28s | 0.6728802 |
| tomlkit-toml-table-converters | 2 | 10m 12.63s | 2 | 9m 44.66s | 0.954342 |
| true-myth-iterable-collection-combinators | 2 | 6m 5.5s | 2 | 6m 30.86s | 1.069391 |
Common successful tasks: 5. Left median of task medians: 8m 24.93s. Right median of task medians: 8m 23.53s. Median task ratio (right / left): 0.8777608.
| Common successful task | Left n | Left median | Right n | Right median | Right / left |
|---|---|---|---|---|---|
| cattrs-partial-structuring-recovery | 1 | 8m 24.93s | 2 | 8m 23.53s | 0.9972304 |
| ink-grid-box-layout | 1 | 10m 43.54s | 2 | 18m 31.08s | 1.726509 |
| psd-tools-blend-range-api | 2 | 7m 0.91s | 2 | 5m 12.82s | 0.743211 |
| textual-richlog-follow-state | 2 | 12m 51.73s | 1 | 11m 17.39s | 0.8777608 |
| true-myth-iterable-collection-combinators | 2 | 6m 5.5s | 2 | 5m 7.79s | 0.8421073 |
Common successful tasks: 6. Left median of task medians: 11m 25.53s. Right median of task medians: 6m 42.23s. Median task ratio (right / left): 0.6076702.
| Common successful task | Left n | Left median | Right n | Right median | Right / left |
|---|---|---|---|---|---|
| cattrs-partial-structuring-recovery | 1 | 13m 25.58s | 2 | 6m 47.48s | 0.5058266 |
| happy-dom-deterministic-intersectionobserver | 1 | 9m 25.48s | 1 | 6m 36.97s | 0.7020055 |
| ink-grid-box-layout | 1 | 16m 44.31s | 2 | 11m 4.31s | 0.6614523 |
| psd-tools-blend-range-api | 2 | 8m 7.38s | 2 | 4m 29.96s | 0.5538882 |
| textual-richlog-follow-state | 2 | 18m 25.74s | 2 | 8m 39.28s | 0.4696226 |
| true-myth-iterable-collection-combinators | 2 | 8m 30.74s | 2 | 6m 30.86s | 0.7652871 |
Common successful tasks: 5. Left median of task medians: 13m 25.58s. Right median of task medians: 8m 23.53s. Median task ratio (right / left): 0.6250561.
| Common successful task | Left n | Left median | Right n | Right median | Right / left |
|---|---|---|---|---|---|
| cattrs-partial-structuring-recovery | 1 | 13m 25.58s | 2 | 8m 23.53s | 0.6250561 |
| ink-grid-box-layout | 1 | 16m 44.31s | 2 | 18m 31.08s | 1.106307 |
| psd-tools-blend-range-api | 2 | 8m 7.38s | 2 | 5m 12.82s | 0.6418453 |
| textual-richlog-follow-state | 2 | 18m 25.74s | 1 | 11m 17.39s | 0.6126147 |
| true-myth-iterable-collection-combinators | 2 | 8m 30.74s | 2 | 5m 7.79s | 0.6026362 |
Common successful tasks: 5. Left median of task medians: 6m 47.48s. Right median of task medians: 8m 23.53s. Median task ratio (right / left): 1.235712.
| Common successful task | Left n | Left median | Right n | Right median | Right / left |
|---|---|---|---|---|---|
| cattrs-partial-structuring-recovery | 2 | 6m 47.48s | 2 | 8m 23.53s | 1.235712 |
| ink-grid-box-layout | 2 | 11m 4.31s | 2 | 18m 31.08s | 1.672542 |
| psd-tools-blend-range-api | 2 | 4m 29.96s | 2 | 5m 12.82s | 1.158799 |
| textual-richlog-follow-state | 2 | 8m 39.28s | 1 | 11m 17.39s | 1.304483 |
| true-myth-iterable-collection-combinators | 2 | 6m 30.86s | 2 | 5m 7.79s | 0.7874643 |
One dot per timing-eligible run; hover or focus for run ID, outcome and duration. All rows in this population share an end-to-end duration axis. Vertical offsets separate repeated observations; they have no quantitative meaning.
Task identity: ["cattrs-partial-structuring-recovery","91df27b2a20f26c35777a92f4555438b62c06345","sha256:b985f2bd17e9e80bee0fa9b7c0747b2ded92d54d070b32ebfe102e8829a44076","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 42m 51.16s | 29m 54.23s | 55m 48.08s | 16m 57.31s | 68m 45s |
| Model | 2 | 0 | 30m 59.73s | 21m 45.88s | 40m 13.58s | 12m 32.03s | 49m 27.43s |
| Non-model | 2 | 0 | 11m 50.22s | 8m 7.14s | 15m 33.3s | 4m 24.06s | 19m 16.38s |
| Startup | 2 | 0 | 1.21s | 1.2s | 1.21s | 1.19s | 1.22s |
| First byte | 2 | 0 | 1.15s | 1.14s | 1.16s | 1.12s | 1.18s |
| API turns | 2 | 0 | 576 | 398 | 754 | 220 | 932 |
| Input tokens | 2 | 0 | 28596110 | 19731320 | 37460890 | 10866540 | 46325680 |
| Output tokens | 2 | 0 | 267557 | 186360 | 348754 | 105163 | 429951 |
| Cached input (%) | 2 | 0 | 97.60649 | 97.50982 | 97.70317 | 97.41314 | 97.79985 |
| Static cost (USD) | 2 | 0 | 0.3519759 | 0.241409 | 0.4625428 | 0.1308422 | 0.5731096 |
| Model request count | 2 | 0 | 576 | 398 | 754 | 220 | 932 |
| Within-run median request duration | 2 | 0 | 2.24s | 2.22s | 2.26s | 2.19s | 2.28s |
| Within-run median request wait | 2 | 0 | 1.15s | 1.14s | 1.16s | 1.12s | 1.18s |
| Within-run median request transfer | 2 | 0 | 1.01s | 958.8827 ms | 1.05s | 912.2614 ms | 1.1s |
| Within-run median inter-request gap | 2 | 0 | 60.06667 ms | 50.58795 ms | 69.54539 ms | 41.10922 ms | 79.02412 ms |
| Largest request duration | 2 | 0 | 31.3s | 29.58s | 33.03s | 27.85s | 34.76s |
| Largest inter-request gap | 2 | 0 | 30.08s | 30.06s | 30.1s | 30.05s | 30.11s |
| Time after last model response | 2 | 0 | 1.52s | 1.44s | 1.6s | 1.35s | 1.69s |
Task identity: ["happy-dom-deterministic-intersectionobserver","32b2a8e60eab482c023af377160bf294d6da0318","sha256:a4df294272f2165ce894306ab1eec5284fe426ddaa66cf08bb3c4755a9f8faa2","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 18m 6.73s | 15m 45.8s | 20m 27.65s | 13m 24.88s | 22m 48.58s |
| Model | 2 | 0 | 14m 57.7s | 12m 6.61s | 17m 48.8s | 9m 15.52s | 20m 39.89s |
| Non-model | 2 | 0 | 3m 7.22s | 2m 37.06s | 3m 37.38s | 2m 6.9s | 4m 7.55s |
| Startup | 2 | 0 | 1.8s | 1.79s | 1.81s | 1.79s | 1.81s |
| First byte | 2 | 0 | 1.13s | 1.11s | 1.15s | 1.1s | 1.16s |
| API turns | 2 | 0 | 226.5 | 174.75 | 278.25 | 123 | 330 |
| Input tokens | 2 | 0 | 10706940 | 8288461 | 13125420 | 5869981 | 15543900 |
| Output tokens | 2 | 0 | 139832.5 | 111770.8 | 167894.3 | 83709 | 195956 |
| Cached input (%) | 2 | 0 | 96.1652 | 95.80499 | 96.52541 | 95.44477 | 96.88563 |
| Static cost (USD) | 2 | 0 | 0.1814995 | 0.1381042 | 0.2248948 | 0.09470885 | 0.2682901 |
| Model request count | 2 | 0 | 226.5 | 174.75 | 278.25 | 123 | 330 |
| Within-run median request duration | 2 | 0 | 2.35s | 2.27s | 2.43s | 2.18s | 2.51s |
| Within-run median request wait | 2 | 0 | 1.13s | 1.11s | 1.15s | 1.1s | 1.16s |
| Within-run median request transfer | 2 | 0 | 1.06s | 1.03s | 1.09s | 993.2209 ms | 1.13s |
| Within-run median inter-request gap | 2 | 0 | 45.21996 ms | 41.90573 ms | 48.53419 ms | 38.59149 ms | 51.84842 ms |
| Largest request duration | 2 | 0 | 44.27s | 38.21s | 50.33s | 32.16s | 56.39s |
| Largest inter-request gap | 2 | 0 | 30.04s | 30.04s | 30.05s | 30.04s | 30.05s |
| Time after last model response | 2 | 0 | 1.53s | 1.49s | 1.56s | 1.45s | 1.6s |
Task identity: ["ink-grid-box-layout","477fda7df7781406d2ba046eb48dd5391e964b72","sha256:73daa7eb11dec0d43e523434d28a66ee9d3ac063fda33d9e26183d867739dbb1","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 24m 36.31s | 21m 12.35s | 28m 0.26s | 17m 48.39s | 31m 24.22s |
| Model | 2 | 0 | 19m 44.81s | 17m 0.22s | 22m 29.4s | 14m 15.63s | 25m 14s |
| Non-model | 2 | 0 | 4m 49.22s | 4m 10.11s | 5m 28.33s | 3m 30.99s | 6m 7.44s |
| Startup | 2 | 0 | 2.28s | 2.02s | 2.53s | 1.77s | 2.79s |
| First byte | 2 | 0 | 1.16s | 1.16s | 1.17s | 1.15s | 1.18s |
| API turns | 2 | 0 | 269 | 222 | 316 | 175 | 363 |
| Input tokens | 2 | 0 | 14481920 | 11752770 | 17211060 | 9023628 | 19940210 |
| Output tokens | 2 | 0 | 165905 | 151232 | 180578 | 136559 | 195251 |
| Cached input (%) | 2 | 0 | 96.99301 | 96.76303 | 97.22298 | 96.53306 | 97.45296 |
| Static cost (USD) | 2 | 0 | 0.2033123 | 0.1791533 | 0.2274713 | 0.1549944 | 0.2516303 |
| Model request count | 2 | 0 | 269 | 222 | 316 | 175 | 363 |
| Within-run median request duration | 2 | 0 | 2.33s | 2.32s | 2.34s | 2.3s | 2.35s |
| Within-run median request wait | 2 | 0 | 1.16s | 1.16s | 1.17s | 1.15s | 1.18s |
| Within-run median request transfer | 2 | 0 | 1.16s | 1.16s | 1.16s | 1.16s | 1.16s |
| Within-run median inter-request gap | 2 | 0 | 48.39781 ms | 48.01641 ms | 48.77921 ms | 47.63501 ms | 49.16061 ms |
| Largest request duration | 2 | 0 | 46.51s | 44.42s | 48.6s | 42.33s | 50.69s |
| Largest inter-request gap | 2 | 0 | 30.06s | 30.06s | 30.06s | 30.05s | 30.06s |
| Time after last model response | 2 | 0 | 1.43s | 1.41s | 1.45s | 1.39s | 1.47s |
Task identity: ["psd-tools-blend-range-api","01595992c5f4d4695251fdf58b3edb5302d3ba46","sha256:fbfcc5b7e68a2d8fd97688c69808049d582113368f388929d7d46ac44a592cd9","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 25m 44.33s | 20m 43.5s | 30m 45.15s | 15m 42.67s | 35m 45.98s |
| Model | 2 | 0 | 20m 54.85s | 17m 15.59s | 24m 34.12s | 13m 36.33s | 28m 13.38s |
| Non-model | 2 | 0 | 4m 48.18s | 3m 26.61s | 6m 9.74s | 2m 5.05s | 7m 31.31s |
| Startup | 2 | 0 | 1.29s | 1.29s | 1.29s | 1.29s | 1.3s |
| First byte | 2 | 0 | 1.14s | 1.13s | 1.15s | 1.12s | 1.16s |
| API turns | 2 | 0 | 365.5 | 294.75 | 436.25 | 224 | 507 |
| Input tokens | 2 | 0 | 19044360 | 14985270 | 23103450 | 10926180 | 27162540 |
| Output tokens | 2 | 0 | 189242 | 160299 | 218185 | 131356 | 247128 |
| Cached input (%) | 2 | 0 | 96.79369 | 96.52256 | 97.06482 | 96.25144 | 97.33594 |
| Static cost (USD) | 2 | 0 | 0.2539685 | 0.2128841 | 0.2950529 | 0.1717997 | 0.3361373 |
| Model request count | 2 | 0 | 365.5 | 294.75 | 436.25 | 224 | 507 |
| Within-run median request duration | 2 | 0 | 2.31s | 2.31s | 2.32s | 2.3s | 2.32s |
| Within-run median request wait | 2 | 0 | 1.14s | 1.13s | 1.15s | 1.12s | 1.16s |
| Within-run median request transfer | 2 | 0 | 1.09s | 1.09s | 1.09s | 1.09s | 1.1s |
| Within-run median inter-request gap | 2 | 0 | 50.43535 ms | 48.30266 ms | 52.56805 ms | 46.16996 ms | 54.70074 ms |
| Largest request duration | 2 | 0 | 41.8s | 41.02s | 42.59s | 40.23s | 43.37s |
| Largest inter-request gap | 2 | 0 | 27.37s | 26.08s | 28.66s | 24.78s | 29.95s |
| Time after last model response | 2 | 0 | 1.7s | 1.51s | 1.89s | 1.33s | 2.08s |
Task identity: ["superjson-error-stack-serialization","f143004734ea2dd4090e45013a07f091f5bbaee7","sha256:dcc10590630fcc927468099eb5118e4bc7170a1f12e01ca5c9404011de072444","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 9m 43.58s | 8m 52.28s | 10m 34.88s | 8m 0.98s | 11m 26.18s |
| Model | 2 | 0 | 9m 25.45s | 8m 34.28s | 10m 16.63s | 7m 43.1s | 11m 7.81s |
| Non-model | 2 | 0 | 16.27s | 16.15s | 16.4s | 16.03s | 16.52s |
| Startup | 2 | 0 | 1.85s | 1.85s | 1.85s | 1.85s | 1.85s |
| First byte | 2 | 0 | 1.16s | 1.14s | 1.18s | 1.12s | 1.2s |
| API turns | 2 | 0 | 92 | 84.5 | 99.5 | 77 | 107 |
| Input tokens | 2 | 0 | 4718935 | 4296629 | 5141241 | 3874323 | 5563547 |
| Output tokens | 2 | 0 | 95419.5 | 91940.75 | 98898.25 | 88462 | 102377 |
| Cached input (%) | 2 | 0 | 96.10447 | 96.02511 | 96.18383 | 95.94574 | 96.26319 |
| Static cost (USD) | 2 | 0 | 0.09823409 | 0.09301214 | 0.103456 | 0.08779019 | 0.108678 |
| Model request count | 2 | 0 | 92 | 84.5 | 99.5 | 77 | 107 |
| Within-run median request duration | 2 | 0 | 2.84s | 2.78s | 2.91s | 2.72s | 2.97s |
| Within-run median request wait | 2 | 0 | 1.16s | 1.14s | 1.18s | 1.12s | 1.2s |
| Within-run median request transfer | 2 | 0 | 1.6s | 1.6s | 1.61s | 1.6s | 1.61s |
| Within-run median inter-request gap | 2 | 0 | 41.29052 ms | 40.97526 ms | 41.60578 ms | 40.66 ms | 41.92104 ms |
| Largest request duration | 2 | 0 | 1m 52.9s | 1m 43.09s | 2m 2.7s | 1m 33.29s | 2m 12.51s |
| Largest inter-request gap | 2 | 0 | 1.41s | 1.39s | 1.43s | 1.37s | 1.45s |
| Time after last model response | 2 | 0 | 1.34s | 1.27s | 1.41s | 1.2s | 1.48s |
Task identity: ["textual-richlog-follow-state","a12215b791587a6d94227bd30a0e9291de932106","sha256:73fd3d800de9d5dae47edd9b5269e40966664b72d49d8f598964b8b91fdd0b92","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 85m 17.74s | 77m 1.54s | 93m 33.94s | 68m 45.34s | 101m 50.14s |
| Model | 2 | 0 | 60m 55.05s | 56m 30.9s | 65m 19.2s | 52m 6.75s | 69m 43.35s |
| Non-model | 2 | 0 | 24m 21.42s | 20m 29.38s | 28m 13.47s | 16m 37.33s | 32m 5.52s |
| Startup | 2 | 0 | 1.27s | 1.26s | 1.27s | 1.26s | 1.27s |
| First byte | 2 | 0 | 1.15s | 1.15s | 1.16s | 1.14s | 1.16s |
| API turns | 2 | 0 | 1021.5 | 898.75 | 1144.25 | 776 | 1267 |
| Input tokens | 2 | 0 | 46257860 | 40802020 | 51713710 | 35346170 | 57169560 |
| Output tokens | 2 | 0 | 494898 | 452980.5 | 536815.5 | 411063 | 578733 |
| Cached input (%) | 2 | 0 | 94.98392 | 94.87807 | 95.08976 | 94.77223 | 95.1956 |
| Static cost (USD) | 2 | 0 | 0.7801969 | 0.6912522 | 0.8691415 | 0.6023076 | 0.9580862 |
| Model request count | 2 | 0 | 1021.5 | 898.75 | 1144.25 | 776 | 1267 |
| Within-run median request duration | 2 | 0 | 2.58s | 2.45s | 2.7s | 2.33s | 2.83s |
| Within-run median request wait | 2 | 0 | 1.15s | 1.15s | 1.16s | 1.14s | 1.16s |
| Within-run median request transfer | 2 | 0 | 1.36s | 1.25s | 1.46s | 1.14s | 1.57s |
| Within-run median inter-request gap | 2 | 0 | 114.4724 ms | 103.7456 ms | 125.1993 ms | 93.01874 ms | 135.9261 ms |
| Largest request duration | 2 | 0 | 49.33s | 38.05s | 1m 0.6s | 26.78s | 1m 11.88s |
| Largest inter-request gap | 2 | 0 | 30.2s | 30.18s | 30.21s | 30.16s | 30.23s |
| Time after last model response | 2 | 0 | 2.01s | 1.86s | 2.16s | 1.71s | 2.31s |
Task identity: ["tomlkit-toml-table-converters","962de91aaf103135f400e5108a798a59a10dc17e","sha256:68dd891458463c9f930dcbbd59b4442065c145d66ce3b5813af82ca5ec8f4966","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 49m 33.34s | 36m 14.2s | 62m 52.48s | 22m 55.07s | 76m 11.61s |
| Model | 2 | 0 | 47m 4.14s | 34m 32.09s | 59m 36.19s | 22m 0.04s | 72m 8.24s |
| Non-model | 2 | 0 | 2m 27.97s | 1m 40.88s | 3m 15.06s | 53.79s | 4m 2.15s |
| Startup | 2 | 0 | 1.22s | 1.22s | 1.23s | 1.21s | 1.23s |
| First byte | 2 | 0 | 1.12s | 1.11s | 1.12s | 1.11s | 1.12s |
| API turns | 2 | 0 | 598.5 | 439.75 | 757.25 | 281 | 916 |
| Input tokens | 2 | 0 | 28379730 | 21399380 | 35360070 | 14419040 | 42340410 |
| Output tokens | 2 | 0 | 397370.5 | 293821.8 | 500919.3 | 190273 | 604468 |
| Cached input (%) | 2 | 0 | 96.27201 | 95.70973 | 96.8343 | 95.14745 | 97.39658 |
| Static cost (USD) | 2 | 0 | 0.5021649 | 0.3573839 | 0.6469458 | 0.2126029 | 0.7917268 |
| Model request count | 2 | 0 | 598.5 | 439.75 | 757.25 | 281 | 916 |
| Within-run median request duration | 2 | 0 | 2.66s | 2.63s | 2.69s | 2.6s | 2.72s |
| Within-run median request wait | 2 | 0 | 1.12s | 1.11s | 1.12s | 1.11s | 1.12s |
| Within-run median request transfer | 2 | 0 | 1.49s | 1.46s | 1.52s | 1.42s | 1.55s |
| Within-run median inter-request gap | 2 | 0 | 69.42084 ms | 59.25739 ms | 79.58429 ms | 49.09394 ms | 89.74773 ms |
| Largest request duration | 2 | 0 | 1m 39.25s | 1m 19.61s | 1m 58.89s | 59.97s | 2m 18.53s |
| Largest inter-request gap | 2 | 0 | 18.93s | 13.37s | 24.49s | 7.81s | 30.05s |
| Time after last model response | 2 | 0 | 1.57s | 1.56s | 1.58s | 1.55s | 1.59s |
Task identity: ["true-myth-iterable-collection-combinators","5945ad1b96e228ff716275d8a273282101152508","sha256:0ecebbb45e561b37ee5d7bcbe80588427f4890fb80fffbbb73ca6b94c7cb8855","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 7m 54.89s | 7m 47.11s | 8m 2.68s | 7m 39.33s | 8m 10.46s |
| Model | 2 | 0 | 7m 4.92s | 6m 51.67s | 7m 18.16s | 6m 38.42s | 7m 31.41s |
| Non-model | 2 | 0 | 48.75s | 43.29s | 54.21s | 37.82s | 59.67s |
| Startup | 2 | 0 | 1.23s | 1.23s | 1.23s | 1.23s | 1.23s |
| First byte | 2 | 0 | 1.14s | 1.14s | 1.15s | 1.13s | 1.15s |
| API turns | 2 | 0 | 117.5 | 115.25 | 119.75 | 113 | 122 |
| Input tokens | 2 | 0 | 6349182 | 6311184 | 6387181 | 6273185 | 6425179 |
| Output tokens | 2 | 0 | 70057 | 69856.5 | 70257.5 | 69656 | 70458 |
| Cached input (%) | 2 | 0 | 96.44369 | 96.43926 | 96.44811 | 96.43484 | 96.45253 |
| Static cost (USD) | 2 | 0 | 0.09427288 | 0.09412182 | 0.09442393 | 0.09397076 | 0.09457499 |
| Model request count | 2 | 0 | 117.5 | 115.25 | 119.75 | 113 | 122 |
| Within-run median request duration | 2 | 0 | 2.21s | 2.21s | 2.21s | 2.2s | 2.22s |
| Within-run median request wait | 2 | 0 | 1.14s | 1.14s | 1.15s | 1.13s | 1.15s |
| Within-run median request transfer | 2 | 0 | 1.07s | 1.03s | 1.11s | 985.8472 ms | 1.15s |
| Within-run median inter-request gap | 2 | 0 | 37.42782 ms | 36.96961 ms | 37.88603 ms | 36.5114 ms | 38.34425 ms |
| Largest request duration | 2 | 0 | 21.18s | 19.99s | 22.37s | 18.8s | 23.56s |
| Largest inter-request gap | 2 | 0 | 5.15s | 4.74s | 5.57s | 4.33s | 5.98s |
| Time after last model response | 2 | 0 | 1.29s | 1.23s | 1.34s | 1.18s | 1.4s |
Task identity: ["cattrs-partial-structuring-recovery","91df27b2a20f26c35777a92f4555438b62c06345","sha256:b985f2bd17e9e80bee0fa9b7c0747b2ded92d54d070b32ebfe102e8829a44076","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 8m 24.93s | unavailable | unavailable | 8m 24.93s | 8m 24.93s |
| Model | 1 | 0 | 6m 28.7s | unavailable | unavailable | 6m 28.7s | 6m 28.7s |
| Non-model | 1 | 0 | 1m 55.6s | unavailable | unavailable | 1m 55.6s | 1m 55.6s |
| Startup | 1 | 0 | 636.6449 ms | unavailable | unavailable | 636.6449 ms | 636.6449 ms |
| First byte | 1 | 0 | 1.33s | unavailable | unavailable | 1.33s | 1.33s |
| API turns | 1 | 0 | 83 | unavailable | unavailable | 83 | 83 |
| Input tokens | 1 | 0 | 9120964 | unavailable | unavailable | 9120964 | 9120964 |
| Output tokens | 1 | 0 | 64441 | unavailable | unavailable | 64441 | 64441 |
| Cached input (%) | 1 | 0 | 98.99445 | unavailable | unavailable | 98.99445 | 98.99445 |
| Static cost (USD) | 1 | 0 | 0.07950974 | unavailable | unavailable | 0.07950974 | 0.07950974 |
| Model request count | 1 | 0 | 83 | unavailable | unavailable | 83 | 83 |
| Within-run median request duration | 1 | 0 | 2.8s | unavailable | unavailable | 2.8s | 2.8s |
| Within-run median request wait | 1 | 0 | 1.33s | unavailable | unavailable | 1.33s | 1.33s |
| Within-run median request transfer | 1 | 0 | 1.47s | unavailable | unavailable | 1.47s | 1.47s |
| Within-run median inter-request gap | 1 | 0 | 58.27591 ms | unavailable | unavailable | 58.27591 ms | 58.27591 ms |
| Largest request duration | 1 | 0 | 45.08s | unavailable | unavailable | 45.08s | 45.08s |
| Largest inter-request gap | 1 | 0 | 10.02s | unavailable | unavailable | 10.02s | 10.02s |
| Time after last model response | 1 | 0 | 5.15s | unavailable | unavailable | 5.15s | 5.15s |
Task identity: ["cattrs-partial-structuring-recovery","91df27b2a20f26c35777a92f4555438b62c06345","sha256:b985f2bd17e9e80bee0fa9b7c0747b2ded92d54d070b32ebfe102e8829a44076","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 6m 47.68s | unavailable | unavailable | 6m 47.68s | 6m 47.68s |
| Model | 1 | 0 | 4m 36.58s | unavailable | unavailable | 4m 36.58s | 4m 36.58s |
| Non-model | 1 | 0 | 2m 10.5s | unavailable | unavailable | 2m 10.5s | 2m 10.5s |
| Startup | 1 | 0 | 592.9269 ms | unavailable | unavailable | 592.9269 ms | 592.9269 ms |
| First byte | 1 | 0 | 1.28s | unavailable | unavailable | 1.28s | 1.28s |
| API turns | 1 | 0 | 41 | unavailable | unavailable | 41 | 41 |
| Input tokens | 1 | 0 | 3910582 | unavailable | unavailable | 3910582 | 3910582 |
| Output tokens | 1 | 0 | 55590 | unavailable | unavailable | 55590 | 55590 |
| Cached input (%) | 1 | 0 | 97.83505 | unavailable | unavailable | 97.83505 | 97.83505 |
| Static cost (USD) | 1 | 0 | 0.05753106 | unavailable | unavailable | 0.05753106 | 0.05753106 |
| Model request count | 1 | 0 | 41 | unavailable | unavailable | 41 | 41 |
| Within-run median request duration | 1 | 0 | 3.62s | unavailable | unavailable | 3.62s | 3.62s |
| Within-run median request wait | 1 | 0 | 1.28s | unavailable | unavailable | 1.28s | 1.28s |
| Within-run median request transfer | 1 | 0 | 2.43s | unavailable | unavailable | 2.43s | 2.43s |
| Within-run median inter-request gap | 1 | 0 | 56.97248 ms | unavailable | unavailable | 56.97248 ms | 56.97248 ms |
| Largest request duration | 1 | 0 | 46.15s | unavailable | unavailable | 46.15s | 46.15s |
| Largest inter-request gap | 1 | 0 | 30.18s | unavailable | unavailable | 30.18s | 30.18s |
| Time after last model response | 1 | 0 | 5.13s | unavailable | unavailable | 5.13s | 5.13s |
Task identity: ["happy-dom-deterministic-intersectionobserver","32b2a8e60eab482c023af377160bf294d6da0318","sha256:a4df294272f2165ce894306ab1eec5284fe426ddaa66cf08bb3c4755a9f8faa2","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 8m 14.26s | unavailable | unavailable | 8m 14.26s | 8m 14.26s |
| Model | 1 | 0 | 6m 2.91s | unavailable | unavailable | 6m 2.91s | 6m 2.91s |
| Non-model | 1 | 0 | 2m 10.16s | unavailable | unavailable | 2m 10.16s | 2m 10.16s |
| Startup | 1 | 0 | 1.18s | unavailable | unavailable | 1.18s | 1.18s |
| First byte | 1 | 0 | 1.34s | unavailable | unavailable | 1.34s | 1.34s |
| API turns | 1 | 0 | 64 | unavailable | unavailable | 64 | 64 |
| Input tokens | 1 | 0 | 5966339 | unavailable | unavailable | 5966339 | 5966339 |
| Output tokens | 1 | 0 | 65271 | unavailable | unavailable | 65271 | 65271 |
| Cached input (%) | 1 | 0 | 98.33514 | unavailable | unavailable | 98.33514 | 98.33514 |
| Static cost (USD) | 1 | 0 | 0.07166327 | unavailable | unavailable | 0.07166327 | 0.07166327 |
| Model request count | 1 | 0 | 64 | unavailable | unavailable | 64 | 64 |
| Within-run median request duration | 1 | 0 | 3.14s | unavailable | unavailable | 3.14s | 3.14s |
| Within-run median request wait | 1 | 0 | 1.34s | unavailable | unavailable | 1.34s | 1.34s |
| Within-run median request transfer | 1 | 0 | 1.79s | unavailable | unavailable | 1.79s | 1.79s |
| Within-run median inter-request gap | 1 | 0 | 55.76959 ms | unavailable | unavailable | 55.76959 ms | 55.76959 ms |
| Largest request duration | 1 | 0 | 1m 3.66s | unavailable | unavailable | 1m 3.66s | 1m 3.66s |
| Largest inter-request gap | 1 | 0 | 30.17s | unavailable | unavailable | 30.17s | 30.17s |
| Time after last model response | 1 | 0 | 5.14s | unavailable | unavailable | 5.14s | 5.14s |
Task identity: ["happy-dom-deterministic-intersectionobserver","32b2a8e60eab482c023af377160bf294d6da0318","sha256:a4df294272f2165ce894306ab1eec5284fe426ddaa66cf08bb3c4755a9f8faa2","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 5m 52.68s | unavailable | unavailable | 5m 52.68s | 5m 52.68s |
| Model | 1 | 0 | 4m 47.18s | unavailable | unavailable | 4m 47.18s | 4m 47.18s |
| Non-model | 1 | 0 | 1m 3.5s | unavailable | unavailable | 1m 3.5s | 1m 3.5s |
| Startup | 1 | 0 | 2s | unavailable | unavailable | 2s | 2s |
| First byte | 1 | 0 | 1.27s | unavailable | unavailable | 1.27s | 1.27s |
| API turns | 1 | 0 | 49 | unavailable | unavailable | 49 | 49 |
| Input tokens | 1 | 0 | 4364608 | unavailable | unavailable | 4364608 | 4364608 |
| Output tokens | 1 | 0 | 58221 | unavailable | unavailable | 58221 | 58221 |
| Cached input (%) | 1 | 0 | 98.08349 | unavailable | unavailable | 98.08349 | 98.08349 |
| Static cost (USD) | 1 | 0 | 0.06032268 | unavailable | unavailable | 0.06032268 | 0.06032268 |
| Model request count | 1 | 0 | 49 | unavailable | unavailable | 49 | 49 |
| Within-run median request duration | 1 | 0 | 3.66s | unavailable | unavailable | 3.66s | 3.66s |
| Within-run median request wait | 1 | 0 | 1.27s | unavailable | unavailable | 1.27s | 1.27s |
| Within-run median request transfer | 1 | 0 | 2.35s | unavailable | unavailable | 2.35s | 2.35s |
| Within-run median inter-request gap | 1 | 0 | 60.34602 ms | unavailable | unavailable | 60.34602 ms | 60.34602 ms |
| Largest request duration | 1 | 0 | 33.26s | unavailable | unavailable | 33.26s | 33.26s |
| Largest inter-request gap | 1 | 0 | 29.75s | unavailable | unavailable | 29.75s | 29.75s |
| Time after last model response | 1 | 0 | 5.16s | unavailable | unavailable | 5.16s | 5.16s |
Task identity: ["ink-grid-box-layout","477fda7df7781406d2ba046eb48dd5391e964b72","sha256:73daa7eb11dec0d43e523434d28a66ee9d3ac063fda33d9e26183d867739dbb1","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 10m 43.54s | unavailable | unavailable | 10m 43.54s | 10m 43.54s |
| Model | 1 | 0 | 7m 43.96s | unavailable | unavailable | 7m 43.96s | 7m 43.96s |
| Non-model | 1 | 0 | 2m 52.72s | unavailable | unavailable | 2m 52.72s | 2m 52.72s |
| Startup | 1 | 0 | 6.86s | unavailable | unavailable | 6.86s | 6.86s |
| First byte | 1 | 0 | 1.32s | unavailable | unavailable | 1.32s | 1.32s |
| API turns | 1 | 0 | 79 | unavailable | unavailable | 79 | 79 |
| Input tokens | 1 | 0 | 8350746 | unavailable | unavailable | 8350746 | 8350746 |
| Output tokens | 1 | 0 | 86489 | unavailable | unavailable | 86489 | 86489 |
| Cached input (%) | 1 | 0 | 98.77345 | unavailable | unavailable | 98.77345 | 98.77345 |
| Static cost (USD) | 1 | 0 | 0.09200226 | unavailable | unavailable | 0.09200226 | 0.09200226 |
| Model request count | 1 | 0 | 79 | unavailable | unavailable | 79 | 79 |
| Within-run median request duration | 1 | 0 | 3.36s | unavailable | unavailable | 3.36s | 3.36s |
| Within-run median request wait | 1 | 0 | 1.32s | unavailable | unavailable | 1.32s | 1.32s |
| Within-run median request transfer | 1 | 0 | 2.06s | unavailable | unavailable | 2.06s | 2.06s |
| Within-run median inter-request gap | 1 | 0 | 215.8768 ms | unavailable | unavailable | 215.8768 ms | 215.8768 ms |
| Largest request duration | 1 | 0 | 53.43s | unavailable | unavailable | 53.43s | 53.43s |
| Largest inter-request gap | 1 | 0 | 30.17s | unavailable | unavailable | 30.17s | 30.17s |
| Time after last model response | 1 | 0 | 5.17s | unavailable | unavailable | 5.17s | 5.17s |
Task identity: ["ink-grid-box-layout","477fda7df7781406d2ba046eb48dd5391e964b72","sha256:73daa7eb11dec0d43e523434d28a66ee9d3ac063fda33d9e26183d867739dbb1","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 14m 27.87s | unavailable | unavailable | 14m 27.87s | 14m 27.87s |
| Model | 1 | 0 | 8m 58.27s | unavailable | unavailable | 8m 58.27s | 8m 58.27s |
| Non-model | 1 | 0 | 5m 22.88s | unavailable | unavailable | 5m 22.88s | 5m 22.88s |
| Startup | 1 | 0 | 6.73s | unavailable | unavailable | 6.73s | 6.73s |
| First byte | 1 | 0 | 1.3s | unavailable | unavailable | 1.3s | 1.3s |
| API turns | 1 | 0 | 82 | unavailable | unavailable | 82 | 82 |
| Input tokens | 1 | 0 | 8838931 | unavailable | unavailable | 8838931 | 8838931 |
| Output tokens | 1 | 0 | 82642 | unavailable | unavailable | 82642 | 82642 |
| Cached input (%) | 1 | 0 | 99.02374 | unavailable | unavailable | 99.02374 | 99.02374 |
| Static cost (USD) | 1 | 0 | 0.08878677 | unavailable | unavailable | 0.08878677 | 0.08878677 |
| Model request count | 1 | 0 | 82 | unavailable | unavailable | 82 | 82 |
| Within-run median request duration | 1 | 0 | 3.44s | unavailable | unavailable | 3.44s | 3.44s |
| Within-run median request wait | 1 | 0 | 1.3s | unavailable | unavailable | 1.3s | 1.3s |
| Within-run median request transfer | 1 | 0 | 2.27s | unavailable | unavailable | 2.27s | 2.27s |
| Within-run median inter-request gap | 1 | 0 | 69.70515 ms | unavailable | unavailable | 69.70515 ms | 69.70515 ms |
| Largest request duration | 1 | 0 | 1m 19.35s | unavailable | unavailable | 1m 19.35s | 1m 19.35s |
| Largest inter-request gap | 1 | 0 | 30.18s | unavailable | unavailable | 30.18s | 30.18s |
| Time after last model response | 1 | 0 | 5.16s | unavailable | unavailable | 5.16s | 5.16s |
Task identity: ["psd-tools-blend-range-api","01595992c5f4d4695251fdf58b3edb5302d3ba46","sha256:fbfcc5b7e68a2d8fd97688c69808049d582113368f388929d7d46ac44a592cd9","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 7m 0.91s | 6m 53.16s | 7m 8.65s | 6m 45.42s | 7m 16.4s |
| Model | 2 | 0 | 6m 14.33s | 6m 8.22s | 6m 20.44s | 6m 2.11s | 6m 26.55s |
| Non-model | 2 | 0 | 45.96s | 44.33s | 47.58s | 42.7s | 49.21s |
| Startup | 2 | 0 | 622.1111 ms | 617.5353 ms | 626.6869 ms | 612.9595 ms | 631.2627 ms |
| First byte | 2 | 0 | 1.33s | 1.33s | 1.33s | 1.32s | 1.34s |
| API turns | 2 | 0 | 62 | 61 | 63 | 60 | 64 |
| Input tokens | 2 | 0 | 6989766 | 6772180 | 7207351 | 6554594 | 7424937 |
| Output tokens | 2 | 0 | 74137.5 | 73467.75 | 74807.25 | 72798 | 75477 |
| Cached input (%) | 2 | 0 | 98.54147 | 98.54038 | 98.54256 | 98.53929 | 98.54365 |
| Static cost (USD) | 2 | 0 | 0.08043955 | 0.07971093 | 0.08116817 | 0.07898231 | 0.08189679 |
| Model request count | 2 | 0 | 62 | 61 | 63 | 60 | 64 |
| Within-run median request duration | 2 | 0 | 3.77s | 3.7s | 3.85s | 3.62s | 3.93s |
| Within-run median request wait | 2 | 0 | 1.33s | 1.33s | 1.33s | 1.32s | 1.34s |
| Within-run median request transfer | 2 | 0 | 2.41s | 2.33s | 2.49s | 2.25s | 2.57s |
| Within-run median inter-request gap | 2 | 0 | 57.49875 ms | 57.43433 ms | 57.56318 ms | 57.36991 ms | 57.6276 ms |
| Largest request duration | 2 | 0 | 28.36s | 25.75s | 30.98s | 23.14s | 33.59s |
| Largest inter-request gap | 2 | 0 | 11.08s | 10.2s | 11.96s | 9.32s | 12.84s |
| Time after last model response | 2 | 0 | 5.14s | 5.14s | 5.15s | 5.14s | 5.15s |
Task identity: ["superjson-error-stack-serialization","f143004734ea2dd4090e45013a07f091f5bbaee7","sha256:dcc10590630fcc927468099eb5118e4bc7170a1f12e01ca5c9404011de072444","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 6m 55.13s | 6m 21.98s | 7m 28.28s | 5m 48.84s | 8m 1.43s |
| Model | 2 | 0 | 6m 38.39s | 6m 5.42s | 7m 11.37s | 5m 32.44s | 7m 44.34s |
| Non-model | 2 | 0 | 14.98s | 14.81s | 15.15s | 14.64s | 15.32s |
| Startup | 2 | 0 | 1.76s | 1.76s | 1.76s | 1.76s | 1.76s |
| First byte | 2 | 0 | 1.23s | 1.2s | 1.26s | 1.17s | 1.29s |
| API turns | 2 | 0 | 50 | 49.5 | 50.5 | 49 | 51 |
| Input tokens | 2 | 0 | 3682024 | 3658567 | 3705481 | 3635110 | 3728938 |
| Output tokens | 2 | 0 | 70225.5 | 68146.25 | 72304.75 | 66067 | 74384 |
| Cached input (%) | 2 | 0 | 98.51793 | 98.51546 | 98.52039 | 98.513 | 98.52285 |
| Static cost (USD) | 2 | 0 | 0.06120287 | 0.06006347 | 0.06234227 | 0.05892407 | 0.06348167 |
| Model request count | 2 | 0 | 50 | 49.5 | 50.5 | 49 | 51 |
| Within-run median request duration | 2 | 0 | 3.47s | 3.41s | 3.54s | 3.34s | 3.6s |
| Within-run median request wait | 2 | 0 | 1.23s | 1.2s | 1.26s | 1.17s | 1.29s |
| Within-run median request transfer | 2 | 0 | 2.22s | 2.21s | 2.23s | 2.19s | 2.25s |
| Within-run median inter-request gap | 2 | 0 | 54.60662 ms | 53.86214 ms | 55.35111 ms | 53.11765 ms | 56.09559 ms |
| Largest request duration | 2 | 0 | 1m 3.14s | 55.7s | 1m 10.58s | 48.26s | 1m 18.02s |
| Largest inter-request gap | 2 | 0 | 1.31s | 1.22s | 1.4s | 1.13s | 1.49s |
| Time after last model response | 2 | 0 | 5.15s | 5.15s | 5.16s | 5.15s | 5.16s |
Task identity: ["textual-richlog-follow-state","a12215b791587a6d94227bd30a0e9291de932106","sha256:73fd3d800de9d5dae47edd9b5269e40966664b72d49d8f598964b8b91fdd0b92","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 12m 51.73s | 11m 28.56s | 14m 14.89s | 10m 5.4s | 15m 38.06s |
| Model | 2 | 0 | 8m 27.1s | 7m 43.37s | 9m 10.82s | 6m 59.65s | 9m 54.55s |
| Non-model | 2 | 0 | 4m 23.98s | 3m 44.54s | 5m 3.43s | 3m 5.09s | 5m 42.88s |
| Startup | 2 | 0 | 645.8202 ms | 636.8684 ms | 654.772 ms | 627.9166 ms | 663.7238 ms |
| First byte | 2 | 0 | 1.37s | 1.37s | 1.37s | 1.37s | 1.37s |
| API turns | 2 | 0 | 83.5 | 77.75 | 89.25 | 72 | 95 |
| Input tokens | 2 | 0 | 9764633 | 8889508 | 10639760 | 8014382 | 11514880 |
| Output tokens | 2 | 0 | 86041.5 | 81420.75 | 90662.25 | 76800 | 95283 |
| Cached input (%) | 2 | 0 | 98.93105 | 98.86791 | 98.99419 | 98.80477 | 99.05732 |
| Static cost (USD) | 2 | 0 | 0.09593764 | 0.09007096 | 0.1018043 | 0.08420428 | 0.107671 |
| Model request count | 2 | 0 | 83.5 | 77.75 | 89.25 | 72 | 95 |
| Within-run median request duration | 2 | 0 | 3.88s | 3.83s | 3.93s | 3.77s | 3.98s |
| Within-run median request wait | 2 | 0 | 1.37s | 1.37s | 1.37s | 1.37s | 1.37s |
| Within-run median request transfer | 2 | 0 | 2.52s | 2.45s | 2.59s | 2.38s | 2.66s |
| Within-run median inter-request gap | 2 | 0 | 58.85193 ms | 58.42963 ms | 59.27423 ms | 58.00733 ms | 59.69653 ms |
| Largest request duration | 2 | 0 | 43.88s | 41.85s | 45.9s | 39.83s | 47.92s |
| Largest inter-request gap | 2 | 0 | 30.18s | 30.18s | 30.18s | 30.18s | 30.18s |
| Time after last model response | 2 | 0 | 5.13s | 5.13s | 5.14s | 5.13s | 5.14s |
Task identity: ["tomlkit-toml-table-converters","962de91aaf103135f400e5108a798a59a10dc17e","sha256:68dd891458463c9f930dcbbd59b4442065c145d66ce3b5813af82ca5ec8f4966","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 10m 12.63s | 9m 53.92s | 10m 31.35s | 9m 35.2s | 10m 50.06s |
| Model | 2 | 0 | 9m 53.43s | 9m 34.62s | 10m 12.23s | 9m 15.82s | 10m 31.04s |
| Non-model | 2 | 0 | 18.15s | 18.06s | 18.24s | 17.96s | 18.34s |
| Startup | 2 | 0 | 1.05s | 1.05s | 1.06s | 1.05s | 1.06s |
| First byte | 2 | 0 | 1.36s | 1.34s | 1.37s | 1.32s | 1.39s |
| API turns | 2 | 0 | 68 | 63.5 | 72.5 | 59 | 77 |
| Input tokens | 2 | 0 | 7875966 | 7131599 | 8620333 | 6387232 | 9364700 |
| Output tokens | 2 | 0 | 96997.5 | 94439.75 | 99555.25 | 91882 | 102113 |
| Cached input (%) | 2 | 0 | 99.0028 | 98.95506 | 99.05054 | 98.90732 | 99.09827 |
| Static cost (USD) | 2 | 0 | 0.09316274 | 0.08885653 | 0.09746896 | 0.08455032 | 0.1017752 |
| Model request count | 2 | 0 | 68 | 63.5 | 72.5 | 59 | 77 |
| Within-run median request duration | 2 | 0 | 4.91s | 4.79s | 5.03s | 4.66s | 5.16s |
| Within-run median request wait | 2 | 0 | 1.36s | 1.34s | 1.37s | 1.32s | 1.39s |
| Within-run median request transfer | 2 | 0 | 3.32s | 3.18s | 3.46s | 3.04s | 3.6s |
| Within-run median inter-request gap | 2 | 0 | 81.43082 ms | 81.32774 ms | 81.53389 ms | 81.22467 ms | 81.63697 ms |
| Largest request duration | 2 | 0 | 1m 5.57s | 59.91s | 1m 11.24s | 54.24s | 1m 16.91s |
| Largest inter-request gap | 2 | 0 | 1.9s | 1.63s | 2.18s | 1.35s | 2.46s |
| Time after last model response | 2 | 0 | 5.16s | 5.15s | 5.16s | 5.15s | 5.16s |
Task identity: ["true-myth-iterable-collection-combinators","5945ad1b96e228ff716275d8a273282101152508","sha256:0ecebbb45e561b37ee5d7bcbe80588427f4890fb80fffbbb73ca6b94c7cb8855","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 6m 5.5s | 5m 8.11s | 7m 2.89s | 4m 10.72s | 8m 0.27s |
| Model | 2 | 0 | 4m 54.9s | 4m 18.39s | 5m 31.4s | 3m 41.88s | 6m 7.91s |
| Non-model | 2 | 0 | 1m 9.94s | 49.05s | 1m 30.83s | 28.17s | 1m 51.72s |
| Startup | 2 | 0 | 661.7406 ms | 654.4772 ms | 669.0039 ms | 647.2139 ms | 676.2672 ms |
| First byte | 2 | 0 | 1.25s | 1.21s | 1.3s | 1.17s | 1.34s |
| API turns | 2 | 0 | 49 | 39.5 | 58.5 | 30 | 68 |
| Input tokens | 2 | 0 | 4758895 | 3585219 | 5932572 | 2411542 | 7106248 |
| Output tokens | 2 | 0 | 57334.5 | 50517.75 | 64151.25 | 43701 | 70968 |
| Cached input (%) | 2 | 0 | 97.70341 | 97.2536 | 98.15322 | 96.80379 | 98.60303 |
| Static cost (USD) | 2 | 0 | 0.06163911 | 0.0532124 | 0.07006582 | 0.04478569 | 0.07849253 |
| Model request count | 2 | 0 | 49 | 39.5 | 58.5 | 30 | 68 |
| Within-run median request duration | 2 | 0 | 4.19s | 3.85s | 4.52s | 3.51s | 4.86s |
| Within-run median request wait | 2 | 0 | 1.25s | 1.21s | 1.3s | 1.17s | 1.34s |
| Within-run median request transfer | 2 | 0 | 2.99s | 2.59s | 3.4s | 2.18s | 3.8s |
| Within-run median inter-request gap | 2 | 0 | 55.99988 ms | 55.24104 ms | 56.75872 ms | 54.4822 ms | 57.51756 ms |
| Largest request duration | 2 | 0 | 25.04s | 22.41s | 27.67s | 19.78s | 30.3s |
| Largest inter-request gap | 2 | 0 | 17.63s | 11.43s | 23.83s | 5.23s | 30.03s |
| Time after last model response | 2 | 0 | 5.14s | 5.13s | 5.14s | 5.13s | 5.15s |
Task identity: ["cattrs-partial-structuring-recovery","91df27b2a20f26c35777a92f4555438b62c06345","sha256:b985f2bd17e9e80bee0fa9b7c0747b2ded92d54d070b32ebfe102e8829a44076","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 13m 25.58s | unavailable | unavailable | 13m 25.58s | 13m 25.58s |
| Model | 1 | 0 | 8m 59.91s | unavailable | unavailable | 8m 59.91s | 8m 59.91s |
| Non-model | 1 | 0 | 4m 21.51s | unavailable | unavailable | 4m 21.51s | 4m 21.51s |
| Startup | 1 | 0 | 4.16s | unavailable | unavailable | 4.16s | 4.16s |
| First byte | 1 | 0 | 1.27s | unavailable | unavailable | 1.27s | 1.27s |
| API turns | 1 | 0 | 110 | unavailable | unavailable | 110 | 110 |
| Input tokens | 1 | 0 | 12504690 | unavailable | unavailable | 12504690 | 12504690 |
| Output tokens | 1 | 0 | 90494 | unavailable | unavailable | 90494 | 90494 |
| Cached input (%) | 1 | 0 | 99.20374 | unavailable | unavailable | 99.20374 | 99.20374 |
| Static cost (USD) | 1 | 0 | 0.1064473 | unavailable | unavailable | 0.1064473 | 0.1064473 |
| Model request count | 1 | 0 | 110 | unavailable | unavailable | 110 | 110 |
| Within-run median request duration | 1 | 0 | 2.97s | unavailable | unavailable | 2.97s | 2.97s |
| Within-run median request wait | 1 | 0 | 1.27s | unavailable | unavailable | 1.27s | 1.27s |
| Within-run median request transfer | 1 | 0 | 1.65s | unavailable | unavailable | 1.65s | 1.65s |
| Within-run median inter-request gap | 1 | 0 | 166.4325 ms | unavailable | unavailable | 166.4325 ms | 166.4325 ms |
| Largest request duration | 1 | 0 | 38.13s | unavailable | unavailable | 38.13s | 38.13s |
| Largest inter-request gap | 1 | 0 | 59.27s | unavailable | unavailable | 59.27s | 59.27s |
| Time after last model response | 1 | 0 | 261.4834 ms | unavailable | unavailable | 261.4834 ms | 261.4834 ms |
Task identity: ["cattrs-partial-structuring-recovery","91df27b2a20f26c35777a92f4555438b62c06345","sha256:b985f2bd17e9e80bee0fa9b7c0747b2ded92d54d070b32ebfe102e8829a44076","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 9m 21.91s | unavailable | unavailable | 9m 21.91s | 9m 21.91s |
| Model | 1 | 0 | 6m 36.5s | unavailable | unavailable | 6m 36.5s | 6m 36.5s |
| Non-model | 1 | 0 | 2m 41.23s | unavailable | unavailable | 2m 41.23s | 2m 41.23s |
| Startup | 1 | 0 | 4.18s | unavailable | unavailable | 4.18s | 4.18s |
| First byte | 1 | 0 | 1.31s | unavailable | unavailable | 1.31s | 1.31s |
| API turns | 1 | 0 | 65 | unavailable | unavailable | 65 | 65 |
| Input tokens | 1 | 0 | 6159176 | unavailable | unavailable | 6159176 | 6159176 |
| Output tokens | 1 | 0 | 66537 | unavailable | unavailable | 66537 | 66537 |
| Cached input (%) | 1 | 0 | 98.64384 | unavailable | unavailable | 98.64384 | 98.64384 |
| Static cost (USD) | 1 | 0 | 0.07067834 | unavailable | unavailable | 0.07067834 | 0.07067834 |
| Model request count | 1 | 0 | 65 | unavailable | unavailable | 65 | 65 |
| Within-run median request duration | 1 | 0 | 3.63s | unavailable | unavailable | 3.63s | 3.63s |
| Within-run median request wait | 1 | 0 | 1.31s | unavailable | unavailable | 1.31s | 1.31s |
| Within-run median request transfer | 1 | 0 | 2.42s | unavailable | unavailable | 2.42s | 2.42s |
| Within-run median inter-request gap | 1 | 0 | 151.5668 ms | unavailable | unavailable | 151.5668 ms | 151.5668 ms |
| Largest request duration | 1 | 0 | 1m 0.72s | unavailable | unavailable | 1m 0.72s | 1m 0.72s |
| Largest inter-request gap | 1 | 0 | 54.06s | unavailable | unavailable | 54.06s | 54.06s |
| Time after last model response | 1 | 0 | 507.6009 ms | unavailable | unavailable | 507.6009 ms | 507.6009 ms |
Task identity: ["happy-dom-deterministic-intersectionobserver","32b2a8e60eab482c023af377160bf294d6da0318","sha256:a4df294272f2165ce894306ab1eec5284fe426ddaa66cf08bb3c4755a9f8faa2","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 9m 25.48s | unavailable | unavailable | 9m 25.48s | 9m 25.48s |
| Model | 1 | 0 | 7m 44.61s | unavailable | unavailable | 7m 44.61s | 7m 44.61s |
| Non-model | 1 | 0 | 1m 36.22s | unavailable | unavailable | 1m 36.22s | 1m 36.22s |
| Startup | 1 | 0 | 4.65s | unavailable | unavailable | 4.65s | 4.65s |
| First byte | 1 | 0 | 1.23s | unavailable | unavailable | 1.23s | 1.23s |
| API turns | 1 | 0 | 96 | unavailable | unavailable | 96 | 96 |
| Input tokens | 1 | 0 | 8434472 | unavailable | unavailable | 8434472 | 8434472 |
| Output tokens | 1 | 0 | 77756 | unavailable | unavailable | 77756 | 77756 |
| Cached input (%) | 1 | 0 | 98.94481 | unavailable | unavailable | 98.94481 | 98.94481 |
| Static cost (USD) | 1 | 0 | 0.08504002 | unavailable | unavailable | 0.08504002 | 0.08504002 |
| Model request count | 1 | 0 | 96 | unavailable | unavailable | 96 | 96 |
| Within-run median request duration | 1 | 0 | 2.48s | unavailable | unavailable | 2.48s | 2.48s |
| Within-run median request wait | 1 | 0 | 1.23s | unavailable | unavailable | 1.23s | 1.23s |
| Within-run median request transfer | 1 | 0 | 1.14s | unavailable | unavailable | 1.14s | 1.14s |
| Within-run median inter-request gap | 1 | 0 | 154.9124 ms | unavailable | unavailable | 154.9124 ms | 154.9124 ms |
| Largest request duration | 1 | 0 | 28.25s | unavailable | unavailable | 28.25s | 28.25s |
| Largest inter-request gap | 1 | 0 | 39.48s | unavailable | unavailable | 39.48s | 39.48s |
| Time after last model response | 1 | 0 | 295.7324 ms | unavailable | unavailable | 295.7324 ms | 295.7324 ms |
Task identity: ["happy-dom-deterministic-intersectionobserver","32b2a8e60eab482c023af377160bf294d6da0318","sha256:a4df294272f2165ce894306ab1eec5284fe426ddaa66cf08bb3c4755a9f8faa2","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 12m 12.12s | unavailable | unavailable | 12m 12.12s | 12m 12.12s |
| Model | 1 | 0 | 7m 19.13s | unavailable | unavailable | 7m 19.13s | 7m 19.13s |
| Non-model | 1 | 0 | 4m 45.43s | unavailable | unavailable | 4m 45.43s | 4m 45.43s |
| Startup | 1 | 0 | 7.56s | unavailable | unavailable | 7.56s | 7.56s |
| First byte | 1 | 0 | 1.21s | unavailable | unavailable | 1.21s | 1.21s |
| API turns | 1 | 0 | 70 | unavailable | unavailable | 70 | 70 |
| Input tokens | 1 | 0 | 5166491 | unavailable | unavailable | 5166491 | 5166491 |
| Output tokens | 1 | 0 | 72141 | unavailable | unavailable | 72141 | 72141 |
| Cached input (%) | 1 | 0 | 98.6864 | unavailable | unavailable | 98.6864 | 98.6864 |
| Static cost (USD) | 1 | 0 | 0.06876052 | unavailable | unavailable | 0.06876052 | 0.06876052 |
| Model request count | 1 | 0 | 70 | unavailable | unavailable | 70 | 70 |
| Within-run median request duration | 1 | 0 | 2.4s | unavailable | unavailable | 2.4s | 2.4s |
| Within-run median request wait | 1 | 0 | 1.21s | unavailable | unavailable | 1.21s | 1.21s |
| Within-run median request transfer | 1 | 0 | 1.23s | unavailable | unavailable | 1.23s | 1.23s |
| Within-run median inter-request gap | 1 | 0 | 123.4377 ms | unavailable | unavailable | 123.4377 ms | 123.4377 ms |
| Largest request duration | 1 | 0 | 26.57s | unavailable | unavailable | 26.57s | 26.57s |
| Largest inter-request gap | 1 | 0 | 3m 0.25s | unavailable | unavailable | 3m 0.25s | 3m 0.25s |
| Time after last model response | 1 | 0 | 283.3951 ms | unavailable | unavailable | 283.3951 ms | 283.3951 ms |
Task identity: ["ink-grid-box-layout","477fda7df7781406d2ba046eb48dd5391e964b72","sha256:73daa7eb11dec0d43e523434d28a66ee9d3ac063fda33d9e26183d867739dbb1","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 16m 44.31s | unavailable | unavailable | 16m 44.31s | 16m 44.31s |
| Model | 1 | 0 | 8m 59.34s | unavailable | unavailable | 8m 59.34s | 8m 59.34s |
| Non-model | 1 | 0 | 7m 36.3s | unavailable | unavailable | 7m 36.3s | 7m 36.3s |
| Startup | 1 | 0 | 8.68s | unavailable | unavailable | 8.68s | 8.68s |
| First byte | 1 | 0 | 1.33s | unavailable | unavailable | 1.33s | 1.33s |
| API turns | 1 | 0 | 106 | unavailable | unavailable | 106 | 106 |
| Input tokens | 1 | 0 | 12632710 | unavailable | unavailable | 12632710 | 12632710 |
| Output tokens | 1 | 0 | 87944 | unavailable | unavailable | 87944 | 87944 |
| Cached input (%) | 1 | 0 | 99.2086 | unavailable | unavailable | 99.2086 | 99.2086 |
| Static cost (USD) | 1 | 0 | 0.1053609 | unavailable | unavailable | 0.1053609 | 0.1053609 |
| Model request count | 1 | 0 | 106 | unavailable | unavailable | 106 | 106 |
| Within-run median request duration | 1 | 0 | 2.44s | unavailable | unavailable | 2.44s | 2.44s |
| Within-run median request wait | 1 | 0 | 1.33s | unavailable | unavailable | 1.33s | 1.33s |
| Within-run median request transfer | 1 | 0 | 1.07s | unavailable | unavailable | 1.07s | 1.07s |
| Within-run median inter-request gap | 1 | 0 | 167.45 ms | unavailable | unavailable | 167.45 ms | 167.45 ms |
| Largest request duration | 1 | 0 | 59.45s | unavailable | unavailable | 59.45s | 59.45s |
| Largest inter-request gap | 1 | 0 | 1m 34.59s | unavailable | unavailable | 1m 34.59s | 1m 34.59s |
| Time after last model response | 1 | 0 | 240.8726 ms | unavailable | unavailable | 240.8726 ms | 240.8726 ms |
Task identity: ["ink-grid-box-layout","477fda7df7781406d2ba046eb48dd5391e964b72","sha256:73daa7eb11dec0d43e523434d28a66ee9d3ac063fda33d9e26183d867739dbb1","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 5m 14.88s | unavailable | unavailable | 5m 14.88s | 5m 14.88s |
| Model | 1 | 0 | 4m 29.3s | unavailable | unavailable | 4m 29.3s | 4m 29.3s |
| Non-model | 1 | 0 | 41.17s | unavailable | unavailable | 41.17s | 41.17s |
| Startup | 1 | 0 | 4.41s | unavailable | unavailable | 4.41s | 4.41s |
| First byte | 1 | 0 | 1.2s | unavailable | unavailable | 1.2s | 1.2s |
| API turns | 1 | 0 | 33 | unavailable | unavailable | 33 | 33 |
| Input tokens | 1 | 0 | 2305332 | unavailable | unavailable | 2305332 | 2305332 |
| Output tokens | 1 | 0 | 43961 | unavailable | unavailable | 43961 | 43961 |
| Cached input (%) | 1 | 0 | 96.97727 | unavailable | unavailable | 96.97727 | 96.97727 |
| Static cost (USD) | 1 | 0 | 0.04353614 | unavailable | unavailable | 0.04353614 | 0.04353614 |
| Model request count | 1 | 0 | 33 | unavailable | unavailable | 33 | 33 |
| Within-run median request duration | 1 | 0 | 3.43s | unavailable | unavailable | 3.43s | 3.43s |
| Within-run median request wait | 1 | 0 | 1.2s | unavailable | unavailable | 1.2s | 1.2s |
| Within-run median request transfer | 1 | 0 | 2.33s | unavailable | unavailable | 2.33s | 2.33s |
| Within-run median inter-request gap | 1 | 0 | 132.1602 ms | unavailable | unavailable | 132.1602 ms | 132.1602 ms |
| Largest request duration | 1 | 0 | 43.11s | unavailable | unavailable | 43.11s | 43.11s |
| Largest inter-request gap | 1 | 0 | 13.43s | unavailable | unavailable | 13.43s | 13.43s |
| Time after last model response | 1 | 0 | 3.6s | unavailable | unavailable | 3.6s | 3.6s |
Task identity: ["psd-tools-blend-range-api","01595992c5f4d4695251fdf58b3edb5302d3ba46","sha256:fbfcc5b7e68a2d8fd97688c69808049d582113368f388929d7d46ac44a592cd9","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 8m 7.38s | 7m 47.51s | 8m 27.26s | 7m 27.64s | 8m 47.13s |
| Model | 2 | 0 | 6m 36.52s | 6m 23.77s | 6m 49.26s | 6m 11.02s | 7m 2.01s |
| Non-model | 2 | 0 | 1m 26.65s | 1m 19.54s | 1m 33.76s | 1m 12.42s | 1m 40.88s |
| Startup | 2 | 0 | 4.22s | 4.2s | 4.23s | 4.19s | 4.24s |
| First byte | 2 | 0 | 1.24s | 1.22s | 1.26s | 1.2s | 1.27s |
| API turns | 2 | 0 | 101.5 | 101.25 | 101.75 | 101 | 102 |
| Input tokens | 2 | 0 | 8957983 | 8580415 | 9335551 | 8202847 | 9713119 |
| Output tokens | 2 | 0 | 63431 | 61986 | 64876 | 60541 | 66321 |
| Cached input (%) | 2 | 0 | 98.96271 | 98.95257 | 98.97285 | 98.94243 | 98.98299 |
| Static cost (USD) | 2 | 0 | 0.0785693 | 0.07612742 | 0.08101118 | 0.07368554 | 0.08345306 |
| Model request count | 2 | 0 | 101.5 | 101.25 | 101.75 | 101 | 102 |
| Within-run median request duration | 2 | 0 | 2.38s | 2.33s | 2.43s | 2.28s | 2.48s |
| Within-run median request wait | 2 | 0 | 1.24s | 1.22s | 1.26s | 1.2s | 1.27s |
| Within-run median request transfer | 2 | 0 | 1.12s | 1.07s | 1.18s | 1.02s | 1.23s |
| Within-run median inter-request gap | 2 | 0 | 165.3606 ms | 165.2384 ms | 165.4828 ms | 165.1162 ms | 165.605 ms |
| Largest request duration | 2 | 0 | 23.66s | 21.95s | 25.37s | 20.24s | 27.08s |
| Largest inter-request gap | 2 | 0 | 12.75s | 12.6s | 12.91s | 12.45s | 13.06s |
| Time after last model response | 2 | 0 | 249.2194 ms | 248.617 ms | 249.8217 ms | 248.0146 ms | 250.4241 ms |
Task identity: ["superjson-error-stack-serialization","f143004734ea2dd4090e45013a07f091f5bbaee7","sha256:dcc10590630fcc927468099eb5118e4bc7170a1f12e01ca5c9404011de072444","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 5m 5.63s | 4m 44.23s | 5m 27.02s | 4m 22.84s | 5m 48.42s |
| Model | 2 | 0 | 4m 35.38s | 4m 13.39s | 4m 57.37s | 3m 51.4s | 5m 19.36s |
| Non-model | 2 | 0 | 25.47s | 24.92s | 26.02s | 24.37s | 26.56s |
| Startup | 2 | 0 | 4.78s | 4.73s | 4.82s | 4.68s | 4.87s |
| First byte | 2 | 0 | 1.22s | 1.21s | 1.24s | 1.2s | 1.25s |
| API turns | 2 | 0 | 40.5 | 40.25 | 40.75 | 40 | 41 |
| Input tokens | 2 | 0 | 2592814 | 2542359 | 2643269 | 2491904 | 2693724 |
| Output tokens | 2 | 0 | 49944 | 47736 | 52152 | 45528 | 54360 |
| Cached input (%) | 2 | 0 | 98.26595 | 98.25932 | 98.27259 | 98.25268 | 98.27923 |
| Static cost (USD) | 2 | 0 | 0.04435602 | 0.04272595 | 0.04598609 | 0.04109587 | 0.04761617 |
| Model request count | 2 | 0 | 40.5 | 40.25 | 40.75 | 40 | 41 |
| Within-run median request duration | 2 | 0 | 2.71s | 2.63s | 2.8s | 2.54s | 2.88s |
| Within-run median request wait | 2 | 0 | 1.22s | 1.21s | 1.24s | 1.2s | 1.25s |
| Within-run median request transfer | 2 | 0 | 1.47s | 1.41s | 1.54s | 1.34s | 1.61s |
| Within-run median inter-request gap | 2 | 0 | 164.1467 ms | 163.4877 ms | 164.8058 ms | 162.8287 ms | 165.4648 ms |
| Largest request duration | 2 | 0 | 1m 36.29s | 1m 32.5s | 1m 40.08s | 1m 28.7s | 1m 43.88s |
| Largest inter-request gap | 2 | 0 | 8.24s | 8.24s | 8.24s | 8.23s | 8.25s |
| Time after last model response | 2 | 0 | 224.4794 ms | 219.8981 ms | 229.0608 ms | 215.3168 ms | 233.6421 ms |
Task identity: ["textual-richlog-follow-state","a12215b791587a6d94227bd30a0e9291de932106","sha256:73fd3d800de9d5dae47edd9b5269e40966664b72d49d8f598964b8b91fdd0b92","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 18m 25.74s | 16m 58.98s | 19m 52.5s | 15m 32.22s | 21m 19.26s |
| Model | 2 | 0 | 13m 34.39s | 12m 40.96s | 14m 27.82s | 11m 47.53s | 15m 21.24s |
| Non-model | 2 | 0 | 4m 46.56s | 4m 12.93s | 5m 20.19s | 3m 39.3s | 5m 53.81s |
| Startup | 2 | 0 | 4.79s | 4.5s | 5.09s | 4.2s | 5.39s |
| First byte | 2 | 0 | 1.32s | 1.32s | 1.32s | 1.32s | 1.32s |
| API turns | 2 | 0 | 186 | 184 | 188 | 182 | 190 |
| Input tokens | 2 | 0 | 23509950 | 22427760 | 24592140 | 21345570 | 25674330 |
| Output tokens | 2 | 0 | 115906.5 | 109612.3 | 122200.8 | 103318 | 128495 |
| Cached input (%) | 2 | 0 | 99.49702 | 99.49298 | 99.50106 | 99.48894 | 99.5051 |
| Static cost (USD) | 2 | 0 | 0.1574307 | 0.1497471 | 0.1651143 | 0.1420634 | 0.1727979 |
| Model request count | 2 | 0 | 186 | 184 | 188 | 182 | 190 |
| Within-run median request duration | 2 | 0 | 2.97s | 2.81s | 3.14s | 2.65s | 3.3s |
| Within-run median request wait | 2 | 0 | 1.32s | 1.32s | 1.32s | 1.32s | 1.32s |
| Within-run median request transfer | 2 | 0 | 1.56s | 1.44s | 1.69s | 1.31s | 1.81s |
| Within-run median inter-request gap | 2 | 0 | 184.2483 ms | 182.2135 ms | 186.2831 ms | 180.1787 ms | 188.3179 ms |
| Largest request duration | 2 | 0 | 40.72s | 34.4s | 47.04s | 28.07s | 53.37s |
| Largest inter-request gap | 2 | 0 | 2m 13.43s | 2m 6.77s | 2m 20.1s | 2m 0.1s | 2m 26.77s |
| Time after last model response | 2 | 0 | 263.1321 ms | 258.4377 ms | 267.8266 ms | 253.7433 ms | 272.521 ms |
Task identity: ["tomlkit-toml-table-converters","962de91aaf103135f400e5108a798a59a10dc17e","sha256:68dd891458463c9f930dcbbd59b4442065c145d66ce3b5813af82ca5ec8f4966","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 18m 59.94s | 16m 21.23s | 21m 38.65s | 13m 42.51s | 24m 17.37s |
| Model | 2 | 0 | 16m 40.49s | 14m 47.07s | 18m 33.91s | 12m 53.65s | 20m 27.32s |
| Non-model | 2 | 0 | 2m 15.2s | 1m 29.91s | 3m 0.49s | 44.62s | 3m 45.78s |
| Startup | 2 | 0 | 4.25s | 4.25s | 4.26s | 4.25s | 4.26s |
| First byte | 2 | 0 | 1.41s | 1.4s | 1.42s | 1.39s | 1.43s |
| API turns | 2 | 0 | 155.5 | 135.75 | 175.25 | 116 | 195 |
| Input tokens | 2 | 0 | 30328770 | 23435320 | 37222230 | 16541860 | 44115680 |
| Output tokens | 2 | 0 | 153487.5 | 134147.3 | 172827.8 | 114807 | 192168 |
| Cached input (%) | 2 | 0 | 99.44213 | 99.38163 | 99.50262 | 99.32114 | 99.56311 |
| Static cost (USD) | 2 | 0 | 0.2054986 | 0.1702579 | 0.2407392 | 0.1350173 | 0.2759799 |
| Model request count | 2 | 0 | 155.5 | 135.75 | 175.25 | 116 | 195 |
| Within-run median request duration | 2 | 0 | 3.56s | 3.53s | 3.6s | 3.49s | 3.64s |
| Within-run median request wait | 2 | 0 | 1.41s | 1.4s | 1.42s | 1.39s | 1.43s |
| Within-run median request transfer | 2 | 0 | 2.11s | 2.05s | 2.17s | 1.98s | 2.23s |
| Within-run median inter-request gap | 2 | 0 | 186.6204 ms | 178.473 ms | 194.7679 ms | 170.3255 ms | 202.9154 ms |
| Largest request duration | 2 | 0 | 1m 40.05s | 1m 15.24s | 2m 4.87s | 50.43s | 2m 29.68s |
| Largest inter-request gap | 2 | 0 | 24.46s | 16.33s | 32.58s | 8.2s | 40.71s |
| Time after last model response | 2 | 0 | 231.3787 ms | 224.9716 ms | 237.7859 ms | 218.5644 ms | 244.1931 ms |
Task identity: ["true-myth-iterable-collection-combinators","5945ad1b96e228ff716275d8a273282101152508","sha256:0ecebbb45e561b37ee5d7bcbe80588427f4890fb80fffbbb73ca6b94c7cb8855","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 8m 30.74s | 8m 18.28s | 8m 43.19s | 8m 5.83s | 8m 55.64s |
| Model | 2 | 0 | 7m 5.64s | 6m 56.83s | 7m 14.45s | 6m 48.02s | 7m 23.26s |
| Non-model | 2 | 0 | 1m 20.89s | 1m 17.33s | 1m 24.46s | 1m 13.76s | 1m 28.03s |
| Startup | 2 | 0 | 4.2s | 4.12s | 4.28s | 4.05s | 4.36s |
| First byte | 2 | 0 | 1.36s | 1.35s | 1.38s | 1.34s | 1.39s |
| API turns | 2 | 0 | 76 | 73.5 | 78.5 | 71 | 81 |
| Input tokens | 2 | 0 | 10573610 | 9927344 | 11219880 | 9281078 | 11866140 |
| Output tokens | 2 | 0 | 75101.5 | 72088.25 | 78114.75 | 69075 | 81128 |
| Cached input (%) | 2 | 0 | 98.70898 | 98.686 | 98.73196 | 98.66302 | 98.75494 |
| Static cost (USD) | 2 | 0 | 0.09676101 | 0.09214494 | 0.1013771 | 0.08752888 | 0.1059932 |
| Model request count | 2 | 0 | 76 | 73.5 | 78.5 | 71 | 81 |
| Within-run median request duration | 2 | 0 | 3.33s | 3.05s | 3.6s | 2.77s | 3.88s |
| Within-run median request wait | 2 | 0 | 1.36s | 1.35s | 1.38s | 1.34s | 1.39s |
| Within-run median request transfer | 2 | 0 | 1.92s | 1.63s | 2.2s | 1.34s | 2.49s |
| Within-run median inter-request gap | 2 | 0 | 174.7069 ms | 168.5113 ms | 180.9024 ms | 162.3157 ms | 187.098 ms |
| Largest request duration | 2 | 0 | 37.86s | 35.17s | 40.55s | 32.48s | 43.24s |
| Largest inter-request gap | 2 | 0 | 8.27s | 8.26s | 8.28s | 8.25s | 8.3s |
| Time after last model response | 2 | 0 | 236.6965 ms | 235.2791 ms | 238.1138 ms | 233.8618 ms | 239.5312 ms |
Task identity: ["cattrs-partial-structuring-recovery","91df27b2a20f26c35777a92f4555438b62c06345","sha256:b985f2bd17e9e80bee0fa9b7c0747b2ded92d54d070b32ebfe102e8829a44076","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 6m 47.48s | 6m 20.77s | 7m 14.2s | 5m 54.05s | 7m 40.91s |
| Model | 2 | 0 | 4m 18.7s | 3m 59s | 4m 38.4s | 3m 39.3s | 4m 58.1s |
| Non-model | 2 | 0 | 2m 28.01s | 2m 21s | 2m 35.02s | 2m 13.99s | 2m 42.04s |
| Startup | 2 | 0 | 768.7774 ms | 765.9402 ms | 771.6147 ms | 763.1029 ms | 774.452 ms |
| First byte | 2 | 0 | 1.18s | 1.17s | 1.18s | 1.17s | 1.19s |
| API turns | 2 | 0 | 48 | 42.5 | 53.5 | 37 | 59 |
| Input tokens | 2 | 0 | 2912162 | 2519956 | 3304369 | 2127749 | 3696575 |
| Output tokens | 2 | 0 | 46925 | 43698 | 50152 | 40471 | 53379 |
| Cached input (%) | 2 | 0 | 98.0813 | 97.83437 | 98.32822 | 97.58745 | 98.57514 |
| Static cost (USD) | 2 | 0 | 0.04453578 | 0.04137379 | 0.04769777 | 0.0382118 | 0.05085976 |
| Model request count | 2 | 0 | 48 | 42.5 | 53.5 | 37 | 59 |
| Within-run median request duration | 2 | 0 | 2.68s | 2.49s | 2.87s | 2.3s | 3.06s |
| Within-run median request wait | 2 | 0 | 1.18s | 1.17s | 1.18s | 1.17s | 1.19s |
| Within-run median request transfer | 2 | 0 | 1.48s | 1.27s | 1.69s | 1.06s | 1.9s |
| Within-run median inter-request gap | 2 | 0 | 34.22659 ms | 32.04207 ms | 36.41112 ms | 29.85755 ms | 38.59564 ms |
| Largest request duration | 2 | 0 | 34.38s | 30.54s | 38.22s | 26.7s | 42.07s |
| Largest inter-request gap | 2 | 0 | 1m 0.83s | 58.57s | 1m 3.08s | 56.31s | 1m 5.34s |
| Time after last model response | 2 | 0 | 199.6017 ms | 173.063 ms | 226.1405 ms | 146.5243 ms | 252.6792 ms |
Task identity: ["happy-dom-deterministic-intersectionobserver","32b2a8e60eab482c023af377160bf294d6da0318","sha256:a4df294272f2165ce894306ab1eec5284fe426ddaa66cf08bb3c4755a9f8faa2","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 6m 36.97s | unavailable | unavailable | 6m 36.97s | 6m 36.97s |
| Model | 1 | 0 | 5m 4.58s | unavailable | unavailable | 5m 4.58s | 5m 4.58s |
| Non-model | 1 | 0 | 1m 31.04s | unavailable | unavailable | 1m 31.04s | 1m 31.04s |
| Startup | 1 | 0 | 1.35s | unavailable | unavailable | 1.35s | 1.35s |
| First byte | 1 | 0 | 1.15s | unavailable | unavailable | 1.15s | 1.15s |
| API turns | 1 | 0 | 53 | unavailable | unavailable | 53 | 53 |
| Input tokens | 1 | 0 | 3138797 | unavailable | unavailable | 3138797 | 3138797 |
| Output tokens | 1 | 0 | 53522 | unavailable | unavailable | 53522 | 53522 |
| Cached input (%) | 1 | 0 | 97.99423 | unavailable | unavailable | 97.99423 | 97.99423 |
| Static cost (USD) | 1 | 0 | 0.05078427 | unavailable | unavailable | 0.05078427 | 0.05078427 |
| Model request count | 1 | 0 | 53 | unavailable | unavailable | 53 | 53 |
| Within-run median request duration | 1 | 0 | 2.43s | unavailable | unavailable | 2.43s | 2.43s |
| Within-run median request wait | 1 | 0 | 1.15s | unavailable | unavailable | 1.15s | 1.15s |
| Within-run median request transfer | 1 | 0 | 1.23s | unavailable | unavailable | 1.23s | 1.23s |
| Within-run median inter-request gap | 1 | 0 | 30.47215 ms | unavailable | unavailable | 30.47215 ms | 30.47215 ms |
| Largest request duration | 1 | 0 | 56.91s | unavailable | unavailable | 56.91s | 56.91s |
| Largest inter-request gap | 1 | 0 | 39.53s | unavailable | unavailable | 39.53s | 39.53s |
| Time after last model response | 1 | 0 | 231.3146 ms | unavailable | unavailable | 231.3146 ms | 231.3146 ms |
Task identity: ["happy-dom-deterministic-intersectionobserver","32b2a8e60eab482c023af377160bf294d6da0318","sha256:a4df294272f2165ce894306ab1eec5284fe426ddaa66cf08bb3c4755a9f8faa2","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 8m 37.93s | unavailable | unavailable | 8m 37.93s | 8m 37.93s |
| Model | 1 | 0 | 5m 54.35s | unavailable | unavailable | 5m 54.35s | 5m 54.35s |
| Non-model | 1 | 0 | 2m 42.25s | unavailable | unavailable | 2m 42.25s | 2m 42.25s |
| Startup | 1 | 0 | 1.33s | unavailable | unavailable | 1.33s | 1.33s |
| First byte | 1 | 0 | 1.23s | unavailable | unavailable | 1.23s | 1.23s |
| API turns | 1 | 0 | 62 | unavailable | unavailable | 62 | 62 |
| Input tokens | 1 | 0 | 5138624 | unavailable | unavailable | 5138624 | 5138624 |
| Output tokens | 1 | 0 | 63729 | unavailable | unavailable | 63729 | 63729 |
| Cached input (%) | 1 | 0 | 98.23517 | unavailable | unavailable | 98.23517 | 98.23517 |
| Static cost (USD) | 1 | 0 | 0.06698441 | unavailable | unavailable | 0.06698441 | 0.06698441 |
| Model request count | 1 | 0 | 62 | unavailable | unavailable | 62 | 62 |
| Within-run median request duration | 1 | 0 | 2.84s | unavailable | unavailable | 2.84s | 2.84s |
| Within-run median request wait | 1 | 0 | 1.23s | unavailable | unavailable | 1.23s | 1.23s |
| Within-run median request transfer | 1 | 0 | 1.27s | unavailable | unavailable | 1.27s | 1.27s |
| Within-run median inter-request gap | 1 | 0 | 41.5106 ms | unavailable | unavailable | 41.5106 ms | 41.5106 ms |
| Largest request duration | 1 | 0 | 32.61s | unavailable | unavailable | 32.61s | 32.61s |
| Largest inter-request gap | 1 | 0 | 39.17s | unavailable | unavailable | 39.17s | 39.17s |
| Time after last model response | 1 | 0 | 221.3579 ms | unavailable | unavailable | 221.3579 ms | 221.3579 ms |
Task identity: ["ink-grid-box-layout","477fda7df7781406d2ba046eb48dd5391e964b72","sha256:73daa7eb11dec0d43e523434d28a66ee9d3ac063fda33d9e26183d867739dbb1","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 11m 4.31s | 10m 18.91s | 11m 49.7s | 9m 33.51s | 12m 35.1s |
| Model | 2 | 0 | 7m 18.18s | 6m 17.68s | 8m 18.67s | 5m 17.19s | 9m 19.17s |
| Non-model | 2 | 0 | 3m 44.86s | 3m 29.76s | 3m 59.95s | 3m 14.67s | 4m 15.04s |
| Startup | 2 | 0 | 1.27s | 1.27s | 1.28s | 1.26s | 1.28s |
| First byte | 2 | 0 | 1.23s | 1.21s | 1.25s | 1.19s | 1.27s |
| API turns | 2 | 0 | 81.5 | 67.75 | 95.25 | 54 | 109 |
| Input tokens | 2 | 0 | 7279030 | 5455210 | 9102849 | 3631390 | 10926670 |
| Output tokens | 2 | 0 | 72986 | 64571.5 | 81400.5 | 56157 | 89815 |
| Cached input (%) | 2 | 0 | 98.81026 | 98.63102 | 98.9895 | 98.45178 | 99.16874 |
| Static cost (USD) | 2 | 0 | 0.07643694 | 0.06464497 | 0.0882289 | 0.052853 | 0.1000209 |
| Model request count | 2 | 0 | 81.5 | 67.75 | 95.25 | 54 | 109 |
| Within-run median request duration | 2 | 0 | 2.93s | 2.87s | 2.99s | 2.81s | 3.05s |
| Within-run median request wait | 2 | 0 | 1.23s | 1.21s | 1.25s | 1.19s | 1.27s |
| Within-run median request transfer | 2 | 0 | 1.64s | 1.62s | 1.65s | 1.61s | 1.66s |
| Within-run median inter-request gap | 2 | 0 | 237.1555 ms | 150.6906 ms | 323.6205 ms | 64.22561 ms | 410.0855 ms |
| Largest request duration | 2 | 0 | 52.27s | 49.96s | 54.58s | 47.66s | 56.89s |
| Largest inter-request gap | 2 | 0 | 1m 30.56s | 1m 30.56s | 1m 30.56s | 1m 30.56s | 1m 30.56s |
| Time after last model response | 2 | 0 | 199.3329 ms | 180.3733 ms | 218.2926 ms | 161.4137 ms | 237.2522 ms |
Task identity: ["psd-tools-blend-range-api","01595992c5f4d4695251fdf58b3edb5302d3ba46","sha256:fbfcc5b7e68a2d8fd97688c69808049d582113368f388929d7d46ac44a592cd9","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 4m 29.96s | 4m 15.78s | 4m 44.13s | 4m 1.6s | 4m 58.31s |
| Model | 2 | 0 | 4m 7.36s | 3m 53.45s | 4m 21.28s | 3m 39.53s | 4m 35.19s |
| Non-model | 2 | 0 | 21.84s | 21.58s | 22.09s | 21.33s | 22.34s |
| Startup | 2 | 0 | 755.8208 ms | 745.3459 ms | 766.2956 ms | 734.871 ms | 776.7705 ms |
| First byte | 2 | 0 | 1.18s | 1.17s | 1.19s | 1.16s | 1.2s |
| API turns | 2 | 0 | 50.5 | 48.25 | 52.75 | 46 | 55 |
| Input tokens | 2 | 0 | 3463246 | 3002380 | 3924111 | 2541515 | 4384976 |
| Output tokens | 2 | 0 | 45522 | 43302.5 | 47741.5 | 41083 | 49961 |
| Cached input (%) | 2 | 0 | 98.02849 | 97.90477 | 98.15221 | 97.78105 | 98.27593 |
| Static cost (USD) | 2 | 0 | 0.04740457 | 0.04398449 | 0.05082465 | 0.04056441 | 0.05424473 |
| Model request count | 2 | 0 | 50.5 | 48.25 | 52.75 | 46 | 55 |
| Within-run median request duration | 2 | 0 | 2.61s | 2.51s | 2.7s | 2.42s | 2.79s |
| Within-run median request wait | 2 | 0 | 1.18s | 1.17s | 1.19s | 1.16s | 1.2s |
| Within-run median request transfer | 2 | 0 | 1.36s | 1.27s | 1.45s | 1.18s | 1.54s |
| Within-run median inter-request gap | 2 | 0 | 39.92659 ms | 31.57239 ms | 48.28079 ms | 23.21819 ms | 56.635 ms |
| Largest request duration | 2 | 0 | 28.63s | 25.53s | 31.73s | 22.43s | 34.83s |
| Largest inter-request gap | 2 | 0 | 8.48s | 8.31s | 8.64s | 8.15s | 8.8s |
| Time after last model response | 2 | 0 | 148.254 ms | 144.9128 ms | 151.5951 ms | 141.5717 ms | 154.9362 ms |
Task identity: ["superjson-error-stack-serialization","f143004734ea2dd4090e45013a07f091f5bbaee7","sha256:dcc10590630fcc927468099eb5118e4bc7170a1f12e01ca5c9404011de072444","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 6m 12.33s | 6m 6.25s | 6m 18.4s | 6m 0.18s | 6m 24.47s |
| Model | 2 | 0 | 5m 51.58s | 5m 45.64s | 5m 57.52s | 5m 39.69s | 6m 3.46s |
| Non-model | 2 | 0 | 19.19s | 19s | 19.38s | 18.8s | 19.58s |
| Startup | 2 | 0 | 1.56s | 1.49s | 1.62s | 1.43s | 1.68s |
| First byte | 2 | 0 | 1.2s | 1.19s | 1.2s | 1.19s | 1.2s |
| API turns | 2 | 0 | 36 | 34.5 | 37.5 | 33 | 39 |
| Input tokens | 2 | 0 | 2196063 | 2139098 | 2253027 | 2082134 | 2309991 |
| Output tokens | 2 | 0 | 69037.5 | 67646.75 | 70428.25 | 66256 | 71819 |
| Cached input (%) | 2 | 0 | 98.36392 | 98.31002 | 98.41782 | 98.25612 | 98.47173 |
| Static cost (USD) | 2 | 0 | 0.05327424 | 0.05257368 | 0.05397481 | 0.05187311 | 0.05467537 |
| Model request count | 2 | 0 | 36 | 34.5 | 37.5 | 33 | 39 |
| Within-run median request duration | 2 | 0 | 3.96s | 3.44s | 4.47s | 2.93s | 4.98s |
| Within-run median request wait | 2 | 0 | 1.2s | 1.19s | 1.2s | 1.19s | 1.2s |
| Within-run median request transfer | 2 | 0 | 2.82s | 2.3s | 3.33s | 1.78s | 3.85s |
| Within-run median inter-request gap | 2 | 0 | 297.9029 ms | 219.9541 ms | 375.8516 ms | 142.0054 ms | 453.8004 ms |
| Largest request duration | 2 | 0 | 1m 8.75s | 1m 7.16s | 1m 10.33s | 1m 5.57s | 1m 11.92s |
| Largest inter-request gap | 2 | 0 | 3.19s | 2.46s | 3.92s | 1.73s | 4.64s |
| Time after last model response | 2 | 0 | 168.4131 ms | 159.4187 ms | 177.4074 ms | 150.4244 ms | 186.4018 ms |
Task identity: ["textual-richlog-follow-state","a12215b791587a6d94227bd30a0e9291de932106","sha256:73fd3d800de9d5dae47edd9b5269e40966664b72d49d8f598964b8b91fdd0b92","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 8m 39.28s | 7m 39.71s | 9m 38.85s | 6m 40.14s | 10m 38.42s |
| Model | 2 | 0 | 6m 27.59s | 6m 16.07s | 6m 39.12s | 6m 4.55s | 6m 50.64s |
| Non-model | 2 | 0 | 2m 10.91s | 1m 22.86s | 2m 58.96s | 34.81s | 3m 47.01s |
| Startup | 2 | 0 | 776.9418 ms | 775.1977 ms | 778.6859 ms | 773.4536 ms | 780.43 ms |
| First byte | 2 | 0 | 1.25s | 1.25s | 1.26s | 1.24s | 1.27s |
| API turns | 2 | 0 | 67 | 66.5 | 67.5 | 66 | 68 |
| Input tokens | 2 | 0 | 4898918 | 4699566 | 5098269 | 4500214 | 5297621 |
| Output tokens | 2 | 0 | 65377 | 62183 | 68571 | 58989 | 71765 |
| Cached input (%) | 2 | 0 | 98.60173 | 98.55577 | 98.6477 | 98.5098 | 98.69366 |
| Static cost (USD) | 2 | 0 | 0.06393858 | 0.06134537 | 0.06653179 | 0.05875216 | 0.069125 |
| Model request count | 2 | 0 | 67 | 66.5 | 67.5 | 66 | 68 |
| Within-run median request duration | 2 | 0 | 3.34s | 3.16s | 3.52s | 2.98s | 3.7s |
| Within-run median request wait | 2 | 0 | 1.25s | 1.25s | 1.26s | 1.24s | 1.27s |
| Within-run median request transfer | 2 | 0 | 1.92s | 1.81s | 2.03s | 1.69s | 2.14s |
| Within-run median inter-request gap | 2 | 0 | 39.85328 ms | 33.76583 ms | 45.94072 ms | 27.67838 ms | 52.02817 ms |
| Largest request duration | 2 | 0 | 56.83s | 56.3s | 57.36s | 55.78s | 57.89s |
| Largest inter-request gap | 2 | 0 | 1m 27.9s | 49.12s | 2m 6.68s | 10.35s | 2m 45.45s |
| Time after last model response | 2 | 0 | 198.8423 ms | 182.2021 ms | 215.4825 ms | 165.5618 ms | 232.1227 ms |
Task identity: ["tomlkit-toml-table-converters","962de91aaf103135f400e5108a798a59a10dc17e","sha256:68dd891458463c9f930dcbbd59b4442065c145d66ce3b5813af82ca5ec8f4966","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 9m 44.66s | 9m 42.13s | 9m 47.19s | 9m 39.61s | 9m 49.72s |
| Model | 2 | 0 | 9m 13.32s | 9m 2.43s | 9m 24.2s | 8m 51.55s | 9m 35.08s |
| Non-model | 2 | 0 | 30.58s | 22.24s | 38.93s | 13.89s | 47.28s |
| Startup | 2 | 0 | 760.8648 ms | 756.5231 ms | 765.2065 ms | 752.1815 ms | 769.5481 ms |
| First byte | 2 | 0 | 1.3s | 1.29s | 1.3s | 1.28s | 1.31s |
| API turns | 2 | 0 | 71 | 65.5 | 76.5 | 60 | 82 |
| Input tokens | 2 | 0 | 6426091 | 5814645 | 7037538 | 5203198 | 7648984 |
| Output tokens | 2 | 0 | 94303.5 | 91414.25 | 97192.75 | 88525 | 100082 |
| Cached input (%) | 2 | 0 | 98.75362 | 98.59454 | 98.91269 | 98.43546 | 99.07177 |
| Static cost (USD) | 2 | 0 | 0.08706221 | 0.08678058 | 0.08734385 | 0.08649895 | 0.08762548 |
| Model request count | 2 | 0 | 71 | 65.5 | 76.5 | 60 | 82 |
| Within-run median request duration | 2 | 0 | 4.07s | 3.78s | 4.37s | 3.48s | 4.67s |
| Within-run median request wait | 2 | 0 | 1.3s | 1.29s | 1.3s | 1.28s | 1.31s |
| Within-run median request transfer | 2 | 0 | 2.7s | 2.39s | 3s | 2.08s | 3.31s |
| Within-run median inter-request gap | 2 | 0 | 45.63356 ms | 45.10808 ms | 46.15905 ms | 44.58259 ms | 46.68453 ms |
| Largest request duration | 2 | 0 | 1m 46.82s | 1m 23.04s | 2m 10.61s | 59.26s | 2m 34.39s |
| Largest inter-request gap | 2 | 0 | 18.18s | 12.97s | 23.39s | 7.76s | 28.6s |
| Time after last model response | 2 | 0 | 214.0648 ms | 194.411 ms | 233.7186 ms | 174.7571 ms | 253.3724 ms |
Task identity: ["true-myth-iterable-collection-combinators","5945ad1b96e228ff716275d8a273282101152508","sha256:0ecebbb45e561b37ee5d7bcbe80588427f4890fb80fffbbb73ca6b94c7cb8855","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 6m 30.86s | 5m 54.52s | 7m 7.2s | 5m 18.18s | 7m 43.54s |
| Model | 2 | 0 | 5m 9.83s | 4m 42.76s | 5m 36.9s | 4m 15.69s | 6m 3.97s |
| Non-model | 2 | 0 | 1m 20.25s | 1m 10.98s | 1m 29.53s | 1m 1.7s | 1m 38.81s |
| Startup | 2 | 0 | 779.8653 ms | 771.756 ms | 787.9747 ms | 763.6466 ms | 796.0841 ms |
| First byte | 2 | 0 | 1.31s | 1.31s | 1.31s | 1.3s | 1.32s |
| API turns | 2 | 0 | 58.5 | 54.25 | 62.75 | 50 | 67 |
| Input tokens | 2 | 0 | 6181474 | 5969969 | 6392978 | 5758464 | 6604483 |
| Output tokens | 2 | 0 | 52370.5 | 49793.25 | 54947.75 | 47216 | 57525 |
| Cached input (%) | 2 | 0 | 98.10791 | 98.03257 | 98.18325 | 97.95723 | 98.25859 |
| Static cost (USD) | 2 | 0 | 0.06706598 | 0.06498144 | 0.06915052 | 0.0628969 | 0.07123507 |
| Model request count | 2 | 0 | 58.5 | 54.25 | 62.75 | 50 | 67 |
| Within-run median request duration | 2 | 0 | 2.71s | 2.69s | 2.74s | 2.66s | 2.76s |
| Within-run median request wait | 2 | 0 | 1.31s | 1.31s | 1.31s | 1.3s | 1.32s |
| Within-run median request transfer | 2 | 0 | 1.28s | 1.27s | 1.3s | 1.25s | 1.31s |
| Within-run median inter-request gap | 2 | 0 | 27.15436 ms | 24.76437 ms | 29.54434 ms | 22.37439 ms | 31.93432 ms |
| Largest request duration | 2 | 0 | 24.76s | 23.28s | 26.24s | 21.8s | 27.72s |
| Largest inter-request gap | 2 | 0 | 17.68s | 11.5s | 23.86s | 5.32s | 30.04s |
| Time after last model response | 2 | 0 | 190.9312 ms | 176.4667 ms | 205.3957 ms | 162.0022 ms | 219.8602 ms |
Task identity: ["cattrs-partial-structuring-recovery","91df27b2a20f26c35777a92f4555438b62c06345","sha256:b985f2bd17e9e80bee0fa9b7c0747b2ded92d54d070b32ebfe102e8829a44076","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 8m 23.53s | 7m 34.05s | 9m 13.01s | 6m 44.58s | 10m 2.49s |
| Model | 2 | 0 | 6m 22.38s | 5m 41.71s | 7m 3.06s | 5m 1.03s | 7m 43.74s |
| Non-model | 2 | 0 | 1m 59.73s | 1m 50.93s | 2m 8.53s | 1m 42.13s | 2m 17.33s |
| Startup | 2 | 0 | 1.42s | 1.41s | 1.42s | 1.41s | 1.42s |
| First byte | 2 | 0 | 1.27s | 1.26s | 1.27s | 1.26s | 1.27s |
| API turns | 2 | 0 | 77.5 | 72.25 | 82.75 | 67 | 88 |
| Input tokens | 2 | 0 | 6611401 | 6327280 | 6895523 | 6043158 | 7179644 |
| Output tokens | 2 | 0 | 68183 | 59592 | 76774 | 51001 | 85365 |
| Cached input (%) | 2 | 0 | 98.62539 | 98.44449 | 98.80629 | 98.26359 | 98.98719 |
| Static cost (USD) | 2 | 0 | 0.07440574 | 0.06606655 | 0.08274493 | 0.05772736 | 0.09108413 |
| Model request count | 2 | 0 | 77.5 | 72.25 | 82.75 | 67 | 88 |
| Within-run median request duration | 2 | 0 | 2.8s | 2.69s | 2.91s | 2.57s | 3.02s |
| Within-run median request wait | 2 | 0 | 1.27s | 1.26s | 1.27s | 1.26s | 1.27s |
| Within-run median request transfer | 2 | 0 | 1.41s | 1.33s | 1.49s | 1.25s | 1.57s |
| Within-run median inter-request gap | 2 | 0 | 42.95303 ms | 40.9597 ms | 44.94637 ms | 38.96636 ms | 46.9397 ms |
| Largest request duration | 2 | 0 | 33.99s | 30s | 37.99s | 26.01s | 41.98s |
| Largest inter-request gap | 2 | 0 | 1m 0.19s | 58.63s | 1m 1.74s | 57.07s | 1m 3.3s |
| Time after last model response | 2 | 0 | 172.6681 ms | 172.2081 ms | 173.1281 ms | 171.7481 ms | 173.5881 ms |
Task identity: ["happy-dom-deterministic-intersectionobserver","32b2a8e60eab482c023af377160bf294d6da0318","sha256:a4df294272f2165ce894306ab1eec5284fe426ddaa66cf08bb3c4755a9f8faa2","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 8m 45.54s | 8m 39.63s | 8m 51.45s | 8m 33.72s | 8m 57.35s |
| Model | 2 | 0 | 6m 36.48s | 6m 32.57s | 6m 40.4s | 6m 28.65s | 6m 44.31s |
| Non-model | 2 | 0 | 2m 6.51s | 2m 4.28s | 2m 8.74s | 2m 2.05s | 2m 10.97s |
| Startup | 2 | 0 | 2.54s | 2.3s | 2.78s | 2.07s | 3.02s |
| First byte | 2 | 0 | 1.28s | 1.28s | 1.29s | 1.28s | 1.29s |
| API turns | 2 | 0 | 84 | 75.5 | 92.5 | 67 | 101 |
| Input tokens | 2 | 0 | 7621791 | 6986911 | 8256672 | 6352030 | 8891552 |
| Output tokens | 2 | 0 | 70098 | 68977.5 | 71218.5 | 67857 | 72339 |
| Cached input (%) | 2 | 0 | 99.0052 | 98.9996 | 99.01081 | 98.99399 | 99.01641 |
| Static cost (USD) | 2 | 0 | 0.07604899 | 0.07395103 | 0.07814694 | 0.07185308 | 0.08024489 |
| Model request count | 2 | 0 | 84 | 75.5 | 92.5 | 67 | 101 |
| Within-run median request duration | 2 | 0 | 2.33s | 2.23s | 2.44s | 2.12s | 2.55s |
| Within-run median request wait | 2 | 0 | 1.28s | 1.28s | 1.29s | 1.28s | 1.29s |
| Within-run median request transfer | 2 | 0 | 1.03s | 936.4409 ms | 1.13s | 841.2368 ms | 1.22s |
| Within-run median inter-request gap | 2 | 0 | 53.11387 ms | 47.45446 ms | 58.77329 ms | 41.79504 ms | 64.43271 ms |
| Largest request duration | 2 | 0 | 39.44s | 32.1s | 46.78s | 24.76s | 54.12s |
| Largest inter-request gap | 2 | 0 | 39.87s | 39.69s | 40.05s | 39.51s | 40.23s |
| Time after last model response | 2 | 0 | 200.1704 ms | 190.5562 ms | 209.7845 ms | 180.9421 ms | 219.3987 ms |
Task identity: ["ink-grid-box-layout","477fda7df7781406d2ba046eb48dd5391e964b72","sha256:73daa7eb11dec0d43e523434d28a66ee9d3ac063fda33d9e26183d867739dbb1","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 18m 31.08s | 17m 6.65s | 19m 55.51s | 15m 42.21s | 21m 19.95s |
| Model | 2 | 0 | 9m 37.47s | 9m 36.44s | 9m 38.5s | 9m 35.41s | 9m 39.53s |
| Non-model | 2 | 0 | 8m 51.64s | 7m 26.18s | 10m 17.1s | 6m 0.73s | 11m 42.55s |
| Startup | 2 | 0 | 1.97s | 1.96s | 1.98s | 1.95s | 1.99s |
| First byte | 2 | 0 | 1.36s | 1.35s | 1.37s | 1.34s | 1.38s |
| API turns | 2 | 0 | 107 | 104 | 110 | 101 | 113 |
| Input tokens | 2 | 0 | 12909490 | 12764410 | 13054570 | 12619330 | 13199650 |
| Output tokens | 2 | 0 | 94567.5 | 91852.25 | 97282.75 | 89137 | 99998 |
| Cached input (%) | 2 | 0 | 99.28941 | 99.2625 | 99.31632 | 99.23559 | 99.34323 |
| Static cost (USD) | 2 | 0 | 0.1089767 | 0.1062501 | 0.1117033 | 0.1035235 | 0.1144299 |
| Model request count | 2 | 0 | 107 | 104 | 110 | 101 | 113 |
| Within-run median request duration | 2 | 0 | 2.69s | 2.68s | 2.71s | 2.66s | 2.72s |
| Within-run median request wait | 2 | 0 | 1.36s | 1.35s | 1.37s | 1.34s | 1.38s |
| Within-run median request transfer | 2 | 0 | 1.3s | 1.28s | 1.33s | 1.25s | 1.35s |
| Within-run median inter-request gap | 2 | 0 | 55.98387 ms | 51.64215 ms | 60.32559 ms | 47.30043 ms | 64.66731 ms |
| Largest request duration | 2 | 0 | 46.47s | 45.68s | 47.27s | 44.89s | 48.06s |
| Largest inter-request gap | 2 | 0 | 1m 53.73s | 1m 35.97s | 2m 11.49s | 1m 18.21s | 2m 29.26s |
| Time after last model response | 2 | 0 | 190.5779 ms | 189.0073 ms | 192.1486 ms | 187.4366 ms | 193.7193 ms |
Task identity: ["psd-tools-blend-range-api","01595992c5f4d4695251fdf58b3edb5302d3ba46","sha256:fbfcc5b7e68a2d8fd97688c69808049d582113368f388929d7d46ac44a592cd9","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 5m 12.82s | 5m 5.27s | 5m 20.38s | 4m 57.71s | 5m 27.94s |
| Model | 2 | 0 | 4m 28.3s | 4m 21.16s | 4m 35.45s | 4m 14.01s | 4m 42.6s |
| Non-model | 2 | 0 | 42.99s | 42.57s | 43.4s | 42.16s | 43.81s |
| Startup | 2 | 0 | 1.54s | 1.54s | 1.54s | 1.54s | 1.54s |
| First byte | 2 | 0 | 1.23s | 1.21s | 1.24s | 1.2s | 1.26s |
| API turns | 2 | 0 | 65.5 | 63.25 | 67.75 | 61 | 70 |
| Input tokens | 2 | 0 | 5210321 | 4975289 | 5445352 | 4740258 | 5680383 |
| Output tokens | 2 | 0 | 48117 | 46765.5 | 49468.5 | 45414 | 50820 |
| Cached input (%) | 2 | 0 | 98.83025 | 98.70054 | 98.95995 | 98.57084 | 99.08966 |
| Static cost (USD) | 2 | 0 | 0.05328125 | 0.05235454 | 0.05420796 | 0.05142784 | 0.05513467 |
| Model request count | 2 | 0 | 65.5 | 63.25 | 67.75 | 61 | 70 |
| Within-run median request duration | 2 | 0 | 2.5s | 2.48s | 2.52s | 2.46s | 2.54s |
| Within-run median request wait | 2 | 0 | 1.23s | 1.21s | 1.24s | 1.2s | 1.26s |
| Within-run median request transfer | 2 | 0 | 1.19s | 1.17s | 1.2s | 1.16s | 1.22s |
| Within-run median inter-request gap | 2 | 0 | 53.34718 ms | 46.70914 ms | 59.98521 ms | 40.07111 ms | 66.62324 ms |
| Largest request duration | 2 | 0 | 28.38s | 27.98s | 28.79s | 27.58s | 29.19s |
| Largest inter-request gap | 2 | 0 | 12.52s | 12.33s | 12.7s | 12.15s | 12.88s |
| Time after last model response | 2 | 0 | 172.5431 ms | 168.3703 ms | 176.7159 ms | 164.1976 ms | 180.8887 ms |
Task identity: ["superjson-error-stack-serialization","f143004734ea2dd4090e45013a07f091f5bbaee7","sha256:dcc10590630fcc927468099eb5118e4bc7170a1f12e01ca5c9404011de072444","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 4m 58.16s | 4m 51.47s | 5m 4.85s | 4m 44.77s | 5m 11.54s |
| Model | 2 | 0 | 4m 44.66s | 4m 38.2s | 4m 51.13s | 4m 31.73s | 4m 57.6s |
| Non-model | 2 | 0 | 11.45s | 11.21s | 11.68s | 10.98s | 11.91s |
| Startup | 2 | 0 | 2.05s | 2.04s | 2.06s | 2.03s | 2.06s |
| First byte | 2 | 0 | 1.22s | 1.22s | 1.23s | 1.21s | 1.23s |
| API turns | 2 | 0 | 39 | 38 | 40 | 37 | 41 |
| Input tokens | 2 | 0 | 2879117 | 2808420 | 2949815 | 2737722 | 3020512 |
| Output tokens | 2 | 0 | 58368.5 | 57081.75 | 59655.25 | 55795 | 60942 |
| Cached input (%) | 2 | 0 | 98.76394 | 98.5183 | 99.00958 | 98.27265 | 99.25523 |
| Static cost (USD) | 2 | 0 | 0.04878772 | 0.04731659 | 0.05025886 | 0.04584545 | 0.05173 |
| Model request count | 2 | 0 | 39 | 38 | 40 | 37 | 41 |
| Within-run median request duration | 2 | 0 | 2.83s | 2.75s | 2.92s | 2.67s | 3s |
| Within-run median request wait | 2 | 0 | 1.22s | 1.22s | 1.23s | 1.21s | 1.23s |
| Within-run median request transfer | 2 | 0 | 1.5s | 1.46s | 1.54s | 1.42s | 1.58s |
| Within-run median inter-request gap | 2 | 0 | 50.21848 ms | 48.53808 ms | 51.89888 ms | 46.85768 ms | 53.57928 ms |
| Largest request duration | 2 | 0 | 1m 18.42s | 1m 13.69s | 1m 23.15s | 1m 8.96s | 1m 27.87s |
| Largest inter-request gap | 2 | 0 | 1.77s | 1.73s | 1.82s | 1.68s | 1.86s |
| Time after last model response | 2 | 0 | 185.5003 ms | 184.1887 ms | 186.8118 ms | 182.8771 ms | 188.1234 ms |
Task identity: ["textual-richlog-follow-state","a12215b791587a6d94227bd30a0e9291de932106","sha256:73fd3d800de9d5dae47edd9b5269e40966664b72d49d8f598964b8b91fdd0b92","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 11m 17.39s | unavailable | unavailable | 11m 17.39s | 11m 17.39s |
| Model | 1 | 0 | 7m 53.56s | unavailable | unavailable | 7m 53.56s | 7m 53.56s |
| Non-model | 1 | 0 | 3m 22.32s | unavailable | unavailable | 3m 22.32s | 3m 22.32s |
| Startup | 1 | 0 | 1.51s | unavailable | unavailable | 1.51s | 1.51s |
| First byte | 1 | 0 | 1.28s | unavailable | unavailable | 1.28s | 1.28s |
| API turns | 1 | 0 | 103 | unavailable | unavailable | 103 | 103 |
| Input tokens | 1 | 0 | 9847811 | unavailable | unavailable | 9847811 | 9847811 |
| Output tokens | 1 | 0 | 74629 | unavailable | unavailable | 74629 | 74629 |
| Cached input (%) | 1 | 0 | 99.07712 | unavailable | unavailable | 99.07712 | 99.07712 |
| Static cost (USD) | 1 | 0 | 0.08768063 | unavailable | unavailable | 0.08768063 | 0.08768063 |
| Model request count | 1 | 0 | 103 | unavailable | unavailable | 103 | 103 |
| Within-run median request duration | 1 | 0 | 2.73s | unavailable | unavailable | 2.73s | 2.73s |
| Within-run median request wait | 1 | 0 | 1.28s | unavailable | unavailable | 1.28s | 1.28s |
| Within-run median request transfer | 1 | 0 | 1.37s | unavailable | unavailable | 1.37s | 1.37s |
| Within-run median inter-request gap | 1 | 0 | 55.48858 ms | unavailable | unavailable | 55.48858 ms | 55.48858 ms |
| Largest request duration | 1 | 0 | 32.91s | unavailable | unavailable | 32.91s | 32.91s |
| Largest inter-request gap | 1 | 0 | 2m 46.15s | unavailable | unavailable | 2m 46.15s | 2m 46.15s |
| Time after last model response | 1 | 0 | 211.5528 ms | unavailable | unavailable | 211.5528 ms | 211.5528 ms |
Task identity: ["textual-richlog-follow-state","a12215b791587a6d94227bd30a0e9291de932106","sha256:73fd3d800de9d5dae47edd9b5269e40966664b72d49d8f598964b8b91fdd0b92","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 1 | 0 | 11m 47.47s | unavailable | unavailable | 11m 47.47s | 11m 47.47s |
| Model | 1 | 0 | 8m 25.77s | unavailable | unavailable | 8m 25.77s | 8m 25.77s |
| Non-model | 1 | 0 | 3m 20.22s | unavailable | unavailable | 3m 20.22s | 3m 20.22s |
| Startup | 1 | 0 | 1.49s | unavailable | unavailable | 1.49s | 1.49s |
| First byte | 1 | 0 | 1.32s | unavailable | unavailable | 1.32s | 1.32s |
| API turns | 1 | 0 | 112 | unavailable | unavailable | 112 | 112 |
| Input tokens | 1 | 0 | 10800230 | unavailable | unavailable | 10800230 | 10800230 |
| Output tokens | 1 | 0 | 75131 | unavailable | unavailable | 75131 | 75131 |
| Cached input (%) | 1 | 0 | 99.38158 | unavailable | unavailable | 99.38158 | 99.38158 |
| Static cost (USD) | 1 | 0 | 0.08729757 | unavailable | unavailable | 0.08729757 | 0.08729757 |
| Model request count | 1 | 0 | 112 | unavailable | unavailable | 112 | 112 |
| Within-run median request duration | 1 | 0 | 3.2s | unavailable | unavailable | 3.2s | 3.2s |
| Within-run median request wait | 1 | 0 | 1.32s | unavailable | unavailable | 1.32s | 1.32s |
| Within-run median request transfer | 1 | 0 | 1.67s | unavailable | unavailable | 1.67s | 1.67s |
| Within-run median inter-request gap | 1 | 0 | 44.78508 ms | unavailable | unavailable | 44.78508 ms | 44.78508 ms |
| Largest request duration | 1 | 0 | 29.9s | unavailable | unavailable | 29.9s | 29.9s |
| Largest inter-request gap | 1 | 0 | 2m 47.09s | unavailable | unavailable | 2m 47.09s | 2m 47.09s |
| Time after last model response | 1 | 0 | 187.145 ms | unavailable | unavailable | 187.145 ms | 187.145 ms |
Task identity: ["tomlkit-toml-table-converters","962de91aaf103135f400e5108a798a59a10dc17e","sha256:68dd891458463c9f930dcbbd59b4442065c145d66ce3b5813af82ca5ec8f4966","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 11m 45.91s | 10m 29.24s | 13m 2.59s | 9m 12.57s | 14m 19.26s |
| Model | 2 | 0 | 11m 11.97s | 9m 56.72s | 12m 27.22s | 8m 41.47s | 13m 42.47s |
| Non-model | 2 | 0 | 32.51s | 31.08s | 33.94s | 29.65s | 35.37s |
| Startup | 2 | 0 | 1.43s | 1.42s | 1.44s | 1.42s | 1.45s |
| First byte | 2 | 0 | 1.4s | 1.39s | 1.41s | 1.37s | 1.42s |
| API turns | 2 | 0 | 118 | 106 | 130 | 94 | 142 |
| Input tokens | 2 | 0 | 15991170 | 13555210 | 18427120 | 11119260 | 20863070 |
| Output tokens | 2 | 0 | 105828.5 | 94377.75 | 117279.3 | 82927 | 128730 |
| Cached input (%) | 2 | 0 | 99.46726 | 99.41923 | 99.51529 | 99.3712 | 99.56332 |
| Static cost (USD) | 2 | 0 | 0.1233058 | 0.1083489 | 0.1382627 | 0.09339193 | 0.1532197 |
| Model request count | 2 | 0 | 118 | 106 | 130 | 94 | 142 |
| Within-run median request duration | 2 | 0 | 3.07s | 2.92s | 3.23s | 2.76s | 3.39s |
| Within-run median request wait | 2 | 0 | 1.4s | 1.39s | 1.41s | 1.37s | 1.42s |
| Within-run median request transfer | 2 | 0 | 1.6s | 1.46s | 1.75s | 1.31s | 1.89s |
| Within-run median inter-request gap | 2 | 0 | 61.67266 ms | 54.66434 ms | 68.68097 ms | 47.65603 ms | 75.68929 ms |
| Largest request duration | 2 | 0 | 1m 7.18s | 48.38s | 1m 25.99s | 29.57s | 1m 44.8s |
| Largest inter-request gap | 2 | 0 | 5.46s | 4.24s | 6.68s | 3.02s | 7.9s |
| Time after last model response | 2 | 0 | 269.5044 ms | 227.5411 ms | 311.4677 ms | 185.5779 ms | 353.4309 ms |
Task identity: ["true-myth-iterable-collection-combinators","5945ad1b96e228ff716275d8a273282101152508","sha256:0ecebbb45e561b37ee5d7bcbe80588427f4890fb80fffbbb73ca6b94c7cb8855","prepared-local"]
| Metric | Valid n | Missing n | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|---|
| End-to-end | 2 | 0 | 5m 7.79s | 4m 27.09s | 5m 48.49s | 3m 46.39s | 6m 29.18s |
| Model | 2 | 0 | 4m 34.43s | 3m 59.92s | 5m 8.94s | 3m 25.42s | 5m 43.44s |
| Non-model | 2 | 0 | 31.47s | 25.07s | 37.88s | 18.67s | 44.28s |
| Startup | 2 | 0 | 1.88s | 1.67s | 2.1s | 1.46s | 2.31s |
| First byte | 2 | 0 | 1.31s | 1.31s | 1.32s | 1.31s | 1.32s |
| API turns | 2 | 0 | 62.5 | 54.75 | 70.25 | 47 | 78 |
| Input tokens | 2 | 0 | 7229000 | 5979712 | 8478289 | 4730423 | 9727577 |
| Output tokens | 2 | 0 | 50426.5 | 43702.75 | 57150.25 | 36979 | 63874 |
| Cached input (%) | 2 | 0 | 98.35655 | 98.13728 | 98.57582 | 97.91801 | 98.79509 |
| Static cost (USD) | 2 | 0 | 0.06779656 | 0.05932641 | 0.0762667 | 0.05085626 | 0.08473685 |
| Model request count | 2 | 0 | 62.5 | 54.75 | 70.25 | 47 | 78 |
| Within-run median request duration | 2 | 0 | 2.73s | 2.69s | 2.77s | 2.66s | 2.81s |
| Within-run median request wait | 2 | 0 | 1.31s | 1.31s | 1.32s | 1.31s | 1.32s |
| Within-run median request transfer | 2 | 0 | 1.33s | 1.32s | 1.34s | 1.32s | 1.34s |
| Within-run median inter-request gap | 2 | 0 | 45.27462 ms | 44.96272 ms | 45.58653 ms | 44.65082 ms | 45.89843 ms |
| Largest request duration | 2 | 0 | 21.44s | 21.05s | 21.82s | 20.67s | 22.2s |
| Largest inter-request gap | 2 | 0 | 7.45s | 5.57s | 9.33s | 3.69s | 11.2s |
| Time after last model response | 2 | 0 | 179.5599 ms | 178.8347 ms | 180.2851 ms | 178.1095 ms | 181.0103 ms |
Deterministic exemplars from this export, not a ranking. Blue bars are successful model calls; red bars are unsuccessful ones. Gray marks time after the latest response through reconciled adapter end. Every curve uses the full timeline, with observations added in response-completion order. Dashed red markers identify missing per-call observations; a flat known subtotal across a marker does not mean zero usage or cost.
Successful-response usage only: input, cached input, uncached input and output trajectories exclude unsuccessful calls. Missing successful usage remains unavailable; curves show only known subtotals when incomplete. Static costs use known per-call estimates and identify every missing call cost.
most model calls among passes. hermes @ Hermes Agent v0.20.5 (2026.8.19) · textual-richlog-follow-state · Pass (completed). Calls 190 (190 successful, 0 failed). Median duration 3.3s; largest gap 2m 26.77s; time after last 253.7433 ms.
largest inter-request gap among passes. qwen @ 0.22.2 · textual-richlog-follow-state · Pass (completed). Calls 103 (103 successful, 0 failed). Median duration 2.73s; largest gap 2m 46.15s; time after last 211.5528 ms.
longest tail after last model response among passes. codex @ codex-cli 0.149.1 · ink-grid-box-layout · Pass (completed). Calls 79 (79 successful, 0 failed). Median duration 3.36s; largest gap 30.17s; time after last 5.17s.
longest single model call among passes. pi @ 0.73.1 · tomlkit-toml-table-converters · Pass (completed). Calls 60 (60 successful, 0 failed). Median duration 4.67s; largest gap 7.76s; time after last 174.7571 ms.
most model calls among verification failures. cline @ 3.0.61 · textual-richlog-follow-state · Verification failure (verify_error). Calls 1267 (1267 successful, 0 failed). Median duration 2.33s; largest gap 30.23s; time after last 2.31s.
most failed model calls among verification failures. cline @ 3.0.61 · cattrs-partial-structuring-recovery · Verification failure (verify_error). Calls 932 (932 successful, 0 failed). Median duration 2.28s; largest gap 30.11s; time after last 1.69s.
Pass = completed native verification; verification failure = verify_error; other = timeout or adapter_error. verify_error alone does not distinguish task failure from verifier infrastructure failure. Failure timing stays separate and is not ranked against passes.
Coverage counts all selected attempts. Each metric has its own valid n and missing n; timing, usage, caching and cost denominators can differ. Unavailable is not zero.
IQR tasks counts successful timing tasks with at least two eligible repetitions, out of all successful timing tasks. Headline spread is the median measurable within-task IQR, not pooled spread.
Outcome spend is the sum of selected static estimates. When any costs are missing, the displayed amount is only a known subtotal. Failed attempts and other outcomes are reported separately from pass spend.
Distributions describe repetitions within one task, harness/version and outcome. Q1–Q3 and min–max describe observed samples, not confidence intervals; singleton quartiles are unavailable.
Matched comparisons include only common successful task identities. Each task contributes one within-task median; summary times are medians across those task medians. The descriptive ratio is the median of right / left task ratios, not a causal effect or ratio of unrelated headline medians.
Task identity includes base revision, verifier image and environment. Repetitions are not paired seeds; execution windows can differ. Recorded host fields do not establish machine identity.
Request metrics come from sanitized C1 model attempts. Wait is first-byte minus request start; transfer is last-byte minus first-byte. Synthetic first-byte markers from network failures are unavailable. Gaps exclude overlapping model intervals; time after last is reconciled adapter end minus the latest response end. Request summary duration, wait, transfer and gap columns are medians of within-run medians, with metric-specific available and missing run counts.
Timing uses the existing C1 interval-union derivation. Unreconciled timing is unavailable; outcome groups remain separate.
Usage covers successful model responses; turns count identifiable API attempts. Static costs are not invoices and exclude attempts absent from this export.
Matched comparisons use only common tasks with reconciled native passes on both sides. Repetition indices are not paired seeds. Samples and execution windows can differ.
No matched-model comparison is produced when the recorded model is unspecified.
Host fields describe recorded hardware, not unique machine identity. Same-host populations do not establish causal harness effects.
Quartiles describe observed runs, not confidence intervals. Singleton spread is unavailable. Non-model time includes unobserved tool and harness work.
Request series are sanitized C1 model attempts: duration, wait, transfer, gaps and per-call tokens/cost. Paths, bodies and logs are not exported. Relationships are within a run or matched task, not ratios of unrelated headline medians.
Uses recorded grades; estimates are excluded.
| Harness | Observed difference | Exploratory 95% interval |
|---|---|---|
| codex @ codex-cli 0.149.1 | +25.0 pp | -12.5 pp to +62.5 ppNo clear separation |
| pi @ 0.73.1 | +25.0 pp | +6.3 pp to +50.0 ppDirectional in this sample |
| hermes @ Hermes Agent v0.20.5 (2026.8.19) | 0.0 pp | -25.0 pp to +25.0 ppNo clear separation |
| cline @ 3.0.61 | -56.3 pp | -87.5 pp to -25.0 ppDirectional in this sample |
Exploratory 95% percentile task-cluster bootstrap, 10,000 deterministic resamples. Each task and all its repetitions move together; harnesses are paired by task, not by random seed. Marginal intervals are not corrected for multiple comparisons or winner selection. Eight task clusters provide limited evidence about other coding tasks. These intervals do not remove configuration, environment or verifier confounding.
Three extra attempts are outside the selected 80. Prefix estimates do not give complete charges.
| Attempt | Outcome | Adapter time | Known prefix reference estimate | Full cost |
|---|---|---|---|---|
| pinned:cline:textual-richlog-follow-state:0 | timeout | 10,800.372 s | $1.836203754 | unknown |
| pinned:cline:tomlkit-toml-table-converters:1 | budget-interrupted | unknown | $0.444080070 | unknown |
| pinned:pi:textual-richlog-follow-state:0 | interrupted | unknown | $0.003784620 | unknown |
All extra attempt facts and captured request series · Original cohorts and collection windows · Collection ledger · Historical snapshot
Estimated pass rates use the retained per-run audit. Other pass rates use recorded grades. Original results remain in the source data.