Nymrel research · 2026-09-04

A small pilot found a ceiling, not a pairing advantage.

All six workflows completed five small, sanitized offline Python repair tasks. Every workflow passed; the two-pass workflows took more component time and did not improve the result.

This does not establish that one workflow is better for production development, harder studio work, or general code quality. It is a narrow, reproducible record with no cash cost claim.

Download the full evidence Read the method

5 tasks37 checks per workflow222 executed checks30 native CLI calls

The result

Each workflow completed every task. The faster one-pass totals are descriptive only; component timing is not an end-to-end productivity or cost measure.

Astra5 / 5 passed49.64 s total component time, five tasks

Fable native workflow5 / 5 passed43.34 s total component time, five tasks

Astra → Astra5 / 5 passed107.52 s total component time, five tasks

Fable → Fable5 / 5 passed103.35 s total component time, five tasks

Astra → Fable5 / 5 passed110.82 s total component time, five tasks

Fable → Astra5 / 5 passed106.55 s total component time, five tasks

Two-pass rows include both calls. Initial answers feed later refinements, so the outcomes are correlated. Adding the six rows together would double-count reused first passes.

Method and evidence

The pilot used the same five offline repair tasks, fresh contexts, unlabeled refinements, held-out checks, and no grader feedback or performance-based retries. Same-model second passes control for extra work; five cases are descriptive, not statistically significant.

The 30 scored calls totaled 335.3 seconds, while claim-to-collection wall time was 33.7 minutes, including 28.1 minutes outside scored calls. That window includes setup, method reviews, fixes, and orchestration; it excludes earlier startup and later reporting or publication. It was not an end-to-end efficiency win.

All evidence is downloadable: tasks, checks, oracle, candidates, grades, and timing.
Published cases are contaminated for future model evaluations.
The included evaluator uses restricted Python and time limits, not an OS sandbox; inspect any cooperative submission before running it.

The original first-call receipt check encountered Windows CRLF serialization. The documented amendment normalized text validation while preserving the original evidence; tasks, order, answers, and grader did not change.

What this cannot show

It does not test production development, isolated base-model ability, or harder studio work.
It does not measure universal code quality, production speed, or end-to-end studio savings.
Provider token definitions and CLI context overhead differ; compute was not equalized.
Fable native calls may include Haiku background work. Astra identity is verified in runtime context, while provider-resolved identity was unavailable.
Subscription use does not establish cash cost, and no cash cost was measured.

Inspect the record

Nymrel publishes the reviewed bundle unchanged. The original public gist remains a historical source, not the primary record for this Nymrel research page.