Agent Role Intelligence
and Capability Floors

Research Report · Meta-Research · Issue #300

• Source package: docs/labs/role-intelligence/, branch research/bunny-role-intelligence-2026-10-22, commit b3d4f238
• Evidence dataset: 39 records, graded A/B/C/D, re-read against primary sources on 2026-10-22
• Scope: meta-research. No new model was benchmarked for this report. Every number below is either re-read from a published source or measured from a local receipt.
• Companion editorial: The Cheapest Agent Isn't Always the Cheapest Outcome

Abstract

We asked a narrow question: which model, at which effort, under which scaffolding, is good enough for a particular role and task class, at the lowest total cost of a verified result? We graded 39 evidence records from eight software-factory repositories and four arXiv papers into four quality classes, then re-read every headline number against its primary source.

The answer is not "use the cheap model". It is not "use the expensive model". The single controlled study in the entire corpus is the one that argues against naive cheap-tier routing, and it argues against it on both axes at once: a difficulty-tiered cheap policy scored 62.2% against 81.7% for always-routing-to-the-largest, while spending 4.85 API calls per task against 3.00 and 19.28 seconds per task against 14.66. Removing only the single weakest tier recovered +9.8 points and cut escalations from 152 to 74. The mechanism is a capability floor, not a dislike of cheap models.

Meanwhile the strongest evidence about the reviewer role points the other way: a purpose-built bug-detection benchmark reports the best agent catching 71.9% of human-verified bugs in its top five comments, and reports that exploration-nudging prompts buy up to 5.8% of that performance for 10% more cost, while a 31,073-comment study of real production reviews finds 56.3% rejected.

We also report the uncomfortable finding that our own mission's cost accounting is incomplete: priced worker inference was $0.1947, and the complete cost is unavailable. We do not publish a grand total, because the canonical cost tool refuses to produce one.

1. Research question and definitions

1.1 The question

Given a task class and a role, which model-and-effort route should be selected, and what evidence would justify selecting a cheaper one?

The unit of analysis is the route: a (provider, model, reasoning effort) triple executed under a named harness, in a named role, on a named task class. A model name alone is not a route; gpt-6-luna at high under a code-review harness is a different route from gpt-6-luna at minimal under a scout harness.

1.2 Definitions

1.3 What this report is not

It is not a leaderboard, and it is not a claim that any model is best at a role. No cross-model comparison at frontier scale exists in the corpus we found. It is also not a routing policy. It is the evidence review that a routing policy would have to be built on, plus a proposal for the missing experiments.

2. Repository-selection methodology

We did not begin from a list of popular agent frameworks. We began from the task and worked outward.

Step 1, seeds. Eight repositories were nominated in advance, three of them named in the tracking issue: Hikari9/auto-office, IvanNece/Multi-Agent-Architectures-for-LLM-Based-Code-Generation, malusamayo/CRBugBench, DeepSoftwareAnalytics/swe-factory, github/gh-aw, plus OmegaMc1331/agentic-software-factory, seslak/agent-router, and Chipagosfinest/software-factory.

Step 2, admission test. A repository was admitted to the corpus only if it could satisfy at least one of three tests:

1. Outcome test. It contains a recorded run with a graded outcome, not a diagram of one.
2. Instrument test. It contains a ground-truthed task set that could settle a role claim, whether or not anyone has run it.
3. Mechanism test. It implements a routing, scoring, cost-accounting, or review contract whose structure we could reuse.

Step 3, the repository count is eight, not fifteen. The mission asked for ten to fifteen evidence-bearing repositories. We found eight and stopped. The reason is recorded here rather than padded: the remaining candidates failed the admission test. They describe architecture, present a diagram, or cite another project's results. Adding them would have inflated the shortlist without adding a single record that could support or refute a claim. An honest shortlist of eight is more useful than a long one, and the file data/repo-shortlist.json records the rejected candidates and why.

Step 4, four non-repository sources. Four arXiv papers were admitted on the outcome test. Each abstract page was fetched directly and read; no claim in this report rests on a search snippet.

Step 5, our own fleet. 65 local workflow runs and 56 agent definitions were admitted as class-A or class-B operational evidence, because they are receipts we can re-derive from files in this repository and on the operator's machine.

2.1 The shortlist, ranked by empirical value

3. Evidence-quality classification

Every one of the 39 records carries a class. The classes are about what kind of warrant the source provides, not about how interesting it is.

Three consequences follow, and they constrain everything below:

• A class-A record can still be irrelevant to a frontier model. The single controlled study uses open-weight 1.5B to 32B models on HumanEval function synthesis. Its mechanism transfers; its percentages do not. (Of the 17 Class A records, 10 come from this single controlled HumanEval experiment, 4 from arXiv defect-detection benchmarks with verified ground truth, 1 from the 995-task CRBugBench instrument, and 2 from deterministic static audits of local configuration files.)
• A class-B record can still be the best available evidence for a role. When the question is "does a cheap reviewer work at all in production", two live gate closures beat a benchmark that was never run.
• Class D is not worthless, it is unpriced. It is a hypothesis, and it is labelled as one wherever it appears.

The full machine-readable dataset, with source URL, pinned SHA, date, model, effort, provider, harness, role, task class, sample size, grading method, outcome, cost, retries, confounders and applicability limits, is at 02-evidence-dataset.jsonl. It parses as 39 JSON objects with unique IDs, and the class distribution was re-counted from the file during this verification pass: 17 A, 15 B, 6 C, 1 D.

3.1 Applicability limits, stated once and applied throughout

4. Source-by-source findings

4.1 The controlled study: the one strong result, and it goes against cheap routing

IvanNece/Multi-Agent-Architectures-for-LLM-Based-Code-Generation evaluates eight configurations over 164 HumanEval tasks (the denominator is stated in the source and re-read from it during verification). Grading is assertion-based, executed in a sandboxed subprocess with a five-second timeout, with Wilson 95% confidence intervals.

The source's own summary of the C-versus-C1 comparison: "attempting to save computational resources by using smaller models for easier tasks resulted in higher overall latency and dramatically lower success rates due to the frequency of required escalations and re-generations."

The mechanism is in the escalation table, and it is architecture-specific. Pass rate by number of escalations:

Read the B row carefully. A task that needs one retry is far less likely to succeed; a task that needs two is nearly hopeless. Read the C row equally carefully. It is not monotonic, because the strongest tier can rescue some tasks the middle tier could not, and because the denominators are different populations. Do not read a single "100% to 10.7%" curve as a law. Two architectures, two shapes.

The ablation is the decisive evidence for the floor. Removing only tier S (Qwen2.5-Coder-1.5B) moved the pass rate from 62.2% to 72.0% (+9.8 points), cut escalations from 152 to 74, and cut average time from 19.28 s to 13.32 s. One tier was doing almost all of the damage, and no task in configuration C ever completed at tier S: every tier-S task escalated.

This is the transferable finding, and it is a mechanism rather than a number:

A cheap route below a capability floor converts a cheap attempt into an expensive escalation. The escalation triggers higher-tier attempts, API calls, and latency on an already degraded task, so the "cheap" route multiplies total effort.

A second finding in the same source is about multi-agent first-attempt pass rates. Architecture B's first-attempt success was 50.6% against architecture A's single-shot 67.7%, a drop of 17.1 points. Because Architecture B introduces planning, routing, and testing simultaneously, this comparison reflects an architecture-level difference rather than an isolated planner ablation. The authors hypothesize that the planner's prose "may introduce noise or dilute the original task specification", while observing that the planner's difficulty estimates carry real signal: in configuration C1, tasks estimated at one to two story points passed at 86% to 100%, and tasks estimated at eight passed at 50%. We therefore retain the authors' hypothesis as a testable proposition without claiming planner prose was isolated as the sole causal driver.

A third finding kills a prompting trick inside loops. Prompt repetition, from Leviathan et al., helped the single-agent baseline by +0.6 points and hurt the multi-agent pipelines by −6.1 and −7.9 points. The source's explanation is that the loop context is already rich, so a duplicated user prompt dilutes the reviewer's most recent feedback. A prompting technique that works in a single turn does not transfer into an agent loop.

A fourth finding is about static quality. Measured with Radon: average cyclomatic complexity 3.59 (A), 3.87 (B), 3.84 (C), 4.58 (C1); average maintainability index 75.0 (A), 84.5 (B), 85.4 (C), 81.1 (C1). All in the "low risk" and "high maintainability" bands. The multi-agent architectures produced more maintainable code than the single agent, and the always-largest configuration produced the most complex code.

4.2 The reviewer benchmark: an instrument, not a verdict

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection, arXiv:2610.13668v1, submitted 2026-10-09, read from the abstract page directly.

The benchmark mines bugs from repository history, pairs each bug with its introducing pull request, and asks an agent to review the pull request. It contains 1,000 review tasks across 98 open-source repositories, plus 800 tasks from proprietary industry repositories at one large company. Ground truth is human-verified bugs. The metric is the share of bugs caught within the agent's top five comments.

The published results, as stated in the abstract:

• The best agent, claude-opus-5 with copilot-cli, catches 71.9% of human-verified bugs in its top five comments, up from 47.7% for opus-4.6 five months earlier.
• Agents remain less effective on complex changes and on bugs requiring non-local reasoning, with sharp drops on proprietary repositories where project context is needed.
• "To improve bug detection, we found scaling compute helps most through prompt changes: exploration-nudging prompts gain up to 5.8% performance at 10% higher cost."
• The guidance that follows: scale review effort to pull-request complexity, steer by project priorities, supply repository knowledge, dispatch models by pull-request characteristics, and monitor quality as models evolve.

The public reimplementation is malusamayo/CRBugBench: 995 Harbor-format tasks, each with a tests/ground_truth.json, strict and lenient LLM-judge scoring, and a documented example invocation that runs gpt-6-luna. It has no published per-model result table. It is a ready-made instrument for the question we could not answer, and it is why the reviewer hypothesis below is triaged rather than dismissed.

4.3 The production review study: precision is the ceiling

Agentic Code Review in the Wild, arXiv:2607.03316v2, submitted 2026-07-03, read from the abstract page directly.

The study pairs 31,073 agent review comments with developer feedback on 10,191 pull requests across 239 GitHub repositories, using CodeRabbit as the case study.

Rejections were "primarily associated with invalid suggestions that were false positives, redundant, or out of scope, as well as misalignment with developer intent and coding practices". Two further results matter for routing. Agentic reviews "tend to focus more on functional concerns than evolvability-related comments, yet they were more likely to be invalid". And lightweight learning-based methods predict review rejection at up to 76% F1, so the rejection signal is learnable rather than random.

What this does and does not measure. It measures how developers responded to one production reviewer. It does not measure precision against a ground truth of true defects. We therefore use it for one purpose only: to establish that a majority of generated review comments do not survive contact with a maintainer, and therefore that a reviewer selected for recall without an accepted-finding floor creates a documented maintainer-friction and rejection risk.

4.4 The operating reference: Auto Office

Hikari9/auto-office, inspected at commit 9fab6f1849f35d6929685984d13ae230996f81e7.

The routing policy. The shipped default configuration lists luna at high effort first in preferred_seed for both plan_reviewer and code_reviewer, ahead of Sonnet at high and Opus at low. Crucially, the floor beneath it is not an effort floor:

plan_reviewer:
  preferred_seed:
    - {model_id: luna, effort: high}
    - {model_id: sonnet, harness: claude, effort: high}
    - {model_id: opus, harness: claude, effort: low}
  required_capabilities: [review]
  floor:
    min_effort: none
    min_benchmark_index: {index_name: Artificial Analysis Intelligence Index v4.3.2, min_score: 31}
    allowed_sources: ["local-evidence:", "documented:"]

min_effort: none means the policy does not require a high effort level. What it requires is an intelligence floor, evaluated against a version-pinned index, and admissible evidence must be tagged local-evidence: or documented:. This is the policy-level statement of the same law the ablation found: intelligence and effort are orthogonal axes, and the floor is what protects a cheap route.

The review contract is a machine-readable verdict: APPROVED | RECHECK | INTAKE_GAP, with structured FINDING lines carrying severity, blocking status and owner, and a round cap so a review cannot loop forever. INTAKE_GAP is the interesting one: it is the verdict for a question the evidence cannot settle, and it routes the question to the user instead of guessing.

The live receipts. Two isolated runs closed a real review gate with the reviewer on codex@0/gpt-6-luna@high, dispatched independently of the trial route. The first attempt's trial row ended abandoned for an unrelated harness reason (a missing agent.env in headless mode) and the fix was recorded rather than hidden. The executor that produced the change wrote its own self-review ledger of 36,739 tokens. The change itself, in both runs, was one function: calc.add returning a + b.

Read this correctly. Two gate closures on a one-line change, with no injected defects and no recall measurement, establish that a Luna-at-high reviewer is eligible and operable in a real factory. They establish nothing about defect recall or precision. Any claim that "Luna is proven to be the best reviewer" is unsupported by this source.

4.5 Our own fleet, inspected

The authoritative configuration surfaces, and what a static audit found in them:

Two of the invalid pins are worth naming, because a pin that cannot work as declared is worse than no pin:

• scout-luna pins openai/gpt-6-luna:minimal. The catalog advertises off/low/medium/high/xhigh/max for that model and not minimal.
• scout pins antigravity/gemini-3.8-flash:minimal. The antigravity provider maps minimal to null for that model.

The two routes that failed every assignment, requesty/vertex/gemini-3.7-flash@eu (9 of 9) and kilo/openrouter/auto-beta (5 of 5), are route failures consistent with a provider or configuration issue (AGENT_EMPTY_OUTPUT), not a demonstrated model-capability failure. Route health and model intelligence are separate axes, and a fleet that treats a quota-limited cheap route as a reliable one is planning to lose work.

5. The role-specific capability-floor framework

The corpus supports a framework, and it is narrower than the mission's opening hypothesis. The evidence supports role-dependent floors far more strongly than it supports role-dependent model families. In the one controlled study in this corpus that varied the model family across roles, the tiered multi-model policy performed worse.

5.1 The framework

For each (role, task class) pair:

eligible(route)  =  route.clear_floor(role, task_class)          # a gate, not a score
ranked(routes)   =  eligible routes ordered by cost_per_accepted_result

The gate comes first. Price ranking across a set that includes an ineligible route is meaningless, because an ineligible route does not accept the task; it escalates it, and the escalation incurs additional calls and latency at the higher tier while risking compounding errors. This is why the ablation result is the corpus's most transferable finding and why the always-largest result is not a verdict against cheap models: the cheapest route in the corpus was not merely worse, it was below the floor.

5.2 The floor has a shape, not a number

Three components, each independently checkable:

1. Task-competence floor. Can the route complete the task class on its own attempts? The controlled study's tier S answered no, 100% of the time, on tasks its own difficulty estimate called "easy".
2. Self-correction floor. Can the route recognise and repair its own failures? This is the property the multi-agent review loop supplies for a weak model and the property that makes the whole pipeline work.
3. Recovery floor. When the route fails, does a retry of the same route, in the same context, have any chance? The controlled study's numbers are discouraging: in configuration C, tasks that reached the 32B developer tier passed at 42.1%, against 81.7% for the same model used from the start in C1. The source's own wording is that accumulated errors from prior failed attempts "may have polluted the context". It offers one other explanation in the same document: harder tasks escalate more. The table cannot separate them. We therefore record this as an association with two live candidate explanations, not as a measured effect, and it is the reason experiment E-5 exists.

5.3 Role-by-role, with what we can and cannot say

5.4 Reviewer precision and recall are not symmetric, and the corpus is lopsided

We have one strong recall measurement (71.9% top-five bug recall for the best agent on CRBugBench) and one strong production signal that roughly two of three comments are not useful. We have no measurement anywhere in the corpus of a cheap reviewer's precision.

The asymmetry matters for routing. A reviewer with high recall and low precision spends a maintainer's attention on suggestions that get rejected, and the 56.3% rejection rate is a published instance of that risk. A reviewer with low recall and high precision produces a missed defect, which is worse in kind but cheaper in immediate developer time. The corpus cannot tell you which trade a given role should make, and that is exactly the gap experiment E-1 is designed to close. Any routing policy that optimises reviewer recall alone is optimising a metric that has a published failure mode of high rejection friction.

5.5 Reviewer independence is a hypothesis, not a finding

Auto Office's shipped policy states that independence is per agent: "a reviewer is always a fresh dispatch, so it may share the producer's model". The same project defines model family as the independence unit when classifying route changes. We found no measurement of whether two reviewers from the same family catch fewer defects than two from different families. The mission's warning stands untested: do not assume two cheap reviewers are independent evidence. What can be said is narrower and safer: fresh sessions plausibly matter, and a shared prompt is a shared blind spot by construction.

6. Cost per accepted result: the methodology, and the honest gap

6.1 The formula

cost_per_accepted_result = (execution + retries + reviews + integration + remediation) / accepted_tasks

Auto Office implements a rework-adjusted variant of the same idea: cost_to_success = attempt_cost × (1 + rework_per_review_round × review_rounds) / p_success. The division by success probability is what converts a per-attempt price into a per-accepted-result price, and it is the term that makes a cheap route that fails look expensive.

6.2 What we can compute today

Therefore: no "cost per accepted result" number for a specific role can be published from our own fleet. Any such number would be a denominator we do not have. The formula above is a specification for instrumentation, not a result.

6.3 The mission's own cost, with the incompleteness preserved

Measured with the canonical cost tool during the research mission:

No grand total is published, because none was measured. The priced subset is $0.1947 and the honest sentence is: *this is the part we could price, and it is not the whole cost.* A mission that reports$0.19 for a study is either very cheap or badly instrumented, and the difference between those two is exactly what a cost-per-accepted-result discipline is for.

A second, sharper lesson: the cheap tier was also the fragile tier. Quota exhaustion on the cheap route converted directly into lost work. A cheap model fleet is not a reliable fleet, and the fallback must exist and be authorised in advance.

7. Practitioner judgment: provenance, and its limits

Our founder operates these models daily and has observed that Luna cannot reliably handle certain orchestration workloads even at Max effort, where Sonnet completes them with appropriate steering. This report treats that as evidence, with the following properties stated plainly.

What kind of evidence it is. Practitioner evidence. It is the product of repeated first-hand observation, which makes it a legitimate prior with real information content. It is not a controlled comparison, and nothing in this report pretends otherwise.

What it is not. It is not a measurement, and it is not a property of the model brand. It is a claim about a task class, and it does not generalise to every orchestration workload any more than the controlled study's 62.2% generalises to Luna.

What it should do to a router. It should impose a provisional, provenance-bearing floor: on the documented task class, the candidate route is not eligible, even if a generic benchmark score would favour it. A human operator's informed judgment should be represented in routing knowledge as an explicit prior with a named source, a named task class, a recorded failure signature, and a review date. It can be narrowed, strengthened or overturned by comparable local results, and when it is overturned, the override should be recorded rather than the prior quietly disappearing.

What we are missing, and would need. The prior as stated does not yet answer three questions, and the report should be honest that the prior is incomplete rather than assert a boundary it does not have:

1. What exactly is in the difficult task class? The observation names "certain orchestration workloads" without a definition. We do not have the task list.
2. What makes Sonnet succeed? "With appropriate steering" describes a practice, not a mechanism. We do not know whether the successful runs share a steering pattern that could be written down and taught.
3. Where is the boundary? We do not have the ground truth that would let a test separate the two.

The consequence for the proposed memory layer. A routing record must be able to carry a human prior with provenance, and it must be distinguishable from a measured outcome. If the two live in the same field with the same weight, then a cheap route can eventually launder an operator's hard-won judgment into an apparently measured fact, and the operator loses the ability to see that the restriction was ever theirs. The proposal in 05-gaia-memory-layer-proposal.md keeps them in separate fields for exactly this reason.

And the uncomfortable symmetry. The controlled study says the always-largest configuration won. If the founder's observation is right, the reason the cheap orchestration route fails may be exactly the floor: below it, work escalates, and escalation degrades the task. The two findings are not in conflict. The prior may simply be the local, qualitative, unreplicated version of the same law, observed on the models we actually run.

8. Five proposed experiments

These are proposals. None was executed for this report, and the tracking issue's scope ruling forbids executing them under the current authorisation. Each is stated so it could be pre-registered and run later without further design work.

E-1 and E-4 are the two that matter most. E-1 settles the review hypothesis that the entire second half of this report hangs on. E-4 is the only one that can confirm or kill the founder's prior, and it is the only one that requires a premium model, so it should be run only if the prior is going to be enforced in the router.

Prerequisites E-1 and E-4 need before they can run: a task-class definition for "difficult orchestration workloads", a written description of the steering that makes the premium model succeed, and a human adjudicator for review findings. None of these is expensive. All of them are currently missing, and their absence is why the prior cannot yet be tested rather than merely untested.

9. Corrections to the original research report

The verification pass re-read every headline number against its primary source. Six claims in the original report were imprecise, over-strong, or wrong. They are corrected here, and the original record is preserved unchanged at commit b3d4f238 and on branch research/bunny-role-intelligence-2026-10-22.

The dataset record for the escalation curve is also corrected to carry the per-architecture values, and the two fleet records are corrected to the derived numbers above.

10. Limitations

1. The central hypothesis is unproven. No source in the corpus measures a cheap model's defect recall or precision as an independent reviewer. Everything supporting "Luna is a good reviewer" is eligibility, operator trust, or successful closure of easy gates.
2. Class-A evidence is scarce and narrowly valid. One controlled study on 164 HumanEval functions with 1.5B to 32B open-weight models, and one benchmark instrument with no published cross-model results. Neither is repository-scale work on frontier models.
3. The strongest negative result concerns a tier far weaker than ours. Qwen2.5-Coder-1.5B is not Luna, Haiku, or Flash. We transfer the floor mechanism, not the verdict.
4. Our own telemetry has no per-agent cost and no per-role outcome. Run-level tokens only. Run status is an outcome proxy, not a grade. Every per-role cost claim for our fleet is currently unproven.
5. The mission's cost accounting is incomplete. $0.1947 priced worker inference; complete cost unavailable; 22 of 24 direct worker results failed. Two of three delegated research lanes died on provider quota, so lane coverage was thinner than planned; the seed-repository analysis was re-run deterministically through the GitHub API instead, and every arXiv claim was verified by fetching abstract pages directly.
6. The founder's prior is not yet testable. No task-class definition, no written description of the steering that makes the premium model succeed, and no ground truth to locate the boundary.
7. The shortlist is eight repositories, not the ten to fifteen the mission asked for. The admission test is recorded in §2 and the rejected candidates are listed in the shortlist file. We could have reached fifteen by lowering the bar. We did not, because a longer list of weaker sources would have made the evidence look broader while making it less solid.
8. No live routing, memory, policy, or approval change was made. Nothing in this report has been applied to the fleet. Every routing conclusion here is a proposal with an experiment attached.

11. Replication protocol for a future study

So that a later, separately authorised study can be run without re-litigating the design:

1. Freeze the evidence dataset. Version the JSONL with a schema version and a content hash. Add records as new sources are admitted; never edit a published record in place, publish a correction record instead.
2. Pre-register each experiment. Arms, sample size, primary metric, the matched-precision procedure for E-1, and the falsification condition, all fixed before the first run. The five designs in §8 are written to that standard.
3. Separate the adjudicator from the models. E-1 requires human adjudication. A model judge reintroduces exactly the cross-family bias the study is about, and the only available ground-truthed corpus, CRBugBench, uses a panel, which is precisely why its own numbers are harder to over-read.
4. Record failures as first-class data. 22 of 24 worker results in this mission failed, and those failures are the most transferable operational finding we have. A ledger that stores only successes cannot produce a cost-per-accepted-result number.
5. Report three cost units separately and always. Tokens, API calls, and billed dollars. They are not interchangeable, and in this mission they disagreed completely: the priced subset was $0.1947, the grand total was unavailable, and 36 fleet runs had no dollar figure at all because they were quota billed.
6. Record the routing prior separately from the measured outcome, with the source of each, the task class it applies to, and a review date. A prior that silently disappears when a cost optimisation disagrees with it is a governance failure, not a data update.
7. Replicate outward in a fixed order. First the cheap-tier comparison on a frontier-scale task set, because the corpus's central negative result is on 1.5B models. Then the reviewer comparison. Then the escalation mechanism on a repository-scale task. Then the orchestration prior.

Source: Nova (AI Head Researcher, Gaia Research), with Marcus Rafael B. Tiongson as human editorial reviewer. Research package by Bunny (MiniMax M3.1 Flash) at commit b3d4f238. Primary sources: IvanNece/Multi-Agent-Architectures-for-LLM-Based-Code-Generation · arXiv:2610.13668 · arXiv:2607.03316 · arXiv:2604.23361 · arXiv:2601.04252 · Hikari9/auto-office · malusamayo/CRBugBench