Parallel Cheap-Scout
Fan-Out Benchmark

Research Receipt · Issue #174 / Idea Bank Rank 20
Authors: Marcus Rafael B. Tiongson & Nova (Head Researcher, Gaia Research)
Pinned commit SHA: 04c2ca1b904623a97aaeafb8d629aa954efb4008
Evaluator: Gold Claude Opus 4.6 (claude-opus-4-6)
Reasoning Effort Contract: Minimal (Low, Light) reasoning effort recorded across all runs; effort calibration is out of scope
Ledger Schema: scout-bench/v1 (360 committed records, 9 tasks × 8 configurations × 5 repeats)
Reproduce: npx tsx scripts/scout-bench/ledger.ts validate
Follow-Up Investigation: Issue #178 (Orchestrator Ingestion Overhead & Multi-Model Explorer Fan-Out)

Abstract

State-of-the-art agent architectures conventionally rely on a single mid-tier LLM for codebase scouting, file localization, and context pruning. We empirically evaluate whether replacing this monolithic scout with $K$ concurrent instances of an ultra-cheap model (gemini-3.5-flash-lite), fused via zero-token deterministic rank aggregation (Reciprocal Rank Fusion, $k=60$), establishes a superior cost-performance Pareto frontier. Across 360 runs on 9 tasks spanning codebase localization, document retrieval, and skill pruning, $K=4$ parallel lite scouts achieve 100.0% recall (surpassing a single Flash 3.7 at 98.9%), elevate prompt-cache hit rates from 35.0% to 80.0%, and reduce quality variance ($\sigma(F_2)$) by over 4.4x (0.138 to 0.031). We evaluate result checks using Claude Opus 4.6 under a strict Minimal (Low, Light) reasoning effort baseline (effort calibration is out of scope). Furthermore, we model the hidden downstream reading costs across the most utilized orchestrator models—Fable 5, Opus 5, Sonnet 5, Gemini Flash 3.7, GPT Sol 5.6, GPT Terra 5.6, Grok 4.6, and ZAI 5.3—demonstrating that unbounded scout output concatenation inflates orchestrator reading costs by $O(K)$, whereas deterministic top-$M$ rank aggregation compresses context overhead by 72.4% while preserving the 100% recall ceiling. Finally, we establish concrete research recommendations for cross-ecosystem investigation, including Anthropic Claude Haiku 4.5 explorers vs. Sonnet 5 and OpenAI GPT Luna / mini explorer tiers.

Introduction

In multi-agent coding harnesses, autonomous agents allocate a large fraction of token expenditures and latency to exploratory reconnaissance—discovering candidate files, inspecting type declarations, filtering tool catalogs, and pruning irrelevant context. Mainstream agent harnesses (claude-code, cursor, copilot-workspace) almost universally deploy a single mid-tier scout model (e.g. gemini-3.7-flash, claude-sonnet-5, gpt-sol-5.6) sequentially per task step.

This standard design suffers from two structural handicaps:

1. Perspective Blindness & Single-Trajectory Failure: A single scout sampling at low temperature commits early to a greedy exploration path. On lightweight models, single-path recall drops to 78.9%; even standard mid-tier models exhibit an average missing-candidate rate of 1.1% (98.9% recall).
2. Context Ingestion Inefficiency: Upgrading the single scout to a larger reasoning model increases per-token cost and latency without addressing directional blindness. Conversely, naive fan-out risks overwhelming the downstream orchestrator with redundant tokens.

This investigation tests the hypothesis that fanning out $K$ lightweight scouts in parallel—each focused on distinct subspaces or prompt framings and sharing a common prefix—achieves higher recall at lower net cost due to prefix prompt-cache reuse. We additionally quantify orchestrator ingestion overhead across leading frontier models and benchmark deterministic aggregation strategies to prevent context bloat.

Methodology

Architectural Postures & Experimental Matrix

We evaluate four structural postures across 9 deterministic benchmark tasks with 5 repeats each ($N=360$ verified runs):

Evaluator & Reasoning Effort Contract

• Gold Evaluator: All ground-truth evaluation and scoring was conducted by Claude Opus 4.6 (claude-opus-4-6).
• Reasoning Effort Discipline: Minimal (Low, Light) reasoning efforts were recorded across all runs. Dynamic effort calibration is treated as out of scope for this benchmark to isolate pure structural routing performance without confounding adaptive inference budgets.

Pricing Contract & Frontier Model Matrix (per 1M Tokens)

Deterministic Aggregation (Reciprocal Rank Fusion)

To avoid LLM token consumption during candidate consolidation, scout outputs are scored deterministically: $\text{Score}(d) = \sum_{k=1}^K \frac{\mathbb{I}(d \in R_k)}{60 + \text{rank}_k(d)}$ Candidates meeting quorum ($q \ge 2$) or exceeding rank thresholds are emitted directly to the downstream recipient.

Results

Benchmark Aggregates Across 360 Runs

Empirical Findings

1. Recall Dominance ($K^* = 4$): $K=4$ lite scouts reach 100.0% recall across all test suites, matching the theoretical ceiling and surpassing Single Flash 3.7 (98.9%).
2. Prompt-Cache Amplification: Because parallel scouts share system prompt prefixes and base file schemas, cache hit rates rise from 35.0% to 80.0% at $K=4$, reducing marginal scout execution costs to pennies.
3. Variance Reduction: Quality variance ($\sigma(F_2)$) drops by over 4.4x (from 0.138 in Single Lite to 0.031 in Parallel Lite $K=4$).
4. Cascaded Funnel Precision: Architecture D (4 Lite Scouts $\to$ RRF $\to$ 1 Flash Verifier) eliminates false positive noise, achieving 95.6% precision and $F_2 = 0.989$.

Orchestrator Ingestion Costs Across Modern Frontier Models

When parallel scouts discover candidates, how much does context reading cost the lead orchestrator? We compare naive raw output concatenation against bounded zero-token RRF across the most utilized orchestrator models in the landscape:

Key finding: In top-tier orchestrators like Claude Opus 5 ($5.00/1M), Claude Fable 5 ($4.00/1M), and GPT Sol 5.6 ($3.00/1M), reading 6.2k tokens of unranked scout noise costs up to$0.0312—exceeding the entire $0.0058 scout fleet execution cost! Bounded RRF aggregation eliminates$0.0226/turn in reading taxes without losing a single relevant candidate.

Conclusions

1. Rule of Thumb for Harness Developers: 4 cheap scouts are much more reliable than one smart one.
2. Default Harness Configuration Guidance: Developers should update their harness's default explore and scouting configuration from 1 monolithic model to parallel scouts ($K=2$ for rapid checks, $K=4$ for Pareto-optimal coverage). This topology is consistently cheaper due to prefix prompt caching and eliminates narrow search blindspots.
3. Reference Implementation Available: Gaia Research ships the reference harness skill gaia-research/skill-scout-fleet (invokable as /scout-fleet via npx skills install gaia-research/skill-scout-fleet).
4. Deterministic Aggregation Is Mandatory: Zero-token RRF deduplication bounds candidate payload size, protecting lead orchestrators (Opus 5, Fable 5, GPT Sol 5.6, Grok 4.6, ZAI 5.3) from context bloat and reading token tax.
5. Cascaded Funnels for Critical Edits: When false positives carry severe downstream execution costs, a two-tier funnel (4 Lite Scouts $\to$ RRF $\to$ 1 Verifier) yields peak precision (95.6%) and $F_2 = 0.989$.

Recommendations

1. Research Agenda: Haiku 4.5 Explorers vs. Sonnet 5

• Baseline Hypothesis: $K=4$ claude-haiku-4-5 explorers ($1.00/1M input,$0.100/1M cached) outperform a single claude-sonnet-5 ($3.00/1M input) in recall while cutting total turn cost by >60% on 5-minute cache windows.
• Empirical Gap: Measure prompt-cache lifetime stability across long-running multi-turn agent sessions and quantify the exact threshold where cache eviction alters the Pareto frontier.

2. Research Agenda: GPT-5.6-Luna Explorer Tier

• Baseline Hypothesis: Parallel $K=4$ gpt-5.6-luna explorers ($0.15/1M input,$0.075/1M cached) compress repository reconnaissance latency by ~45% relative to monolithic gpt-sol-5.6 or gpt-terra-5.6 runs.
• Empirical Gap: Benchmark multi-root mono-repo cross-package symbol discovery to test whether lightweight models maintain 100% recall as repository depth exceeds 10,000 files.

3. Research Agenda: Orchestrator Cognitive Load & Attention Dispersion

• Active Investigation: GitHub Issue #178 investigates downstream orchestrator task accuracy when ingesting bounded vs. unbounded scout candidate payloads.
• Key Metrics to Track: Downstream code edit accuracy ($F_1$), hallucination rates on distractors, and total orchestrator token expenditure across varying candidate limits ($M \in \{1, 3, 5, \infty\}$).

Figure 1: Orchestrator Ingestion Overhead vs. Recall Retention

Lead orchestrator context reading costs across 360 benchmark runs. Bounded RRF aggregation eliminates 72.4% of context overhead.