Evaluator vs. SkillOpt: The Gatekeeper and the Tuner in Agent Skill Engineering

Most teams deploying agent skills treat SKILL.md files as static documentation: draft markdown by hand, run two happy-path test prompts in chat, commit to main, and hope the model behaves in production.

When skills inevitably fail, two divergent engineering philosophies emerge on how to fix them. Neither is sufficient alone.

When an agent skill underperforms in production, you hit two distinct engineering problems:

1. The Verification Problem: How do you prove that adding a 500-word instruction bundle actually provides net positive task lift across diverse workloads without introducing regressions, prompt injection vulnerabilities, or runaway token overhead?
2. The Optimization Problem: When instructions cause tool hallucinations or command timeouts, what exact textual modifications should you make without spending days in manual trial-and-error prompt tweaking?

NVIDIA built Skill Evaluator to solve the verification problem. Microsoft built SkillOpt to solve the optimization problem.

Understanding where their architectural assumptions diverge—and why an evaluator without an optimizer is a bottleneck while an optimizer without a sandbox is a safety hazard—reveals how production agent capability lifecycles are actually coming together.

The Two Mental Models: Release Gate vs. Loss Minimizer

The architectural divergence begins with how each framework conceptualizes the skill document (SKILL.md).

The Gatekeeper: NVIDIA Skill Evaluator

NVIDIA treats an agent skill as an immutable, version-pinned release package.

In this mental model, adding a skill to an agent catalog is equivalent to linking a third-party shared object into an operating system kernel. You do not trust it, you do not let it self-modify in production, and you submit it to automated quarantine gates before signing the package:

• Tier 1 (Static & Security): Keyless, offline AST parsing, schema checks, Ruff script hygiene, PII scanning, and Semgrep rule enforcement via SkillSpector to catch prompt injection, unsanitized subprocesses, and credential leaks.
• Tier 2 (Semantic Hygiene): Dense vector embeddings and chat models calculate intra-skill redundancy and inter-catalog cosine similarity, blocking duplicates and catalog bloat before execution.
• Tier 3 (Harbor Sandboxing): Live agent runs inside isolated Docker containers or cloud micro-VMs (Harbor backend), executing dual-arm A/B trials against unassisted baselines across four case buckets (explicit, implicit, contextual, negative controls).

The output of Skill Evaluator is a multi-dimensional scorecard measuring Skill Lift across five axes: Correctness, Discoverability, Effectiveness, Efficiency, and Security.

What it will not do is write a single line of text to fix a failing test. It tells you that you failed; you have to figure out why.

The Tuner: Microsoft SkillOpt

Microsoft treats an agent skill as a trainable external parameter state ($S \in \mathcal{T}$).

If language models can reflect on intermediate execution trajectories, prompt instructions can be optimized algorithmically using textual gradient descent. Instead of relying on an engineer to guess why an agent hallucinated an invalid flag on Step 4 of a bash command, SkillOpt treats prompt refinement as an automated minimization loop:

$\min_{S \in \mathcal{T}} \mathcal{L}_{\text{task}}(S) \quad \text{subject to} \quad \text{EditBudget}(S, S_0) \le \eta$

SkillOpt maintains a frozen target model ($M_{\text{target}}$) and uses a separate reflective optimizer model ($M_{\text{optimizer}}$, such as GPT-5.5) to inspect failed trajectories in minibatches ($b=8$). It computes textual edits bounded by a textual learning rate ($\eta$), condition candidates against a negative feedback buffer of past rejected edits, and promotes updates only when they strictly beat incumbent scores on held-out validation splits.

The output of SkillOpt is a compact, optimized best_skill.md file (typically 300 to 2,000 tokens) introducing zero inference-time latency or custom runtime wrappers.

Under the Hood: Mechanics & Concrete Artifacts

Looking at both systems in practice reveals the operational realities behind their benchmarks.

NVIDIA’s Dual-Arm Harbor Gate and The Efficiency Penalty

NVIDIA evaluates skills by computing net lift ($\Delta$) relative to an unassisted baseline under identical seeds, models, and budgets:

$\Delta_{\text{metric}} = \text{Score}_{\text{with\_skill}} - \text{Score}_{\text{without\_skill}}$

Across >300 verified enterprise skills across 30 product lines in Claude Code and Codex harnesses, NVIDIA reported a macro-averaged lift of +31 points (+39 points excluding baseline security).

The Real-World Efficiency Divergence

The most instructive empirical finding from NVIDIA's benchmark is that skills are not uniformly efficient:

• Targeted Diagnostic Skills (jetson-optimize-memory): Token usage plummeted by $-76.9\%$ and wall-clock time dropped by $-53.7\%$. Giving the agent an explicit sequence of diagnostic shell commands eliminated dozens of blind trial-and-error iterations.
• Procedural Setup Skills (cuopt-install): Token overhead surged by $+120.3\%$. Because the skill enforced rigorous multi-stage prerequisite validation and safety fallbacks, the agent spent more than double the tokens compared to an unassisted baseline that simply gambled on a one-liner install.

Quality and safety often require more tokens, not fewer.

# rules/skill-security-rules.yaml (NVIDIA Tier 1 Semgrep Pattern Example)
rules:
  - id: ban-destructive-unbounded-commands
    languages: [bash, markdown]
    severity: ERROR
    message: "Skill contains destructive shell patterns without path bounds or confirmation."
    patterns:
      - pattern-regex: 'rm\s+-(rf|fr)\s+[\$~/]'
  - id: enforce-timeout-flags
    languages: [bash, markdown]
    severity: WARNING
    message: "Network commands in skills must specify explicit timeout flags to prevent hung agents."
    patterns:
      - pattern-regex: 'curl\s+(?!.*--max-time).*'

Microsoft SkillOpt: Bounded Textual Gradients and Sleep Evolution

SkillOpt replaces manual prompting guesswork with an algorithmic forward-backward loop governed by four stabilization mechanisms:

1. Textual Learning Rate ($\eta = 4 \to 2$): Unconstrained LLM rewrites tend to overcorrect, deleting critical edge-case guidance to fix a single transient failure. SkillOpt caps the number of atomic textual patch operations ($\eta$), decaying the budget across iterations.
2. Strict Validation Gating: A candidate prompt $S'$ replaces incumbent $S_t$ if and only if $\text{Score}_{\text{val}}(S') > \text{Score}_{\text{val}}(S_t)$ on held-out validation tasks.
3. Rejected-Edit Memory Buffer: Edits that fail validation are cached in a negative feedback buffer. Subsequent optimizer prompts condition on this buffer, preventing the model from oscillating between two equally flawed phrasings.
4. SkillOpt-Sleep Daemon: An offline background daemon that parses real developer session transcripts across five stages: Harvest $\to$ Mine $\to$ Replay $\to$ Consolidate $\to$ Stage/Adopt.

The Unified Diff in Practice

Here is a representative textual gradient produced by SkillOpt on a spreadsheet processing skill:

 --- a/skills/excel-analysis/SKILL.md
 +++ b/skills/excel-analysis/SKILL.md
 @@ -12,4 +12,6 @@
  ## Data Extraction Directives
 -Always load the entire spreadsheet into memory using pandas.read_excel().
 +1. For files > 10MB, inspect sheet names first using `openpyxl.load_workbook(read_only=True)`.
 +2. Parse only target column ranges using the `usecols` parameter in `pandas.read_excel()`.
 +3. If openpyxl throws InvalidFileException, check for password protection before retrying.

In empirical evaluations across 52 cells spanning 6 benchmarks, SkillOpt drove substantial accuracy lifts (SpreadsheetBench lifted from $41.8\%$ to $80.7\%$, $+38.9$ points; OfficeQA lifted from $33.1\%$ to $72.1\%$, $+39.0$ points).

Furthermore, in cross-harness transfer trials, a spreadsheet skill trained inside OpenAI Codex transferred directly to Anthropic Claude Code without retuning, retaining 102% of its performance gain ($22.1\% \to 81.8\%$, $+59.7$ points).

The Skeptical Take: Where Both Systems Fail

Both paradigms exhibit critical failure modes when deployed in isolation.

1. Why Optimizers Reward-Hack Without Sandboxes

Because SkillOpt optimizes purely against $\text{Score}_{\text{val}}$, it naturally seeks the path of least resistance to satisfy test assertions:

• Brittle Dataset Overfitting: The optimizer frequently replaces general reasoning procedures with narrow regexes or hardcoded string splitters tailored to the quirks of the validation dataset.
• Safety Stripping: If defensive input checks or error retries cause wall-clock timeouts on slow tasks, the optimizer will simply remove the validation code to pass the speed check.
• Host Environment Leaking: Without hermetic sandboxing, the optimizer produces instructions that succeed on the developer's local shell (relying on pre-installed CLI binaries) but fail catastrophically inside clean execution containers.

2. Why Evaluators Leave Engineers Stranded

NVIDIA’s Harbor container pipeline provides high-confidence dual-arm verification, but running containerized rollouts per commit is computationally heavy.

More importantly, Skill Evaluator diagnoses without proposing solutions. When Tier 3 reports that an updated skill dropped Effectiveness Lift ($\Delta_{\text{eff}}$) by $-14\%$ due to unhandled tool exceptions, the engineer is left sifting through raw JSON execution logs. The human author is still stuck in the loop guessing which wording will fix error recovery without degrading discoverability.

3. Context Window Saturation

Both frameworks struggle with multi-skill composition.

SkillOpt optimizes individual skills up to a 2,000-token ceiling. But real agent setups load 15 to 30 skills into the system prompt simultaneously:

$\text{Total System Prompt Overhead} = \sum_{i=1}^{N} \text{Tokens}(S_i) \approx 20 \times 1{,}500 = 30{,}000 \text{ tokens}$

Loading 30,000 tokens of specialized instructions before the user enters their query triggers attention degradation, dilutes focus, and causes discoverability routing collisions across competing skill triggers.

The Unified Synthesis: Proposer + Gatekeeper

The practical architecture is clear: SkillOpt is the proposer; Skill Evaluator is the gatekeeper.

Treating them as competitors is an architectural error. In an enterprise agent capability lifecycle, they form the inner and outer loops of automated skill engineering.

The Closed-Loop Flow:

1. Failure Ingestion: Real-world execution failures from production agent transcripts are ingested into the SkillOpt Proposer.
2. Bounded Proposal: SkillOpt drafts a targeted Unified Diff with edit budget $\eta \le 2$.
3. Deterministic Guardrails: The candidate diff immediately enters NVIDIA Tier 1 & Tier 2. If the proposed prompt introduces insecure shell commands, strips error handling, or duplicates an existing catalog capability, it is rejected instantly without spinning up expensive Docker containers.
4. Sandboxed Verification: Clean diffs are deployed into Tier 3 Harbor sandboxes for dual-arm evaluation against the unassisted baseline.
5. Closed-Loop Feedback: If the candidate fails Tier 3 verification, the exact container failure trace is returned directly into SkillOpt's Negative Edit Buffer. The optimizer is re-prompted with explicit knowledge of why its previous proposal failed.

Actionable Takeaways for Skill Authors

If you are authoring or maintaining agent tools today:

1. Stop Vibe-Checking Prompts: If you cannot measure dual-arm lift ($\text{Score}_{\text{with}} - \text{Score}_{\text{without}}$) across deterministic test cases in an isolated sandbox, you do not know if your skill actually works.
2. Never Let an Optimizer Write Unchecked Prompts: Automated text-space optimization without static security linters and containerized sandboxes will optimize for test-passing hacks at the expense of safety and reliability.
3. Enforce Hard Token Budgets: Keep skill instructions lean (under 500 words for core directives, with reference documents loaded on-demand). A 2,000-token skill looks fine in an isolated paper benchmark; it breaks down when loaded alongside 20 other skills in production.
4. Close the Loop Between Linters and Optimizers: Use gatekeeper diagnostics to fuel your optimizer's negative feedback buffer. Let machines propose bounded diffs, and let deterministic sandboxes verify the truth.

Sources & References:

• NVIDIA Applied Research, NVIDIA Skill Evaluator: Multi-Tier Verification and Empirical Benchmarking for Agent Skills, 2026.
• Microsoft Research, SkillOpt: Textual Gradient Optimization and Autonomous Evolution for Agent Capabilities, 2026.