codex-mmo/docs/EVALUATION.md

Profile evaluation

The generation-8 evaluation format asks whether a composition improves a workload, not whether workers merely returned. Every variant starts from the same fixture and hidden trial mutation, is scored by topology-neutral outcomes, and records coordination diagnostics separately.

Suites from any other MMO generation are rejected. There is no evaluation migration or compatibility reader.

Bundled suites

Suite	Profile	Distinguishing workload
adaptive-change	adaptive-engineering	Decomposable versus tightly coupled changes and delegation selectivity
codex-harness	codex-harness-team	Native context isolation, homogeneous peers, and a fresh supervised critic
debugging-confidence	high-confidence-debugging	Misleading symptoms, independent reproduction, hidden regressions, correction cycles
research-currentness	research-backed-engineering	Conflicting current sources, authority, re-fetching, and source-to-code fidelity
security-assurance	secure-change	Vulnerability recall, false positives, empirical reproduction, remediation validation
visual-conformance	visual-engineering	Real browser screenshots, responsive behavior, pixel/edge evidence, accessibility
contract-refactoring	contract-first-refactoring	Hidden invariants and plausible-but-incompatible implementations
incident-triage	incident-hypothesis-triage	Noisy evidence, competing causes, route outage, time to supported diagnosis
access-efficient	access-efficient-escalation-lab	Quality versus local/API/scarce-tier consumption
route-resilience	route-resilience-lab	Typed credential-loss, rate-limit, and timeout faults; serving-provider identity; fallback behavior
bounded-research	bounded-research-organization-lab	Flat versus bounded hierarchical research and handoff loss
competing-implementations	competing-implementations-lab	Two empirical candidates versus one after integration effort and cost

There is no shared “engineering core” that accidentally rewards one topology. Each suite is workload-specific and declares its own neutral outcome contract.

Matched variants

Every suite contains, as applicable:

• the configured root with delegation disabled;
• the strongest task-specific single-agent candidate;
• one single-agent candidate for each available access product;
• the configured root plus its highest-value worker;
• the complete profile;
• one full_without_worker ablation for every reachable worker.

Unavailable access-product controls are recorded as unavailable rather than rebound to a different route. Each pruned topology removes spawn and control authority to unreachable roles, regenerates its immutable guidance/tool surface, and recomputes exact weighted-resource ceilings through the same derivation used by profile resolution. A comparison is meaningful only when fixture, hidden mutation, task, trial index, and release baseline match.

Each task runs three development trials by default and five release trials with --trial-mode release. Negative controls test whether the root avoids delegation when launch cost exceeds expected value. Difficulty bands distinguish easy tasks from the hard cases a profile claims to improve.

Hidden work

Suite directories may include:

fixture/                 public starting repository
holdout/TASK_ID/         validation-only files installed for that task
mutations/TASK_ID/*.patch

For a given task/trial, the evaluator deterministically chooses and applies the same hidden mutation to every matched variant. Holdout files are copied only into the trial workspace and are hash-recorded in the run. Symlinks, special files, escaping paths, and non-patch mutation members fail validation.

This is concealment from the evaluated prompt, not a claim of cryptographic secrecy from a user who owns the package.

Suite shape

schema_version = 8
id = "example"
profile = "adaptive-engineering"
name = "Example workload"
description = "A matched, measurable task."
fixture = "fixture"
development_trials = 3
release_trials = 5

[promotion]
primary_metric = "success_rate"
direction = "higher"
strongest_success_tolerance = 0.02
minimum_relative_improvement = 0.10
minimum_absolute_improvement = 0.05
worker_minimum_success_contribution = 0.02
worker_minimum_metric_contribution = 0.10
no_regression_higher_metrics = []
no_regression_lower_metrics = []
require_complete_api_cost = true

[[variants]]
id = "configured-root"
purpose = "Configured root with delegation disabled."
topology = "root_only"
comparison_class = "configured_root_alone"

[[variants]]
id = "full-profile"
purpose = "Complete composition."
topology = "full"
comparison_class = "full_profile"

[[tasks]]
id = "repair"
description = "Repair one bounded defect."
sandbox = "workspace-write"
difficulty = "hard"
negative_control = false
wall_timeout_seconds = 1800
validation_timeout_seconds = 120
prompt = "Repair the defect and end with EVAL_DONE."
images = ["reference.png"]

[tasks.outcome_assertions]
expected_patterns = ["EVAL_DONE"]
forbidden_patterns = ["UNVERIFIED"]
validation_commands = ["python -m unittest -v"]

[tasks.orchestration_assertions]
min_peak_mcp_workers = 1
max_jobs = 3
min_result_acceptance_rate = 1.0
max_contract_failures = 0
max_observed_mcp_wait_ratio = 0.8

outcome_assertions are valid for every topology. orchestration_assertions are diagnostics for variants where the referenced roles exist; the compiler rejects unknown roles and impossible declarations rather than encoding a general expression language.

wall_timeout_seconds is an evaluator boundary, not a model instruction or profile execution budget. If it expires, the noninteractive root first detaches with its durable thread and partial evidence; the evaluator immediately performs a full stop before installing holdouts or running validators, so no detached worker can continue mutating the trial workspace. The task fails visibly and the retained session/job evidence remains available for diagnosis.

Positive decomposable tasks use required_agents for the profile's intended MCP contributions. A role pruned from a matched root-only or ablation variant is reported as not applicable; a reachable required role that was never launched fails the diagnostic. Negative controls retain their workload-specific caps and forbidden roles so generated delegation guidance does not turn atomic work into mandatory fan-out.

The finite orchestration fields are required_agents, forbidden_agents, min_peak_mcp_workers, max_jobs, min_result_acceptance_rate, max_contract_failures, and max_observed_mcp_wait_ratio.

Promotion gates

Labs always report experimental_lab; they are never assigned a superiority verdict. They still publish hypothesis_passed and every underlying check, so experimental maturity is not an exemption from measurement.

A featured full profile passes only when all configured checks pass:

• it has nonzero task success;
• its success rate is within two percentage points of the strongest executed root/single-agent/access-product control;
• it improves the declared primary metric over its declared configured-root or strongest-single baseline by at least 10% relative or five absolute percentage points;
• it has no write-scope conflict, contract failure, or incomplete route telemetry;
• every observed API cost category required by policy is complete;
• configured safety/quality metrics do not regress;
• every reachable worker has an ablation and contributes at least two success points or 10% relative improvement in the target metric;
• at least one accessible service-specific control actually executes.

A composition is not promoted because it is cheaper while failing. Conversely, a large council is not promoted merely because it uses more models.

Relative improvement from an exact zero baseline has no finite percentage. Evaluation records relative = null and relative_unbounded = true for a positive improvement from zero, applies the relative gate explicitly, and never writes non-standard JSON Infinity.

Recorded evidence

Run records include:

• final patterns, validation commands, numeric outcome metrics, exit status, and elapsed time;
• exact profile/snapshot/model/route identities and typed route-fault injections;
• worker lineage, task kinds, retries, failures, cancellations, contradictions, and contract results;
• result reads, accept/reject decisions, patch integrations, integration corrections, and taint;
• peak and overlapping MCP execution, first-useful-result timing, explicit wait intervals, and root activity events observed while workers run;
• requested OpenRouter policy, serving provider/endpoint when reported, and route-telemetry completeness;
• token categories, subscription request units, local resource seconds, actual API dollars, and API-equivalent estimates in separate ledgers;
• hidden mutation and holdout hashes.

Observed root activity is event coverage, not mind reading. The runtime can prove that a root emitted non-wait events during worker execution and can measure explicit MCP waits; it cannot prove that every other second was cognitively productive.

Cost ledgers are intentionally not summed into one aggregate score. Subscription units, local resource time, actual API charges, and API-equivalent estimates answer different questions. Missing price categories make the relevant estimate incomplete rather than silently zero.

The access-efficient suite declares exact scarce model keys, counts root and worker requests to those keys, and compares the complete profile with the strongest single-agent control. Its hypothesis requires at least 50% lower scarce-tier use and no more than a two-point success loss. “Scarce” is suite policy for this experiment, not an inferred catalog property.

Evaluation tasks may declare route_faults = { route_id = "credential_loss" }. The only fault values are credential_loss, rate_limit, and timeout; the named route becomes unavailable with that immutable reason for the trial. The retired disabled_routes field is invalid. This is a typed evaluation injection, not an automatic failover or workflow language.

Visual evaluation

visual-conformance uses a real Playwright browser render at multiple widths. The original reference and generated screenshots travel through the same multimodal attachment path used by normal sessions. Validation combines deterministic behavior/accessibility checks with pixel/edge evidence; success cannot be inferred from HTML text or a synthetic image description.

The evaluator rejects a profile binding whose complete transport cannot preserve image inputs and tool-produced image results.

Commands

codex-mmo eval suites
codex-mmo eval validate adaptive-change
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change --dry-run
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change --trial-mode release
codex-mmo eval list
codex-mmo eval show RUN_ID
codex-mmo eval compare RUN_ID_A RUN_ID_B

Invalid suites, failed tasks, and unfinished runs return nonzero status while preserving diagnostic records. A development run may complete while missing promotion; release mode also returns nonzero when an eligible featured profile fails its promotion checks. Inspect promotion.checks instead of equating task completion with profile efficacy.

Interpretation

Provider quota, authentication, route outage, and serving drift are operational failures, not evidence about model reasoning quality. Report them separately. Do not compare unlike suites or repository states, and do not treat a single development run as release evidence. Featured release claims require the five-trial mode, authenticated inventory verification, complete ledgers, and retained task-level artifacts.