😏
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
.coverage
|
||||
dist/
|
||||
build/
|
||||
+332
@@ -0,0 +1,332 @@
|
||||
# Changelog
|
||||
|
||||
## 8.0.0 — 2026-08-21
|
||||
|
||||
### Durable long-running app-server operation
|
||||
|
||||
- Collapse every MMO-owned serialized artifact onto one schema generation derived from the package major (`8`), remove the redundant lifecycle version, and require exact package/profile version 8.0.0. Other generations are rejected at load time; there is no migration, compatibility-inspection, or fallback surface. Upstream Codex, MCP, provider, and Switchyard formats retain their own version identifiers.
|
||||
- Make session and supervised-worker pause cold and capacity-releasing: persist the pause and partial evidence first, then retire fingerprint-matched runner/app-server groups. Continue re-admits the same durable threads and resumes the exact paused worker set; detach remains the explicit live-background mode.
|
||||
- Normalize authoritative failed `turn/completed` records into durable typed failures. Provider usage limits and recoverable transports retain the raw Codex error, retryability, reset evidence, and partial output. An attached TUI keeps its healthy app-server after a retryable failed turn; headless execution or explicit detach cold-suspends the same thread for later resume.
|
||||
- Add exact package/code fingerprints to root and worker state and expose whether a live or retained host matches the installed runtime. Historical evidence must live in an external manual archive and re-enter only through an explicit handoff.
|
||||
- Make `agents_wait` compact and change-aware through per-job `progress_revision` cursors, with result previews disabled by default. Add lossless cursor paging for an individual oversized filtered trace record; durable results and empirical trace evidence no longer require direct supervisor-file reads.
|
||||
- Add `session compact` for paused durable roots. It briefly hosts the exact thread, invokes native app-server compaction, and returns the complete session to a cold pause.
|
||||
- Require every profile to match the exact package version. Identical content is an idempotent no-op; differing current-generation or opaque incompatible installed content requires explicit staged atomic `--replace`, without upgrade/downgrade comparisons or migration.
|
||||
- Update all bundled profiles and the reverse-engineering bundle to version 8.0.0. Reverse engineering reserves capacity for a semantic specialist's nested challenger and requires worker-private IDA copies with hash provenance so IDALib side effects cannot modify the canonical database through a nominally read-only worker.
|
||||
|
||||
### Command-line interaction audit — 2026-08-23
|
||||
|
||||
- Give every command and option useful local help, examples for the primary workflows, conservative typo suggestions, non-abbreviated parsing, and early validation for contradictory or ignored options. The primary and control wrappers now delegate to one Python router instead of maintaining a second command allowlist.
|
||||
- Render structured results for people on interactive terminals and strict JSON in pipes and redirected files, with `--json` as an explicit override. Preserve raw TUI/log/auth streams and concise scalar output, keep identifiers lossless in narrow terminals, and make empty results unambiguous.
|
||||
- Keep requested data on stdout and progress, warnings, usage, and failures on stderr. Add quiet, debug, structured error, signal, broken-pipe, and nonblocking `exec` stdin contracts plus stage-oriented progress for long diagnostics and evaluations.
|
||||
- Document CLI concepts, configuration precedence, lifecycle verbs, exit statuses, paging, and automation behavior; add focused regression coverage for the complete command surface and installer-generated wrappers.
|
||||
|
||||
### External-veracity audit — 2026-08-23
|
||||
|
||||
- Repin the only supported Codex runtime and app-server contract to exactly 0.149.0. Runtime admission, optional installation, generated-schema verification, built-in model evidence, tests, profiles, and documentation now share that one version; 401 generated schema files hash to `fcfeaf23728b96ab73916a21302eb7a16629e67ee99f7ee47b60fad6b6e5ee1a`. No compatibility reader, migration, or alternate-version path was added.
|
||||
- Match the official app-server wire contract by omitting the JSON-RPC version member, validating response envelopes, and retrying only the documented overload error with exponential delay and jitter. Harden the Unix WebSocket client to RFC 6455 handshake, masking, framing, fragmentation, UTF-8, length, close, and protocol-error requirements.
|
||||
- Refresh the exact 0.149 built-in model source and the 2026-08-23 public inventories: 29 OpenCode Go, 64 OpenCode Zen, and 422 OpenRouter records. Preserve tiered, dash-valued, missing-documentation, provider-protocol-disagreement, and listing-only evidence without manufacturing scalar prices or capabilities; the generated catalog contains 560 models, including 462 agent-compatible bindings.
|
||||
- Keep OpenRouter discovery credentials scoped to the reviewed official origin, so explicit blank-key and custom-source diagnostics do not consult stored credentials. Identify catalog discovery with the package version and strengthen source-specific regression coverage.
|
||||
|
||||
### Free-model tool compatibility — 2026-08-23
|
||||
|
||||
- Represent OpenAI Responses custom-tool support independently from ordinary function-tool support. Codex 0.149 can now omit its free-form `apply_patch` tool for a qualified endpoint while preserving function-based shell and MCP tools, instead of incorrectly treating tool calling as an all-or-nothing capability.
|
||||
- Restore Codex 0.149 MCP tools across Switchyard 0.2.0 by exposing Codex-granted MCP schemas as flat app-server dynamic functions and routing calls back through Codex's own allowlist-enforcing MCP API. Use the non-reserved `mmo_mcp__` dynamic prefix after live qualification proved Codex rejects its native `mcp__` prefix for dynamic declarations, and omit only structured result copies that are exactly represented by an existing JSON text block. Pin activation to the gateway's recorded 0.2.0 executable identity; a baseline-change test and documented live-qualification checklist require complete removal when upstream preserves Responses namespace tools natively. Track merged upstream Switchyard PR #384/commit `c7beccd4891fa5cfe3a3b94fdd376f5765864507`; the first published release containing it is the explicit removal trigger.
|
||||
- Teach smoke evidence verification to recognize successful temporary dynamic-tool bridge calls as the required canonical `server.tool`, while ignoring unrelated nested objects whose own `type` member is not a string. Contract-valid worker prose without the required direct calls remains a failed qualification.
|
||||
|
||||
## 7.0.0 — 2026-08-20
|
||||
|
||||
### Reusable active capacity, lossless terminalization, and root-context succession
|
||||
|
||||
- Replace lifetime `max_total_spawns` and per-role `max_spawns` quotas with active-only admission. Terminal jobs release capacity while global, role, child, depth, resource, writer, write-scope, trust, and atomic-batch constraints remain mechanical. Profile schema 5 rejects both retired keys without aliases, migrations, fallbacks, or compatibility readers.
|
||||
- Preserve the canonical worker control-socket identity across suspended continuation and finalization. A resumed worker proves its private socket is bound before model work proceeds; startup failure returns durable suspension instead of an uncaught control-thread `RuntimeError`.
|
||||
- Re-admit both continuation and evidence-only finalization against current scheduler and write leases, reuse the exact app-server thread, and make recovery-control delivery pending until the relaunched worker publishes a usable control socket.
|
||||
- When Codex reports `budgetLimited` or `usageLimited`, allow the already-admitted turn to reach its authoritative `turn/completed` event. Publish a fully contract-valid terminal message as `completed_with_warnings`; otherwise retain partial evidence and permit explicit same-thread finalization without reactivating the goal.
|
||||
- Update all twelve bundled profiles and the external reverse-engineering profile to schema 5/version 7.0.0. Session schema 7/lifecycle 4 adds an ordered root-thread lineage and staged successor transition; job schema 7 and run schema 2 remain unchanged. Earlier session/lifecycle records are rejected without a compatibility reader or automatic migration.
|
||||
- Treat a fresh top-level context observed while a fingerprinted stock TUI is attached as a new canonical root-thread generation inside the same immutable MMO session/run. Native/fork threads are excluded, predecessor IDs still locate the session, and controller recovery consults only the pinned app-server's session-isolated thread index rather than rollout files.
|
||||
- Restore a stopped Switchyard gateway before validating generated homes on resume, recycle the root app-server only when its effective gateway endpoint or compiled route availability changes, and reconcile dead current-run workers to suspended partial evidence before returning control. A replacement turn-mode worker settles an orphaned turn interruption before launching one continuation, preserving a result that completed during the race.
|
||||
|
||||
## 6.0.0 — 2026-08-19
|
||||
|
||||
### Lossless app-server lifecycle and result delivery
|
||||
|
||||
- Keep full root prompts and delegated task briefs in their initial turns while bounding only the Codex goal objective to 4,000 characters. Long objectives retain a deterministic prefix and SHA-256 correlation note; an oversized goal label can no longer discard task context or leave an empty allocated thread falsely recoverable.
|
||||
- Seed interactive goal sessions with a persistent paused objective before the TUI attaches, activate it only after the first accepted turn, and map active, paused/blocked, usage-limited, detached, recovering, and terminal states without conflating them. Controller and worker replacement preserve the original start timestamp, reuse the exact thread, and emit distinct resume audit events.
|
||||
- Derive root terminal output from the last completed user-presentable `plan` or `agentMessage` item for the exact completed turn. A stale turn summary or older turn can no longer replace a later plan, and failed startup without durable work no longer manufactures partial evidence.
|
||||
- Replace head/tail result truncation with contiguous cursor pagination. `agent_result` returns content once per page with `total_chars` and `next_cursor`; `agents_wait` returns a bounded preview and instructs callers to consume all pages. Model-visible MCP responses omit supervisor result/event/stderr/socket metadata without rewriting same-named fields inside opaque model results, and compiled guidance prohibits bypassing the lifecycle through direct state-file reads.
|
||||
- Verify historical snapshots against the exact guidance bytes already bound into their content address instead of regenerating prose with the currently installed compiler. Guidance evolution across an MMO upgrade no longer makes an intact immutable session appear corrupt; non-guidance payloads, manifest semantics, and stored guidance remain cryptographically checked without rewriting or migrating the snapshot.
|
||||
- Advertise one stable object-root spawn schema for each caller and enforce the selected role's exact task, literal-task, permission, and length rules server-side. Keep both single and atomic batch spawn tools without top-level schema unions that fragile clients mis-handle.
|
||||
- Apply the lifecycle guidance and release version to all twelve bundled profiles and the reverse-engineering bundle. This is a breaking in-place release with no migration, compatibility, alias, fallback, or alternate-host surface.
|
||||
|
||||
## 5.0.0 — 2026-08-18
|
||||
|
||||
### Canonical Unix app-server session lifecycle
|
||||
|
||||
- Make one isolated `codex app-server --listen unix://…` host the only execution host for every root and supervised MCP worker; native agents remain durable child threads in their root host. Interactive Codex is now a detachable `--remote` client of that same root host and thread; startup, reattach, controller replacement, and transport recovery never create a replacement MMO run.
|
||||
- Replace profile schema 3 with schema 4, session schema 5 with schema 6, job schema 5 with schema 7, and lifecycle 2 with lifecycle 3. Job schema 7 pins both app-server and control sockets to stable hashed private runtime paths, avoiding AF_UNIX path loss under long state roots. Reject the retired active-work policy, rollout-path, and flat `can_control` fields. There is no compatibility reader, migration, alias, fallback host, or automatic rewrite.
|
||||
- Replace model-blind wall clocks with Codex goal/turn lifecycle. Goal roles have host-observed token budgets and a compiled maximum; turn roles have no task deadline. Stall intervals warn without interrupting, provider slowness never erases evidence, and only external smoke/evaluation harnesses retain explicit wall limits.
|
||||
- Add lossless pause, detach, continue, full stop, interrupt, finalization, compaction, steering, pending-request response, effort change, trace, inspection, and fork controls for root, native, and MCP runs through opaque run references and action-specific control grants. Full stop retires the host; detach leaves it working; interruption affects only the current turn.
|
||||
- Persist authoritative app-server event streams and terminal thread history independently of controller processes. Recovery resumes the exact recorded thread, consumes a turn already committed before transport loss, and retains partial results, patches, server requests, goal state, and terminal evidence on every non-success path.
|
||||
- Pin runtime admission to the exact reported Codex 0.147.0 release and validate the reviewed message/response shapes in-process. `doctor --live` performs the complete 361-file generated-schema digest check (`a29c348318b077f122e8e271340f25dfc0b19d74f77e452b27f539fbf5444221`). Server requests use their exact method-specific response contracts; unimplemented requests fail explicitly.
|
||||
- Compile the complete lifecycle and control surface into immutable per-role `AGENTS.md` and `mmo-profile-orchestration/SKILL.md` guidance. Update all twelve bundled profiles and the external reverse-engineering profile in place, including bounded goal budgets, warning-only stall policy, exact action grants, and autonomous app-server operation.
|
||||
- Teach generated goal-role guidance and bounded doctor/smoke prompts to call `update_goal(status="complete")` only after their required evidence is ready. A terminal message no longer leaves a harness goal auto-continuing until its token budget merely because the probe prohibited the lifecycle tool. Root and worker hosts now wait for the authoritative terminal `turn/completed` after a goal-completion event, so an in-flight final message cannot be discarded; the doctor also requires the exact probe marker rather than a substring.
|
||||
- Preserve one protected session capability set across detach/restart and destroy it when the immutable session becomes terminal. Terminal transcripts and evidence remain readable; terminal sessions cannot be resumed or regain control authority.
|
||||
- Refresh the OpenCode Go inventory to its 2026-08-19 public 28-model listing, including a fully evidenced Muse Spark 1.2 Responses binding and a non-agent-compatible contributor ID, while retaining the changed DeepSeek Flash tier evidence exactly. The generated catalog now contains 548 models, of which 452 are agent compatible; unresolved live Go, Z.AI-document, and OpenRouter endpoint-health evidence remains explicit rather than triggering a route fallback.
|
||||
|
||||
## 4.0.0 — 2026-08-17
|
||||
|
||||
### Breaking durable app-server execution cutover
|
||||
|
||||
- Replace every Agent-MCP `codex exec` child with a privately hosted Codex app-server thread. Persist exact thread and rollout identity, a complete JSON-RPC event trace, terminal output, bounded partial evidence, and isolated writable patches independently of the disposable transport process.
|
||||
- Replace profile and smoke schema 2 with schema 3, session/job schema 4 with schema 5, and the prior session lifecycle with lifecycle 2. Old execution-timeout/progress fields and smoke `timeout_seconds` are rejected; there is no reader, migration, alias, fallback key, or automatic state rewrite.
|
||||
- Replace model-timed progress/watchdog policy with supervisor-measured `renewable_active_work` and explicit `hard_wall` policy. Pending input and deliberate pauses do not consume active-work capacity; slow provider/model execution does. Renewable allowances grow host-side to a compiled maximum, then reserve a bounded evidence-only finalization grace.
|
||||
- Add policy-bounded cross-agent app-server control: inspect, filtered durable trace, steer, interrupt, same-thread continue, evidence-only finalize, native compaction, pending-input/MCP-elicitation/approval response, reasoning-effort change, active-work-cap change, and durable thread fork. Every mutation uses a compare-and-swap revision, exact `can_control` edges, session/run identity, and auditable delivery status.
|
||||
- Keep strict JSON contracts on the same agent thread. Send a projected strict `outputSchema` through app-server when the route supports structured output, validate the full original schema independently, and permit one same-thread shape-only repair turn. No JSON-fixer role can manufacture missing evidence.
|
||||
- Make noninteractive roots persistent app-server threads too, with the canonical v5 state value `session_kind = "noninteractive"`. External `--wall-timeout` now detaches rather than deletes work, pending input detaches for interactive resumption, and profile active-work/finalization policy is enforced independently of that harness wall limit.
|
||||
- Record each run's actual root execution host and whether MMO is enforcing its app-server execution policy, so a noninteractive session resumed through the native TUI does not misreport the lifecycle controlling that run.
|
||||
- Add `session detach` (root disconnect while independently hosted workers continue), `session stop --grace` (full evidence-preserving finalization followed by bounded termination), and retain `session cancel` as immediate cancellation. A disappeared worker host becomes `suspended`, never `lost`; its thread, trace, partial result, and patch remain available for authorized continuation.
|
||||
- Publish app-server PID/process-group ownership before protocol bootstrap, retain root ownership until detach termination succeeds, fingerprint it against PID reuse, retire orphan descendants even after their recorded leader exits, and rerun full admission before hosting a suspended continuation.
|
||||
- Compile lifecycle/control guidance into every immutable profile snapshot, update all twelve bundled profiles and the reverse-engineering profile with realistic long-running limits and explicit control graphs, and update smoke/evaluation harnesses to use `wall_timeout_seconds` only at their external boundary.
|
||||
- Require the locally installed Codex app-server v2 thread, turn, control, pending-input, elicitation, and approval method/parameter/response contracts during doctor validation. Thread lifecycle requests allow slow external MCP initialization without using model-authored clock instructions.
|
||||
|
||||
## 3.0.1 — 2026-08-16
|
||||
|
||||
### Breaking route/catalog/runtime/profile v2 cutover
|
||||
|
||||
- Compile exact profile-derived `AGENTS.md` and `mmo-profile-orchestration/SKILL.md` guidance into immutable snapshot identity, enable the skill only for spawn-or-control coordination roles through standard Codex skill discovery, require an early three-call delegation decision, and make positive evaluation tasks measure whether their intended MCP roles participated. Legacy snapshots without canonical guidance are rejected rather than migrated.
|
||||
- Replace the provider-shaped catalog with schema-v2 route identity: maker, API operator, access product, wire protocol, billing mode, endpoint, credential contract, gateway, and reported serving provider remain distinct. Replace every binding with an exact route-qualified key and hard-reject v1 keys and schemas.
|
||||
- Make the cutover cleanly breaking. Remove retired profile/evaluation IDs and empty directories; add no migration command, compatibility reader, lookup alias, automatic rewrite, fallback key, or compatibility profile. Legacy durable session/job state is rejected in place without mutation.
|
||||
- Put OpenRouter, OpenCode Go/Zen, Z.AI General API, Z.AI Coding Plan, Codex, and local deployments through one declarative inventory-snapshot and full-record-fingerprint pipeline. The 2026-08-16 baseline contains 21 routes, 546 models, 451 agent-compatible models, and 9 resource groups.
|
||||
- Make OpenCode Go transport provenance explicit: the provider endpoint table is route authority, current Models.dev disagreements for four Qwen records are retained in fingerprint evidence, and live verification reads current docs instead of mistaking an immutable reviewed capture for a current source.
|
||||
- Add typed OpenRouter route policies and serving telemetry. Bundled OpenRouter roles pin an allowlist and disable fallback; records distinguish the requested aggregator route from the downstream serving provider actually reported.
|
||||
- Refresh the complete OpenRouter snapshot after the live DeepSeek Flash Latest price changed, verify both selected endpoints as healthy, and normalize OpenRouter's `~maker/...` rolling IDs to the actual maker without turning them into lookup aliases.
|
||||
- Bind every bundled GLM-5.3 role exactly to `zai_coding_responses__glm_5_3` with `ZAI_CODING_API_KEY`. The separate Z.AI General API remains available only as an explicit route and is never a fallback for Coding Plan roles.
|
||||
- Add route-level startup availability. An unavailable root blocks session creation; unavailable optional workers remain explicit typed admission failures without creating jobs or silently substituting a model.
|
||||
- Add lifetime `max_spawns`, structured `literal_task` admission for low-trust roles, monotonic result read/accept/reject/integrate states, isolated Git worktrees, binary-safe patch/artifact capture, taint, command/artifact correlation, and root-controlled canonical integration.
|
||||
- Split token/cost telemetry into actual API dollars, API-equivalent estimates, subscription request units, and local resource seconds. Record root activity events during worker execution without mislabelling unobserved time as productive cognition.
|
||||
- Replace the previous portfolio with eight featured profiles—`adaptive-engineering`, `codex-harness-team`, `high-confidence-debugging`, `research-backed-engineering`, `secure-change`, `visual-engineering`, `contract-first-refactoring`, and `incident-hypothesis-triage`—plus four experimental labs: `access-efficient-escalation-lab`, `route-resilience-lab`, `bounded-research-organization-lab`, and `competing-implementations-lab`.
|
||||
- Balance maker and access diversity intentionally: ChatGPT-backed Codex, DeepSeek/MiniMax/Kimi through economical Go access, Claude/MiniMax through Zen, GLM-5.3 through the Z.AI Coding Plan, pinned Nemotron/DeepSeek through OpenRouter, and one mechanically contained local Qwen role.
|
||||
- Replace topology-coupled evaluation with schema v2: topology-neutral outcomes, optional orchestration diagnostics, matched root/single/access/full variants, complete worker ablations, deterministic hidden mutations, holdouts, negative controls, three development trials, five CLI-selectable release trials, explicit promotion gates, and no aggregate cost/quality score.
|
||||
- Use real Playwright screenshots and image-preserving worker transport for visual evaluation. Add workload-specific suites for all twelve profiles and retain labs as experiments rather than unearned flagship claims.
|
||||
|
||||
### Persistent interactive session lifecycle — 2026-08-16
|
||||
|
||||
- Add `codex-mmo resume` as the only MMO resume route. A logical interactive session retains its immutable snapshot, generated Codex home, working-directory boundary, pinned executable, and exact root Codex thread; retained pre-lifecycle sessions are not adopted or migrated.
|
||||
- Separate the logical session from bounded execution runs. Each explicit resume creates a durable run record, rotates Agent-MCP capabilities, revalidates routes and generated model-catalog hashes, rejects stale worker and launcher identities, and replenishes the documented per-run spawn budgets.
|
||||
- Preserve one-shot `exec` behavior, make cleanup age persistent sessions from their latest run, and expose run history and job filtering through the normal session/job CLI surfaces.
|
||||
|
||||
### Post-cutover implementation audit
|
||||
|
||||
- Correct the Z.AI Coding Plan Responses client to the provider's dedicated Codex base `https://api.z.ai/api/v1`; keep the Chat-compatible `/api/coding/paas/v4` route separate and add no route fallback or compatibility shim.
|
||||
- Replace permissive JSON Schema format checks with dependency-free RFC 3339 `date`/`date-time` and RFC 3986 absolute-URI validation, including leap seconds, lowercase `t`/`z`, strict offsets, percent encodings, IP literals, and case-insensitive HTTP header-name collision rejection.
|
||||
- Refresh the unchanged 26-ID Go, 62-ID Zen, and 413-ID OpenRouter inventories from current primary sources. Preserve OpenCode's DeepSeek peak/off-peak rows without flattening them into false scalar prices, and update current Qwen, Nemotron, and rolling DeepSeek OpenRouter fingerprints.
|
||||
- Remove the last catalog model-alias field and the unused general Z.AI URL shortcut from generators, snapshots, CLI parsing, and tests. Both are rejected rather than migrated; exact route-qualified keys and explicit discovery commands are the only v2 surfaces.
|
||||
- Validate the complete merged catalog, including unused profile-local rows and resource references, before resolving bindings or generating catalog output. Reject non-finite OpenRouter price policy values.
|
||||
- Derive profile and evaluation-ablation concurrency from one exact weighted-resource calculation covering root reservations and the lifetime spawn budget, and reject any role whose requested units exceed its resource capacity.
|
||||
- Share Switchyard only when deterministic transport/model route semantics match, keep injected faults session-local, validate persisted loopback endpoints before network use, and terminate and reap superseded or partially initialized gateway processes.
|
||||
- Bind durable session and job identity to their containing directory, fail strict accounting closed on invalid state, expose the complete result disposition/integration lifecycle in generated Agent MCP configuration, and correct MCP mutability annotations.
|
||||
- Attempt to roll back an applied worker patch if publishing its integrated lifecycle state fails; if reverse application also fails, persist a session taint when possible and report explicitly when even that durable containment marker cannot be written.
|
||||
- Add opt-in bounded Agent-MCP progress checkpoints with a monotonic watchdog when a job timeout exceeds its configured checkpoint interval, pass compatible strict worker contracts through Codex's output-schema interface where the selected route supports it, and forward authenticated job identity into a delegated worker's own `mmo_mesh` server.
|
||||
- Let smoke tasks require successful direct `server.tool` calls from Codex event evidence, so contract-valid blocked prose cannot pass a live integration check. Limit staged core-install validation to bundled profiles so unrelated operator profile packs remain preserved and cannot break a runtime upgrade.
|
||||
|
||||
### Comprehensive quality audit follow-up — 2026-08-15
|
||||
|
||||
- Enforce per-role MCP concurrency across sessions of the same stable profile, prevent root exec from raising a read-only profile root to workspace-write, and pin every evaluation task to the one immutable snapshot validated at run creation.
|
||||
- Terminate a launched worker child when runner setup or state publication fails, retire residual worker process-group descendants after normal completion, skip malformed non-object records during non-destructive state enumeration, fail admission accounting and idle-gateway cleanup closed when durable ownership is unreadable, and return explicit object-root errors when a selected runtime state document is corrupt.
|
||||
- Require complete inventory discovery metadata, HTTP(S)-only discovery URLs, a live Codex source before installing an overlay, well-formed discovery reasoning/modalities, and model input modalities that the selected provider transport can carry.
|
||||
- Replace OpenCode Go's inferred blanket `high` reasoning setting with exact Models.dev effort selectors; omit the parameter for routes that expose only toggle/budget controls or no verified selector, while retaining the supported `high` bindings for DeepSeek V4 Flash.
|
||||
- Reject NUL-bearing Tool MCP arguments and duplicate runtime bindings, report bound models consistently during profile validation, dispatch `tool-mcp` through the primary installed wrapper, and require the Tool MCP module explicitly in release archives.
|
||||
- Consolidate HTTP header-value validation at the shared utility boundary, use canonical plural credential configuration in provider documentation, and add focused regressions for each corrected contract.
|
||||
|
||||
### Independent adversarial verification follow-up — 2026-08-15
|
||||
|
||||
- Make pre-spawn interactive setup failures terminalize their already-created session and discard its root MCP capability; make blank process credentials fall back consistently to a configured nonblank credential; validate cancellation reasons instead of truncating audit and state differently.
|
||||
- Keep diagnostic records for an atomic batch whose runners could not all launch without charging that rejected batch to the lifetime spawn budget.
|
||||
- Make generated evaluation run IDs loadable at the maximum valid profile/suite ID lengths, reject blank pattern/command entries, and align the CLI result-length boundary with the runtime/MCP contract.
|
||||
- Remove the retired OpenCode credential-name rewrite entirely; installation preserves credential files byte-for-byte and obsolete names remain invalid. Use a unique validation-state directory rather than deleting a fixed name.
|
||||
- Refuse to let retained uninstall ownership markers authorize a recreated install path, reject symlink/special release publication targets, and report successful non-check catalog regeneration truthfully.
|
||||
|
||||
### Profile-scoped Tool MCP support — 2026-08-15
|
||||
|
||||
- Add an operator-owned, lexically layered Tool MCP registry for reviewed STDIO and Streamable HTTP servers, with closed transport-specific schemas, finite tool allowlists, explicit approval policy, environment-backed authentication, and no profile-supplied executable, endpoint, or OAuth state.
|
||||
- Add per-agent grants that can only narrow operator tool exposure, include normalized definitions and grants in immutable snapshots, and compile an explicit enable/disable matrix that remains safe across native-agent config merges while coexisting with the reserved internal `mmo_mesh` Agent MCP server.
|
||||
- Forward only selected Tool MCP environment variables into generated Codex processes, warn when native roles share credential scope, add non-launching `tool-mcp list/show/validate` diagnostics and doctor gates, and document the external-server sandbox and prompt-injection boundary.
|
||||
|
||||
### OpenRouter, OpenCode Zen, and unified inventory snapshots — 2026-08-15
|
||||
|
||||
- Add all 62 currently live OpenCode Zen IDs by joining the public listing to OpenCode-maintained Models.dev metadata and commit-pinned provider documentation. Route 24 through Responses, 19 through Chat Completions, and 13 through Anthropic Messages with the shared `OPENCODE_API_KEY`; retain 6 Google-native entries as catalog-only until a faithful installed transport exists; label 9 still-listed but officially deprecated IDs with their effective dates.
|
||||
- Preserve exact Zen reasoning controls and structured-output claims, take scalar billing rates from the provider's own table, omit unrepresentable budget-only efforts and context-tiered prices, expose executable translated routes as text-only, and add public discovery/drift comparison plus snapshot reconstruction and route-contract coverage. Normalize away the live endpoint's request-time `created` field while retaining its raw capture hash, and confirm all three executable protocols with a real Switchyard 0.2.0/local-mock transport probe.
|
||||
- Add an executable OpenRouter Switchyard provider using `https://openrouter.ai/api/v1`, OpenAI Chat format, and `OPENROUTER_API_KEY`.
|
||||
- Populate a reviewed 2026-08-15 OpenRouter baseline with all 413 text-input/text-output IDs from the official Models API, including exact supported-parameter, reasoning, expiry, alias, modality, and pricing evidence.
|
||||
- Expose only evidence-backed OpenRouter capabilities: 346 tool-capable entries, 335 structured-output entries, 6 parallel-tool-call entries, and exact advertised reasoning efforts. Keep the executable route text-only until translated multimodal fidelity is verified.
|
||||
- Omit misleading scalar OpenRouter rates for dynamic, tiered, fixed, or differently priced internal-reasoning cases while retaining the upstream pricing evidence and provenance.
|
||||
- Add public/optional-auth OpenRouter discovery, remote drift comparison, CLI endpoint overrides, and secret-leakage regression coverage.
|
||||
- Replace the size-dependent embedded inventory tables with one versioned, canonical-hash-checked `config/inventory-snapshots/` envelope for OpenRouter, OpenCode Zen/Go, Z.AI API, Z.AI Coding Plan, and Codex. Catalog generation remains deterministic and offline for every inventory size.
|
||||
- Replace the OpenCode credential contract with the shared `OPENCODE_API_KEY` and update all generated routes and examples. The installer does not recognize or rewrite `OPENCODE_GO_API_KEY`.
|
||||
- Derive aggregate inventory metadata from the common snapshots, verify exact normalized snapshot-to-catalog records at runtime, require complete OpenRouter pagination metadata, and prevent discovery URL overrides from receiving stored OpenRouter credentials.
|
||||
- Replace every model key with the exact route-qualified form `<route>__<model>`, update every bundled profile directly, and reject mismatched namespaces or duplicate route/upstream bindings. Remove the duplicate OpenCode Go DeepSeek V4 Flash and Z.AI Coding Plan GLM-5.3 rows rather than retaining aliases.
|
||||
- Complete the key cutover by removing catalog lookup's case-insensitive raw upstream-ID/search-alias fallback; `catalog model` now accepts only an exact provider-qualified key, including when an upstream ID is currently unique.
|
||||
- Refresh the official 413-row OpenRouter capture after Kimi K2.6 pricing and Qwen3.6 27B pricing/output-limit changes; the advertised ID set remains unchanged.
|
||||
|
||||
### Runtime quality and security remediation — 2026-08-15
|
||||
|
||||
- Serialize stale session/job reconciliation with normal state transitions and re-read state under the runtime lock before publishing recovery results.
|
||||
- Close descendant admission atomically when cancellation begins, and enforce absolute write-scope leases across active sessions even when their workspace roots are nested.
|
||||
- Authenticate root, job, and native MCP callers with distinct capabilities tied to durable lineage; persist only root/job digests, keep the root plaintext process-local, discard it at terminal session state, and confine native-role plaintext to its protected generated configuration.
|
||||
- Forward the caller capability through Codex's declared MCP environment-variable allowlist without writing it into generated TOML, and validate every tool call against its advertised input schema.
|
||||
- Reject cross-origin catalog-discovery redirects before authorization can be forwarded, and restrict implicit OpenRouter/Z.AI credentials to their reviewed official endpoints.
|
||||
- Reject pre-v2 gateway state in place; never validate, rekey, rewrite, or otherwise mutate it into a current state document.
|
||||
- Return structured worker results only after a successfully enforced contract, validate smoke tasks against the same task/backend/permission boundaries as runtime admission, and reject contradictory shared resource capacities or transport capabilities.
|
||||
- Remove correctness-critical `assert` statements from optimized-Python execution paths, harden profile cloning across valid TOML quoting styles, and make per-native-role network policy explicit in generated configuration.
|
||||
- Add regression coverage for state races, cancellation/admission races, nested workspaces, capability forgery and lifetime, redirect credential leakage, v1 gateway-state rejection, result contracts, resource definitions, and generated MCP/native configuration.
|
||||
|
||||
### External contract audit — 2026-08-15
|
||||
|
||||
- Enforce the MCP 2025-06-18 initialization lifecycle, supported-version negotiation (including the doctor's two-phase probe), string/integer request IDs, object-only parameters, strict finite JSON, and pre-initialization tool rejection; return JSON-RPC invalid-params errors for unknown tools/schema-invalid arguments and reserve `isError` results for execution failures.
|
||||
- Follow XDG Base Directory 0.8 handling for empty/relative environment values and document `XDG_BIN_HOME` as a project extension.
|
||||
- Parse Linux `/proc/<pid>/stat` without misreading command names containing spaces or closing parentheses.
|
||||
- Normalize provider header aliases to the exact Codex (`http_headers`, `env_http_headers`) and Switchyard 0.2.0 (`extra_headers`) schemas, reject the complete downstream-invalid control-character set, preserve configured Codex environment-backed header secrets, and reject unsupported environment-backed Switchyard headers.
|
||||
- Reject malformed or suffix-unsafe provider base URLs before Codex/Switchyard endpoint construction, while accepting the HTTP-standard case-insensitive scheme.
|
||||
- Validate direct Codex retry and idle-timeout settings as non-negative TOML integers before materialization instead of coercing incompatible catalog types late.
|
||||
- Document Switchyard retries as additional attempts, including MMO's explicit default of one versus Switchyard 0.2.0's omitted-field default of two and its exact maximum of ten.
|
||||
- Validate the exact Switchyard IP-address host type before launch and restrict its unauthenticated managed gateway ingress to IPv4/IPv6 loopback literals.
|
||||
- Include context-window, tool-calling, and reasoning route semantics in the reusable Switchyard gateway identity so distinct route configurations cannot collide.
|
||||
- Require live Switchyard route discovery to match the snapshot exactly, reporting both missing and unexpected advertisements.
|
||||
- Align root/worker usage extraction with Codex 0.147's cache-write and reasoning-output JSONL fields; retain usage from failed workers, calculate cached tokens at their own rate, and include root usage in total cost estimates.
|
||||
- Price cache reads, cache writes, and base input as distinct token categories; omit misleading scalar prices for context-tiered OpenCode Go models and make estimates unknown when an observed category lacks a rate.
|
||||
- Stop defaulting OpenCode Go structured-output support to true: assert it only for the nine IDs explicitly marked by the OpenCode-maintained Models.dev snapshot, and document the conservative text-only translated-route boundary for upstream multimodal models.
|
||||
- Gate doctor credential success for built-in models on both Codex's actual `login status` result and file-backed auth transferable into the generated home; report home-scoped keyring-only login as unsupported.
|
||||
- Fail an explicitly requested Codex catalog verification when no installed-client source runs; continue treating account-gated omissions from a working client as non-fatal.
|
||||
- Defer built-in context/summary capabilities to the active Codex catalog, align the static 0.147 baseline reasoning/context metadata, and distinguish public, deprecated, and internal discovered rows.
|
||||
- Record split OpenCode Go provenance for live availability, provider pricing/protocol documentation, and OpenCode-maintained Models.dev capabilities; mark `hy3-preview` limits unverified.
|
||||
- Split Z.AI provenance by availability, Codex integration, core parameters, and pricing; correct the GLM-5.3 slug, `low`/`high`/`max` effort set, Coding Plan alias behavior, older-model effort support, and documented output limits.
|
||||
- Mark the Z.AI Coding Plan Anthropic-compatible route catalog-only because its documented Bearer authentication cannot be represented by Switchyard 0.2.0's fixed Anthropic `x-api-key` behavior; Responses and Chat paths remain executable.
|
||||
- Parse and emit RFC 8259 finite JSON, TOML 1.0 basic strings (including U+007F), Draft 2020-12 boolean/integer schema semantics, and RFC 9110 2xx-only readiness behavior.
|
||||
- Keep the truncation marker inside, rather than beyond, the configured hard result-character ceiling.
|
||||
- Verify exact archive file and directory permission modes and enforce member-count and expanded-size limits progressively while streaming tar archives and profile packs.
|
||||
- Require release archives to contain exactly the explicit parent-directory entries implied by their payload, rejecting both implicit missing entries and extra empty directories.
|
||||
- Filter optional npm/Cargo installer subprocess environments and require the requested Codex or Switchyard executable to exist after a successful package-manager exit.
|
||||
- Document Switchyard 0.2.0's declared Rust 1.96.1 minimum for the optional Cargo installation path and replace its broken registry-page citation with the versioned published-package documentation.
|
||||
- Audit both sides of staged and committed Git renames with `--no-renames`/NUL-delimited paths, include permission-only changes to pre-dirty files, and preserve legal whitespace in `git rev-parse --show-toplevel` output.
|
||||
- Decode and re-encode Git's verbatim NUL-delimited pathname bytes with Python's filesystem codec and `surrogateescape`, rather than assuming UTF-8 independently of the runtime locale.
|
||||
- Correct the single-model profile guide and document the deliberately limited JSON Schema contract language and Python/ECMA-portable regex boundary.
|
||||
- Describe the control plane as free of third-party Python packages rather than free of all runtime dependencies, and list Bash as an installer/wrapper requirement.
|
||||
- Distinguish Qwen3.5-9B's native 262K multimodal capability from the intentional text-only 32K/8K local scout cap, and pin matching current llama.cpp example flags.
|
||||
- Terminate Codex option parsing before stdin prompts so its variadic `--image` flag cannot consume `-`, and pin optional Switchyard installation to the verified 0.2.0 schema target.
|
||||
- Correct troubleshooting to use the package's `codex-mmo version` control command; `codex-mmo --version` is intentionally passed through to the underlying Codex CLI.
|
||||
- Correct the interactive image example to put `--` before Codex-owned options, keeping them unambiguous from orchestrator `run` options.
|
||||
- Correct the profile smoke-task example to use the implemented `task`, `task_kind`, and `mode` fields and document its actual pass criteria.
|
||||
- Resolve and pin one Codex executable per session so detached workers honor `settings.toml` and cannot diverge from the root after PATH or configuration changes.
|
||||
- Make the global built-in auth link mode effective, force file storage in generated homes, and stop propagating Codex's unrelated `.credentials.json` MCP OAuth state.
|
||||
- Return nonzero CLI status for invalid validation, failed evaluation tasks, and unfinished waits; reject a model probe without live mode and honor configured/explicit catalog discovery sources without silent option loss.
|
||||
- Expose generated homes and pinned runtime details from `session show`, and document the name-based environment-filter boundary without claiming arbitrary secret discovery.
|
||||
- Reject missing, linked, or special source-install payload members and require manifest-bound ownership for uninstall/purge targets, including retained-state manifests for later purges.
|
||||
- Close schema-v1 settings, profile, catalog, agent, resource, and smoke field sets so typos and missing explicitly named files fail validation instead of becoming inert configuration.
|
||||
- Remove config/model-catalog keys absent from Codex 0.147's exact `ConfigToml`/`ModelInfo`/`ModelMessages` schemas; retain no MMO compatibility fields.
|
||||
- Admit Codex 0.147's `indexed` web-search mode and reject arbitrary approval-policy strings outside its canonical scalar policy set.
|
||||
- Reject unknown fields in schema-v1 evaluation suites and tasks rather than silently ignoring misspelled execution or validation controls.
|
||||
- Preserve and validate Switchyard 0.2.0's per-target `extra_body` map in the now-closed model schema, including finite JSON/TOML value checks.
|
||||
- Enforce provider fields per selected driver so a recognized but inapplicable transport/authentication key cannot be silently retained and ignored.
|
||||
- Prevent `profile.smoke` paths from escaping the profile pack, require non-empty relative catalog/smoke paths, and enforce the documented required profile metadata.
|
||||
- Refuse installer upgrades over an unrelated nonempty install root unless its version-3 ownership manifest matches the canonical target.
|
||||
- Enforce schema-v1 scalar/container types without Python boolean/integer equivalence or string coercion, constrain role instruction/contract members to their documented pack directories, require an absolute expanded base Codex home, and validate exact Codex 0.147 built-in provider/auth pairs before generation.
|
||||
- Make evaluation validation reject boolean schema versions, coerced task IDs, nonrelative fixtures, and nested fixture links/special files before a dry run can report success.
|
||||
- Reject duplicate JSON object members, lone surrogate code points, and boolean-as-integer manifest schema versions across runtime state, MCP input, output contracts, ownership records, and release verification.
|
||||
- Correct the documented active-agent count to include the root and the derived native subagent ceiling to reserve that root slot; describe the exact Codex/Switchyard endpoint normalization behavior rather than generic string concatenation.
|
||||
|
||||
### Interactive terminal fidelity
|
||||
|
||||
- Run interactive Codex as a real foreground terminal process group instead of detaching it into a new session.
|
||||
- Translate Python's negative signal return codes to the shell's `128 + signal` convention at CLI and worker-runner exit boundaries.
|
||||
- Preserve the user's terminal capability environment and stop forcing `NO_COLOR` for interactive runs.
|
||||
- Restore terminal foreground ownership and attributes on normal exit, errors, and shell job-control transitions.
|
||||
- Treat terminal stdin/stdout as interactive even when stderr is intentionally redirected to a log.
|
||||
- Add a PTY regression test that verifies redirected stderr does not detach the TUI, ANSI color bytes pass through unchanged, and `SIGWINCH` delivers a live 80x24 to 132x42 resize.
|
||||
|
||||
### Model metadata
|
||||
|
||||
- Generate an exact per-process Codex startup model catalog for every external or translated model alias.
|
||||
- Preserve the active Codex binary's bundled catalog when built-in and external models share one native-agent process.
|
||||
- Cache bundled catalog discovery by Codex binary fingerprint and perform discovery outside the runtime admission lock.
|
||||
- Generate schema-valid freeform `apply_patch` metadata for tool-capable external route rows and omit patch-tool metadata for models that do not support tools.
|
||||
- Add regression coverage for GLM-5.3, DeepSeek V4 Flash, Qwen3.5-9B, mixed native/external catalogs, reasoning levels, context windows, and exact generated slugs.
|
||||
|
||||
### Lifecycle
|
||||
|
||||
- Record whether the root owns an isolated process group and use single-process termination when a group cannot safely be targeted.
|
||||
- Preserve cancellation authority during the narrow process-launch/session-publication race.
|
||||
- Match persisted process IDs with Linux start-time fingerprints before acting on them, and retire residual root descendants after normal completion as well as cancellation.
|
||||
|
||||
### Correctness hardening
|
||||
|
||||
- Serialize concurrent audit-log appends across short-write retries and preserve cancellation diagnostics at terminal-state races.
|
||||
- Audit paths committed by a worker even when its worktree is clean at completion.
|
||||
- Enforce strict finite JSON output, JSON Schema numeric equality, and arbitrarily large finite integer handling.
|
||||
- Publish archives and their validation/integrity metadata as one verified rollback-safe release set.
|
||||
|
||||
## 3.0.0 — 2026-08-14
|
||||
|
||||
### Architecture
|
||||
|
||||
- Replaced the fixed GLM/DeepSeek/Qwen runtime with a provider- and model-neutral profile compiler.
|
||||
- Added immutable, content-addressed profile snapshots with cryptographically verified payload manifests so active sessions cannot be changed by profile switching or unnoticed on-disk mutation.
|
||||
- Added generic provider drivers for built-in Codex/ChatGPT, Codex custom Responses endpoints, Switchyard, Ollama, LM Studio, and catalog-only services.
|
||||
- Added per-snapshot Switchyard gateways with deterministic route generation, credential isolation, reuse, idle shutdown, and routing logs.
|
||||
|
||||
### Orchestration
|
||||
|
||||
- Added first-class Codex native subagent, Agent MCP, and hybrid orchestration modes.
|
||||
- Added bounded nested delegation, caller authentication, directed spawn graphs, depth limits, per-agent child limits, session spawn budgets, ancestor-role rejection, result visibility, and cancellation cascades.
|
||||
- Added atomic asynchronous batch spawning with whole-batch capacity reservation and rollback if any runner fails to launch, plus a root productivity contract that keeps the critical path active while participants run.
|
||||
- Added native-agent role generation and authenticated native-to-MCP delegation for enforceable nested hybrid meshes.
|
||||
|
||||
### Safety and correctness
|
||||
|
||||
- Added weighted resource groups and global capacity admission.
|
||||
- Added disjoint MCP write-scope leasing and completed-job Git audits, including detection of modifications to files that were already dirty before a job started.
|
||||
- Added static profile-pack validation, safe archive extraction, no-symlink policy, and atomic install/replace/remove/clone operations.
|
||||
- Added low-trust mechanical constraints: MCP-only, read-only, mandatory verification, strict structured output, and bounded task categories.
|
||||
- Added multimodal model/provider transport validation to reject lossy role bindings at compile time.
|
||||
- Added isolated worker homes, credential filtering, bounded execution, process-group cancellation, durable state, and descendant termination when a root exits.
|
||||
|
||||
### Catalog
|
||||
|
||||
- Added the complete release-time OpenCode Go inventory.
|
||||
- Added the complete release-time Z.AI API and Z.AI Coding Plan inventories, including catalog-only media and hosted-agent services.
|
||||
- Added all officially documented models available through Codex at release time.
|
||||
- Added live OpenCode Go, authenticated Z.AI, and installed/authenticated Codex discovery and verification.
|
||||
- Added conservative authenticated Codex discovery overlays without guessing capabilities for unknown external provider IDs.
|
||||
|
||||
### Profiles
|
||||
|
||||
- Added `glm-deepseek-qwen`.
|
||||
- Added `codex-glm-deepseek`.
|
||||
- Added `three-expert-council`.
|
||||
- Added `actor-critic`.
|
||||
- Added `proposal-debate-judge`.
|
||||
- Added `security-review-council`.
|
||||
- Added `vision-code-verifier`.
|
||||
- Added `local-first`.
|
||||
- Added `single-model-parallel`.
|
||||
- Added `codex-ultra-native`, a native-only Codex expert team spanning architecture, implementation, testing, review, and research.
|
||||
|
||||
### Operations
|
||||
|
||||
- Added unified `codex-mmo` command dispatch while retaining `codex-mmoctl`.
|
||||
- Added profile selection, validation, compilation, installation, removal, cloning, and runtime model rebinding.
|
||||
- Added catalog inspection, inventory verification, live discovery, and refresh commands.
|
||||
- Added session, job, gateway, smoke-test, doctor, cleanup, authentication, and evaluation commands.
|
||||
- Added atomic XDG-compliant installation, credential-preserving upgrades, backups, and clean uninstallation.
|
||||
- Added deterministic release archives, schema-v2 per-file manifests, SHA-256 checksums, safe clean-extraction verification, and byte-for-byte reproducibility verification.
|
||||
- Added atomic publication and clean-extraction manifest verification so incomplete staging trees cannot be released.
|
||||
|
||||
### Evaluation and validation
|
||||
|
||||
- Added the `engineering-core` evaluation suite and cross-profile comparison metrics.
|
||||
- Added a standard-library JSON Schema subset for worker result contracts.
|
||||
- Added an offline integration suite using deterministic Codex and Switchyard stand-ins.
|
||||
- Added a release acceptance validator that checks source syntax, catalogs, profiles, snapshots, static safety, installer behavior, runtime controls, evaluation, and release hygiene.
|
||||
|
||||
## 2.0.0
|
||||
|
||||
- Introduced the GLM-5.3 root, DeepSeek V4 Flash engineering workers, constrained local Qwen scout, and isolated Agent MCP worker supervisor.
|
||||
- Added scope leasing, worker cancellation, persistent results, Switchyard integration, and an installable per-user package.
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Codex Multi-Model Orchestrator contributors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,30 @@
|
||||
PYTHON ?= python3
|
||||
PYTHONPATH := libexec:tests
|
||||
|
||||
.PHONY: test quality validate release clean-bytecode catalog
|
||||
|
||||
test:
|
||||
PYTHONDONTWRITEBYTECODE=1 PYTHONWARNINGS=error::ResourceWarning PYTHONPATH=$(PYTHONPATH) \
|
||||
$(PYTHON) scripts/run_tests.py
|
||||
|
||||
quality:
|
||||
ruff format --check .
|
||||
ruff check .
|
||||
mypy
|
||||
pyright
|
||||
shellcheck install.sh uninstall.sh bin/*.in
|
||||
shfmt -d install.sh uninstall.sh bin/*.in
|
||||
|
||||
validate:
|
||||
PYTHONDONTWRITEBYTECODE=1 $(PYTHON) scripts/validate_package.py
|
||||
|
||||
release:
|
||||
PYTHONDONTWRITEBYTECODE=1 $(PYTHON) scripts/build_release.py
|
||||
|
||||
catalog:
|
||||
PYTHONDONTWRITEBYTECODE=1 $(PYTHON) scripts/generate_catalog.py
|
||||
|
||||
clean-bytecode:
|
||||
find . -type d -name __pycache__ -prune -exec rm -rf {} +
|
||||
find . -type f \( -name '*.pyc' -o -name '*.pyo' \) -delete
|
||||
find . -type d \( -name .mypy_cache -o -name .ruff_cache \) -prune -exec rm -rf {} +
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,305 @@
|
||||
# Codex Multi-Model Orchestrator 8.0.0
|
||||
|
||||
Codex Multi-Model Orchestrator (Codex MMO) compiles provider-agnostic agent profiles into immutable snapshots used by persistent interactive sessions. A profile chooses models by exact access route, assigns roles and trust boundaries, and combines Codex native agents with a mechanically supervised Agent MCP runtime when useful.
|
||||
|
||||
The bundled portfolio is designed to show practical reasons for composition: useful concurrency, independent reproduction and verification, source-to-code audit, multimodal browser review, contract separation, security challenge, controlled weak-model use, and route resilience. It does not prefer one maker merely because its frontier model wins an aggregate benchmark.
|
||||
|
||||
## Single generation-8 format
|
||||
|
||||
This release is a clean break. It rejects:
|
||||
|
||||
- any MMO-owned document whose `schema_version` is not `8`, including settings, catalogs, Tool MCP registries, inventory snapshots, profiles, smoke/evaluation definitions, compiled snapshots/guidance, gateways, sessions, runs, jobs, and install/release manifests;
|
||||
- legacy provider-qualified model keys;
|
||||
- retired profile and evaluation IDs;
|
||||
- flat `can_control`, active-work/renewal/hard-wall fields, lifetime `max_total_spawns` and `max_spawns`, rollout-path state, per-spawn `timeout_seconds`, role progress-checkpoint fields, and old smoke timeout keys.
|
||||
|
||||
There are no migration commands, compatibility readers, alternate execution hosts, aliases, fallback key lookups, automatic rewrites, or compatibility profiles. Recreate configuration and profile packs directly in generation 8, and externally archive or remove any older runtime state before installation. Profile `version` must exactly equal the package version (`8.0.0`); it is not an independent upgrade axis. External formats owned by Codex app-server, MCP, providers, and Switchyard retain their upstream version identifiers and are not MMO schema surfaces.
|
||||
|
||||
## What the runtime enforces
|
||||
|
||||
- Exact route/model/reasoning bindings with separate maker, API operator, access product, protocol, billing, and serving-provider identities.
|
||||
- Content-addressed immutable snapshots inherited by every descendant.
|
||||
- Native, Agent-MCP, and hybrid orchestration selected per profile.
|
||||
- Authenticated lineage, directed delegation/control graphs, depth, reusable active capacity, resource groups, and cancellation. Terminal workers release admission capacity, so long sessions can delegate sequentially without exhausting a lifetime counter.
|
||||
- One isolated Unix app-server host for every root and Agent-MCP worker. Workers retain one exact persistent thread; an interactive root retains an ordered, auditable lineage when the attached stock TUI deliberately starts a fresh top-level context. Durable event/terminal history, transport recovery, continuation, and bounded partial evidence remain inside the same immutable MMO session and run.
|
||||
- Goal-mode token budgets with profile ceilings, turn mode without a task clock, and warning-only stall intervals. Provider/model slowness never discards work and models never track elapsed time themselves.
|
||||
- Compare-and-swap inspect/trace/steer/interrupt/pause/continue/detach/stop/finalize/compact/respond/effort/fork controls constrained by per-target action grants; pending user-input, MCP-elicitation, and approval responses retain their method-specific app-server shapes. MCP pause is a durable cold pause that releases host capacity; detach is the live-background operation.
|
||||
- Typed failed-turn retention, provider-limit suspension with raw/reset evidence, and exact runtime package/code fingerprints make recovery causes and stale live hosts explicit.
|
||||
- Per-route availability: a missing optional worker degrades explicitly; an unavailable root prevents startup.
|
||||
- MCP-only structured literal tasks, read-only workspaces, strict contracts, and mandatory verification for weak roles.
|
||||
- Isolated Git worktrees for delegated writers, binary-safe patches, artifact/hash evidence, taint detection, and explicit root integration.
|
||||
- Lossless cursor-paginated worker results and valid filtered traces: `agents_wait` returns compact revision-addressed state with previews opt-in, `agent_result --cursor` returns contiguous result pages, and `agent_trace_record` pages one oversized filtered event until `next_cursor` is null. Malformed trace records remain bounded because they cannot be structurally filtered. Reading, accepting/rejecting, and—only for accepted patches—integrating remain separate monotonic steps. Model-visible MCP responses do not expose supervisor result/event/stderr/socket paths.
|
||||
- Complete-route modality checks, including image and tool-produced media preservation.
|
||||
- Typed OpenRouter serving policies plus requested-route and actual-serving telemetry.
|
||||
- Separate subscription, local-resource, actual-API, and API-equivalent cost ledgers.
|
||||
- Operator-owned Tool MCP definitions that profiles may only narrow.
|
||||
|
||||
Structured contracts prove shape and correlated artifacts/commands, not semantic truth. Native Codex controls remain less observable than Agent MCP; all bundled featured native roles are read-only.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
immutable MMO session and Unix app-server root host
|
||||
├── detachable stock Codex TUI or noninteractive client on the current root generation
|
||||
├── direct critical-path work and canonical workspace
|
||||
├── selected operator-owned Tool MCP servers
|
||||
├── native Codex agents (optional, low overhead, advisory controls)
|
||||
└── mmo_mesh Agent MCP (optional, mechanical controls)
|
||||
└── isolated persistent app-server workers
|
||||
├── pinned route/model/reasoning
|
||||
├── lineage, goal budgets, scopes, contracts, live control
|
||||
└── durable trace/partial/result and isolated Git patch lifecycle
|
||||
|
||||
catalog identity
|
||||
model maker + upstream ID
|
||||
×
|
||||
API operator + access product + wire protocol + billing route
|
||||
×
|
||||
actual serving provider when the endpoint reports it
|
||||
```
|
||||
|
||||
The runtime contains no model-family scheduling branches. Profiles express intelligence policy; the kernel enforces generic process, resource, result, and integration mechanics.
|
||||
|
||||
## Bundled portfolio
|
||||
|
||||
### Featured profiles
|
||||
|
||||
| Profile | Workload | Root | Composition | Engine |
|
||||
|---|---|---|---|---|
|
||||
| `adaptive-engineering` | General decomposable engineering | GPT-5.6 Sol via ChatGPT | Luna scout, DeepSeek V4 Pro implementer, Opus 5 reviewer | Hybrid |
|
||||
| `codex-harness-team` | ChatGPT-only context isolation/control | GPT-5.6 Sol via ChatGPT | Luna scout, Terra invariant role, fresh Sol critic | Hybrid |
|
||||
| `high-confidence-debugging` | Reproduce, repair, falsify | DeepSeek V4 Pro via Go | Sonnet 5 reproducer, Sol verifier | Agent MCP |
|
||||
| `research-backed-engineering` | Current-source implementation | MiniMax M3 via Go | Luna researchers, Sonnet 5 source auditor | Agent MCP |
|
||||
| `secure-change` | Security-sensitive change | Opus 5 via Zen | GLM-5.3 Coding Plan hunter, DeepSeek verifier | Agent MCP |
|
||||
| `visual-engineering` | Reference-driven UI/browser work | GPT-5.6 Sol via ChatGPT | Terra analyst, fresh multimodal Sol verifier | Hybrid |
|
||||
| `contract-first-refactoring` | Compatibility-sensitive refactor | Sonnet 5 via Zen | MiniMax invariant miner, Luna test author, DeepSeek verifier | Agent MCP |
|
||||
| `incident-hypothesis-triage` | Read-only causal incident analysis | GLM-5.3 via Z.AI Coding Plan | DeepSeek Flash runner, pinned Nemotron challenger | Agent MCP |
|
||||
|
||||
### Experimental labs
|
||||
|
||||
| Profile | Hypothesis |
|
||||
|---|---|
|
||||
| `access-efficient-escalation-lab` | Contained local/lower-cost work can reduce scarce-tier use without unacceptable quality loss |
|
||||
| `route-resilience-lab` | Matched DeepSeek work through Go and pinned OpenRouter exposes route drift and failure behavior |
|
||||
| `bounded-research-organization-lab` | A depth-two research hierarchy can outperform the same flat fan-out on long corpora |
|
||||
| `competing-implementations-lab` | Two isolated implementations selected by tests can beat one after total cost and integration effort |
|
||||
|
||||
Labs report matched evidence but make no superiority claim. See [Bundled profiles](docs/PROFILES.md) and [coverage](docs/PLAN-COVERAGE.md).
|
||||
|
||||
## Catalog
|
||||
|
||||
The reviewed baseline, current through 2026-08-19, contains 21 routes, 548 models, 452 agent-compatible models, and 9 resource groups across Codex/ChatGPT, OpenCode Go, OpenCode Zen, OpenRouter, Z.AI General API, Z.AI Coding Plan, and local routes.
|
||||
|
||||
Keys are route-qualified:
|
||||
|
||||
```text
|
||||
codex_chatgpt_builtin__gpt_5_6_sol
|
||||
opencode_go_openai_chat__deepseek_v4_pro
|
||||
opencode_zen_anthropic_messages__claude_opus_5
|
||||
zai_coding_responses__glm_5_3
|
||||
openrouter_openai_chat__nvidia_nemotron_3_ultra_550b_a55b
|
||||
```
|
||||
|
||||
Bundled GLM-5.3 roles use only the Z.AI Coding Plan Responses route and `ZAI_CODING_API_KEY`. The separate Z.AI General API remains available to users who have it, but is never an implicit fallback.
|
||||
|
||||
Every external inventory, from three Coding Plan records to 413 OpenRouter records, follows the same reviewed snapshot/fingerprint/generation pipeline. Live discovery reports drift; it never invents capabilities for a new ID.
|
||||
|
||||
```bash
|
||||
codex-mmo catalog summary
|
||||
codex-mmo catalog routes
|
||||
codex-mmo catalog model zai_coding_responses__glm_5_3
|
||||
codex-mmo catalog inventory
|
||||
codex-mmo catalog verify --remote --codex
|
||||
```
|
||||
|
||||
See [Route and model catalog](docs/CATALOG.md).
|
||||
|
||||
## Installation
|
||||
|
||||
Requirements:
|
||||
|
||||
- Linux, Bash, Git, and Python 3.11+;
|
||||
- Codex CLI exactly 0.149.0 with the pinned experimental app-server schema;
|
||||
- Switchyard 0.2.0 for translated/external routes;
|
||||
- Rust/Cargo 1.96.1+ only when asking the installer to build Switchyard;
|
||||
- credentials only for routes you intend to use.
|
||||
|
||||
```bash
|
||||
./install.sh
|
||||
|
||||
# Or install missing Codex/Switchyard executables too:
|
||||
./install.sh --install-codex --install-switchyard
|
||||
```
|
||||
|
||||
Configure credentials and ChatGPT-backed Codex authentication:
|
||||
|
||||
```bash
|
||||
${EDITOR:-vi} ~/.config/codex-mmo/credentials.env
|
||||
codex-mmo auth login
|
||||
codex-mmo auth status
|
||||
```
|
||||
|
||||
Validate, then run:
|
||||
|
||||
```bash
|
||||
codex-mmo validate --all-profiles
|
||||
codex-mmo doctor --profile adaptive-engineering --live
|
||||
|
||||
cd /path/to/repository
|
||||
codex-mmo
|
||||
```
|
||||
|
||||
See [Installation](docs/INSTALLATION.md). Run `codex-mmo --help` for the complete command map;
|
||||
the [CLI guide](docs/CLI.md) documents output, automation, exit-status, input, and precedence
|
||||
contracts.
|
||||
|
||||
## Command-line behavior
|
||||
|
||||
Structured commands render human-readable tables or labeled details when stdout is a terminal and
|
||||
strict JSON when stdout is piped or redirected. `--json` forces JSON; `--quiet` suppresses only
|
||||
interactive stage progress; `--debug` adds exception diagnostics. Requested data goes to stdout,
|
||||
while progress, warnings, usage guidance, and errors go to stderr. Global flags may appear before or
|
||||
after a control command as long as they precede the explicit `--` separator used for Codex-owned
|
||||
arguments.
|
||||
|
||||
`codex-mmo exec` accepts a positional prompt or non-empty piped stdin and fails immediately rather
|
||||
than reading from an interactive terminal when both are absent. `codex-mmoctl` exposes the same
|
||||
control tree without the primary command's implicit interactive `run`. See the [CLI guide](docs/CLI.md)
|
||||
for shell examples, machine-error shape, lifecycle vocabulary, and exit statuses.
|
||||
|
||||
## Profiles and runs
|
||||
|
||||
```bash
|
||||
codex-mmo profile list
|
||||
codex-mmo profile show adaptive-engineering --resolved
|
||||
codex-mmo profile validate adaptive-engineering
|
||||
codex-mmo profile compile adaptive-engineering
|
||||
codex-mmo profile use high-confidence-debugging
|
||||
|
||||
codex-mmo run --profile adaptive-engineering
|
||||
codex-mmo run --profile visual-engineering -- --image screenshot.png
|
||||
codex-mmo resume --last
|
||||
codex-mmo resume SESSION_ID
|
||||
codex-mmo exec --profile incident-hypothesis-triage "Analyze the supplied incident evidence"
|
||||
```
|
||||
|
||||
`codex-mmo` starts one persistent Unix app-server session and attaches the stock Codex TUI as a remote client when no control command is supplied. `exec` uses the same host/thread model without a TUI. `resume` reattaches to the exact active immutable session, snapshot, working directory, generated `CODEX_HOME`, and current root-thread generation; any predecessor thread ID in that session's lineage resolves to the current generation. It reuses the live app-server host when its generated routing configuration remains compatible, starts a replacement when that host is gone, and deliberately recycles it when the restored gateway endpoint or compiled route availability changes. A terminal session cannot be resumed; start a deliberately new session instead. `codex-mmoctl` invokes the same control plane without implicit interactive behavior. Put `--` before Codex-owned options such as `--image` or `--search`.
|
||||
|
||||
When a fresh top-level context is observed while a fingerprinted stock TUI is attached and the prior root has no active turn, MMO stages and commits a new root-thread generation without creating another MMO session or run. Native child/fork threads are never eligible. If the controller loses the notification, recovery consults only the pinned app-server's session-isolated thread index, records the recovered transition, and resumes the latest canonical generation; it never searches rollout files or unrelated Codex history. A fresh interactive goal is restored before new work is treated as active.
|
||||
|
||||
`exec --wall-timeout SECONDS` is an external caller/harness limit, not an agent budget. Reaching it detaches the client while the root host and goal continue, preserving the exact thread and partial evidence for `resume`. A successful root turn may also return the session as detached when recoverable descendants remain active or suspended; inspect, continue, finalize, or fully stop those workers instead of silently abandoning them. Goal-mode profile limits use Codex's token accounting, not a model-authored or supervisor-guessed clock.
|
||||
|
||||
Codex app-server accepts at most 4,000 characters in a goal objective. MMO treats that field as a lifecycle label, not the task transport: the complete prompt remains unchanged in the initial turn, while a long goal label is a deterministic prefix plus a SHA-256 correlation note. Interactive goal sessions are seeded paused before the TUI attaches and become active only after the first accepted turn.
|
||||
|
||||
Profile-local rebinding remains exact and capability-checked:
|
||||
|
||||
```bash
|
||||
codex-mmo run \
|
||||
--profile adaptive-engineering \
|
||||
--bind implementation_specialist=opencode_go_openai_chat__deepseek_v4_pro
|
||||
```
|
||||
|
||||
User profiles live under `~/.config/codex-mmo/profiles.d/`. Packs are static TOML, Markdown, and JSON only; executable code, symlinks, credentials, and Tool MCP endpoint definitions are rejected.
|
||||
|
||||
## Tool MCP versus Agent MCP
|
||||
|
||||
`mmo_mesh` is the runtime-owned Agent MCP server for delegation. Third-party Tool MCP servers—Firecrawl, IDA, browser tools, or organization-specific services—are a separate operator boundary:
|
||||
|
||||
```text
|
||||
~/.config/codex-mmo/tool-mcp.d/*.toml
|
||||
```
|
||||
|
||||
Profiles reference stable server/tool IDs and can only narrow the operator’s finite allowlist and approval ceiling. A read-only agent role does not make an external MCP tool read-only in its own system.
|
||||
|
||||
```bash
|
||||
codex-mmo tool-mcp list
|
||||
codex-mmo tool-mcp show firecrawl
|
||||
codex-mmo tool-mcp validate
|
||||
codex-mmo profile doctor adaptive-engineering
|
||||
```
|
||||
|
||||
See [Tool MCP](docs/TOOL_MCP.md).
|
||||
|
||||
## Sessions and jobs
|
||||
|
||||
```bash
|
||||
codex-mmo session list
|
||||
codex-mmo session show SESSION_ID
|
||||
codex-mmo session runs SESSION_ID
|
||||
codex-mmo session detach SESSION_ID
|
||||
codex-mmo session pause SESSION_ID
|
||||
codex-mmo session compact SESSION_ID
|
||||
codex-mmo session continue SESSION_ID --input "Proceed from the retained evidence"
|
||||
codex-mmo session continue SESSION_ID --goal-token-budget 800000
|
||||
codex-mmo session stop SESSION_ID --grace 300
|
||||
codex-mmo session cancel SESSION_ID
|
||||
|
||||
codex-mmo jobs list --session SESSION_ID
|
||||
codex-mmo jobs list --run RUN_ID
|
||||
codex-mmo jobs status JOB_ID
|
||||
codex-mmo jobs result JOB_ID
|
||||
codex-mmo jobs result JOB_ID --cursor NEXT_CURSOR
|
||||
codex-mmo jobs wait --session SESSION_ID JOB_ID
|
||||
codex-mmo jobs cancel JOB_ID --reason "superseded"
|
||||
|
||||
codex-mmo gateway list
|
||||
codex-mmo gateway logs PROFILE
|
||||
codex-mmo gateway stop-idle
|
||||
```
|
||||
|
||||
One logical session contains exactly one immutable run and one current root-thread generation. `detach` disconnects a client while root and workers continue. `pause` durably checkpoints partial evidence, pauses/interrupts admitted turns, retires root and supervised-worker hosts, and releases their execution capacity without changing thread identity. `compact` hosts the paused root just long enough to invoke app-server compaction and then returns the session to a cold pause. `continue` re-admits and reactivates the same root generation plus the exact paused worker set, and may extend a goal only within its compiled token ceiling. `stop` asks active work to finalize from retained evidence, waits the requested operator grace, then retires every host in the run. `cancel` is immediate. A controller, provider, or transport failure produces typed recoverable state on the persisted lineage; it does not erase events, messages, tool observations, terminal history, partial results, or a bounded writable patch.
|
||||
|
||||
`session_kind` records whether the first client was interactive or noninteractive; it does not select an execution backend. `root_execution_host` is always `app_server`. An interactive TUI attaches with Codex `--remote` to the existing Unix socket and never becomes the owner of thread or worker lifetime.
|
||||
|
||||
Terminal sessions and jobs are never revived. Active admission and goal-token ceilings belong to the one immutable run; terminal jobs release their simultaneous-capacity leases, and all jobs remain inspectable after the run ends. The root performs accept/reject/integrate transitions for successfully completed results through its authenticated Agent-MCP tools. Stopped, failed, and cancelled jobs remain readable but cannot be dispositioned as successes. Reading a CLI job result for inspection does not accept it or apply its patch. When `next_cursor` is non-null, request that exact cursor and concatenate text pages in order; no head/tail elision occurs.
|
||||
|
||||
## Smoke tests and evaluation
|
||||
|
||||
```bash
|
||||
codex-mmo profile smoke adaptive-engineering --cwd "$PWD"
|
||||
codex-mmo profile smoke secure-change --workers-only --cwd "$PWD"
|
||||
|
||||
codex-mmo eval suites
|
||||
codex-mmo eval validate adaptive-change
|
||||
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change --dry-run
|
||||
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change
|
||||
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change --trial-mode release
|
||||
codex-mmo eval compare RUN_ID_A RUN_ID_B
|
||||
```
|
||||
|
||||
Every suite uses matched controls and worker ablations. Development mode runs three trials per task; release mode runs five. Evaluations record outcomes, explicit waits, root activity events, concurrency, result disposition, integration, route/serving identity, and separate cost ledgers. They publish no aggregate score.
|
||||
|
||||
Bundled status is not evidence that every composition has passed live promotion. See [Evaluation](docs/EVALUATION.md).
|
||||
|
||||
## Development and release checks
|
||||
|
||||
```bash
|
||||
make test
|
||||
make validate
|
||||
make release
|
||||
```
|
||||
|
||||
Release validation checks the single MMO schema-8/package-8.0.0 generation; catalog reproducibility; all profiles/suites; the exact Codex 0.149.0 protocol pin and offline app-server behavior; package safety; deterministic archives; and integration tests. `doctor --live` separately proves that the installed binary generates the pinned 401-file upstream app-server schema digest. Authenticated provider availability and five-trial efficacy remain separate live acceptance evidence.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Architecture](docs/ARCHITECTURE.md)
|
||||
- [Command-line interface](docs/CLI.md)
|
||||
- [Profile schema](docs/PROFILE_SCHEMA.md)
|
||||
- [Bundled profiles](docs/PROFILES.md)
|
||||
- [Orchestration backends](docs/ORCHESTRATION_BACKENDS.md)
|
||||
- [Route and model catalog](docs/CATALOG.md)
|
||||
- [Provider/route drivers](docs/PROVIDER_DRIVERS.md)
|
||||
- [Tool MCP](docs/TOOL_MCP.md)
|
||||
- [Security model](docs/SECURITY.md)
|
||||
- [Evaluation](docs/EVALUATION.md)
|
||||
- [Release acceptance](docs/ACCEPTANCE.md)
|
||||
- [Troubleshooting](docs/TROUBLESHOOTING.md)
|
||||
|
||||
## Scope
|
||||
|
||||
Codex MMO is an orchestration and enforcement runtime, not a workflow programming language. Profiles use agents, exact bindings, directed edges, bounded nesting, budgets, resources, permissions, scopes, contracts, trust rules, and instructions. They do not embed Python, arbitrary loops, expression DSLs, or scheduler plugins.
|
||||
|
||||
## License
|
||||
|
||||
See [LICENSE](LICENSE).
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
# Validation
|
||||
|
||||
Codex MMO has two validation levels.
|
||||
|
||||
## Release acceptance
|
||||
|
||||
Run from the package source or an extracted release:
|
||||
|
||||
```bash
|
||||
PYTHONDONTWRITEBYTECODE=1 python3 scripts/validate_package.py
|
||||
```
|
||||
|
||||
Write a machine-readable report:
|
||||
|
||||
```bash
|
||||
python3 scripts/validate_package.py \
|
||||
--report codex-multimodel-orchestrator-8.0.0-VALIDATION.json
|
||||
```
|
||||
|
||||
The release validator checks:
|
||||
|
||||
1. Required source, documentation, profile, configuration, and evaluation files.
|
||||
2. Exact source-tree agreement with the checked-in `PACKAGE-MANIFEST.json`.
|
||||
3. Python AST and bytecode compilation without writing into the source tree.
|
||||
4. Shell syntax for launchers and wrapper templates.
|
||||
5. TOML and JSON parsing across the package.
|
||||
6. No symlinks, special files, bytecode caches, unresolved release placeholders, embedded private keys, or common credential formats.
|
||||
7. Static non-executable profile-pack constraints.
|
||||
8. Exact twelve-profile inventory—eight featured profiles and four labs—with no retired-ID aliases and coverage of MCP, native, and hybrid orchestration.
|
||||
9. Resolution and immutable compilation of every profile.
|
||||
10. Closed Tool MCP registry definitions, per-agent subset grants, and secret-free snapshot material.
|
||||
11. Low-trust mechanical boundaries.
|
||||
12. Complete, hash-checked catalog snapshots: OpenRouter, OpenCode Zen/Go, Z.AI API, Z.AI Coding Plan, and Codex.
|
||||
13. Deterministic regeneration of the catalog and upstream inventory.
|
||||
14. Evaluation-suite validity, including safe image fixtures and closed worker assertions.
|
||||
15. The full offline integration suite unless `--skip-tests` is specified.
|
||||
|
||||
## Development quality gates
|
||||
|
||||
Run `make quality` to check Ruff formatting and lint, mypy, Pyright, ShellCheck, and shfmt. These gates are complementary to the behavioral and release-acceptance checks above.
|
||||
|
||||
## Offline integration suite
|
||||
|
||||
```bash
|
||||
make test
|
||||
```
|
||||
|
||||
The suite uses deterministic local stand-ins rather than external APIs. It covers:
|
||||
|
||||
- Common inventory-snapshot integrity, catalog baselines, OpenCode Zen/OpenRouter discovery, and runtime Codex discovery.
|
||||
- Authenticated Z.AI discovery without credential leakage.
|
||||
- All profile resolution and content-addressed snapshots.
|
||||
- MCP/native/hybrid configuration generation.
|
||||
- Tool MCP registry layering, profile grant bounds, native-role enable/disable overlays, credential forwarding, and CLI readiness diagnostics.
|
||||
- Native-agent capability tokens and caller authentication.
|
||||
- Persistent interactive and noninteractive app-server resume with immutable snapshot/home/run
|
||||
reuse, ordered top-level root-thread succession, predecessor-ID resolution, stable run-scoped
|
||||
capabilities, stale worker/launcher reconciliation, reusable active-only admission capacity,
|
||||
and token budgets that are never replenished by reattachment.
|
||||
- Asynchronous app-server worker completion, strict `outputSchema` projection plus full-contract
|
||||
validation, same-thread shape repair, orphaned-turn settlement before exactly one replacement
|
||||
continuation, durable partial evidence, and structured result contracts.
|
||||
- Codex goal-token budgets, clock-free turn execution, warning-only stalls, external harness
|
||||
detachment, finalization grace, slow-provider accounting, durable pending requests, transport
|
||||
recovery, suspended continuation, live controls, graceful stop, and immediate cancellation.
|
||||
- Low-trust task, permission, and backend enforcement.
|
||||
- Resource limits and conflicting write-scope rejection.
|
||||
- Cancellation reasons and descendant lifecycle handling.
|
||||
- Git audit of new and pre-existing dirty files.
|
||||
- Native and MCP smoke paths.
|
||||
- Atomic installation, upgrade backup, credential preservation, paths containing spaces, and uninstallation.
|
||||
- Evaluation image transport, closed MCP worker assertions, observed wait/parallelism telemetry, execution, and comparison metrics.
|
||||
- The complete 69-node CLI parser surface, command-local help, typo guidance, early cross-option
|
||||
validation, stdin nonblocking behavior, stdout/stderr separation, strict JSON failures, TTY and
|
||||
redirected rendering, narrow-terminal identifier fidelity, quiet progress, wrapper routing, and
|
||||
destructive dry-run reporting.
|
||||
|
||||
## Live acceptance
|
||||
|
||||
Offline success does not prove that a provider account, entitlement, endpoint, model, or local server is currently available. Run:
|
||||
|
||||
```bash
|
||||
codex-mmo catalog verify --remote --codex
|
||||
codex-mmo tool-mcp validate
|
||||
codex-mmo doctor --profile PROFILE --live --probe
|
||||
codex-mmo profile smoke PROFILE --cwd "$PWD"
|
||||
```
|
||||
|
||||
`catalog verify` compares observed IDs with the release baseline. `tool-mcp validate` performs non-launching command, working-directory, and credential readiness checks. `doctor --live` validates binaries, credentials, Switchyard configuration, routes, the internal Agent MCP handshake, and optionally the root model. `profile smoke` exercises each task through its declared root, native, or Agent-MCP backend; exercise third-party Tool MCP behavior explicitly when it is part of deployment acceptance.
|
||||
|
||||
## Reproducible release build
|
||||
|
||||
```bash
|
||||
make release
|
||||
```
|
||||
|
||||
The release builder:
|
||||
|
||||
- Runs the full acceptance validator.
|
||||
- Excludes development state and bytecode.
|
||||
- Normalizes ownership, permissions, ordering, and timestamps.
|
||||
- Adds a file-level `PACKAGE-MANIFEST.json` to the archive.
|
||||
- Builds tar.gz and ZIP artifacts in a private temporary directory.
|
||||
- Safely extracts each artifact and verifies the exact file set, explicit parent-directory set, file/directory modes, sizes, and SHA-256 values against `PACKAGE-MANIFEST.json`.
|
||||
- Requires each extracted artifact to match the source tree and each other.
|
||||
- Builds both artifacts a second time independently and fails if either pair differs byte-for-byte.
|
||||
- Publishes the verified archives, SHA-256 file, integrity report, and generated validation report as one rollback-safe set.
|
||||
- Verifies every final artifact while prior artifacts remain recoverable and restores the prior set if final-name verification fails.
|
||||
|
||||
The complete requirement-to-implementation mapping is in [docs/ACCEPTANCE.md](docs/ACCEPTANCE.md).
|
||||
The external dependency and standards evidence baseline is in [docs/EXTERNAL-VERACITY.md](docs/EXTERNAL-VERACITY.md).
|
||||
|
||||
## Independent archive verification
|
||||
|
||||
```bash
|
||||
python3 scripts/verify_release.py --source-tree . \
|
||||
../codex-multimodel-orchestrator-8.0.0-linux.tar.gz \
|
||||
../codex-multimodel-orchestrator-8.0.0-linux.zip
|
||||
```
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
# Installed by codex-multimodel-orchestrator.
|
||||
set -euo pipefail
|
||||
exec @@PYTHON_BIN@@ @@UNINSTALL_PATH@@ "$@"
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Installed by codex-multimodel-orchestrator.
|
||||
set -euo pipefail
|
||||
export MMO_CLI_ENTRYPOINT=codex-mmo
|
||||
exec @@PYTHON_BIN@@ @@MMOCTL_PATH@@ "$@"
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Installed by codex-multimodel-orchestrator.
|
||||
set -euo pipefail
|
||||
export MMO_CLI_ENTRYPOINT=codex-mmoctl
|
||||
exec @@PYTHON_BIN@@ @@MMOCTL_PATH@@ "$@"
|
||||
+16621
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,9 @@
|
||||
# Provider credentials used by bundled profiles.
|
||||
# Leave unused values empty. Switchyard receives provider values; direct
|
||||
# codex_custom providers and explicitly granted Tool MCP servers are the
|
||||
# documented worker-process exceptions. Add operator Tool MCP variables below.
|
||||
ZAI_CODING_API_KEY=
|
||||
OPENCODE_API_KEY=
|
||||
OPENROUTER_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
ANTHROPIC_API_KEY=
|
||||
@@ -0,0 +1,570 @@
|
||||
{
|
||||
"adapter": "codex_installed_models_join",
|
||||
"as_of": "2026-08-23",
|
||||
"captures": [
|
||||
{
|
||||
"response_sha256": "c18214b1ba88ab9bd164753115324a7a29c0582e8d071f7b3babf749d892f549",
|
||||
"retrieved_at": "2026-08-24T00:32:47Z",
|
||||
"source": "openai-codex-client-models"
|
||||
}
|
||||
],
|
||||
"discovery": {
|
||||
"ignored_model_prefixes": [
|
||||
"codex-auto-"
|
||||
],
|
||||
"known_deprecated_models": [
|
||||
"gpt-5.2",
|
||||
"gpt-5.3-codex",
|
||||
"gpt-5.3-codex-spark"
|
||||
]
|
||||
},
|
||||
"dynamic": false,
|
||||
"fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
],
|
||||
"inventory": "openai-codex",
|
||||
"models": {
|
||||
"codex_chatgpt_builtin__gpt_5_4": {
|
||||
"catalog": {
|
||||
"agent_compatible": true,
|
||||
"availability": "retires-chatgpt-2026-08-31",
|
||||
"capability_confidence": "codex-0.149.0-baseline",
|
||||
"context_window": 272000,
|
||||
"default_reasoning": "medium",
|
||||
"description": "Previous frontier Codex model; ChatGPT-authenticated access retires 2026-08-31",
|
||||
"display_name": "GPT-5.4 via ChatGPT",
|
||||
"inventory": "openai-codex",
|
||||
"kind": "chat",
|
||||
"maker": "openai",
|
||||
"modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"parallel_tool_calls": true,
|
||||
"reasoning_levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh"
|
||||
],
|
||||
"resource_group": "chatgpt_subscription",
|
||||
"route": "codex_chatgpt_builtin",
|
||||
"source": "openai-codex-client-models",
|
||||
"structured_output": false,
|
||||
"supports_reasoning_summaries": true,
|
||||
"tool_calling": true,
|
||||
"upstream_id": "gpt-5.4"
|
||||
},
|
||||
"evidence": {
|
||||
"codex_runtime": {
|
||||
"comp_hash": "2911",
|
||||
"context_window": 272000,
|
||||
"input_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"multi_agent_version": null,
|
||||
"reasoning": {
|
||||
"default": "medium",
|
||||
"levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh"
|
||||
],
|
||||
"summaries": true
|
||||
},
|
||||
"service_tiers": [
|
||||
{
|
||||
"description": "1.5x speed, increased usage",
|
||||
"id": "priority",
|
||||
"name": "Fast"
|
||||
}
|
||||
],
|
||||
"slug": "gpt-5.4",
|
||||
"structured_output": false,
|
||||
"supported_in_api": true,
|
||||
"tools": {
|
||||
"apply_patch_tool_type": "freeform",
|
||||
"parallel_tool_calls": true,
|
||||
"search": true,
|
||||
"shell_type": "shell_command",
|
||||
"tool_mode": null
|
||||
},
|
||||
"visibility": "hide"
|
||||
},
|
||||
"verified_fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
]
|
||||
}
|
||||
},
|
||||
"codex_chatgpt_builtin__gpt_5_4_mini": {
|
||||
"catalog": {
|
||||
"agent_compatible": true,
|
||||
"availability": "retires-chatgpt-2026-08-31",
|
||||
"capability_confidence": "codex-0.149.0-baseline",
|
||||
"context_window": 272000,
|
||||
"default_reasoning": "medium",
|
||||
"description": "Fast previous-generation Codex subagent model; ChatGPT-authenticated access retires 2026-08-31",
|
||||
"display_name": "GPT-5.4 Mini via ChatGPT",
|
||||
"inventory": "openai-codex",
|
||||
"kind": "chat",
|
||||
"maker": "openai",
|
||||
"modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"parallel_tool_calls": true,
|
||||
"reasoning_levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh"
|
||||
],
|
||||
"resource_group": "chatgpt_subscription",
|
||||
"route": "codex_chatgpt_builtin",
|
||||
"source": "openai-codex-client-models",
|
||||
"structured_output": false,
|
||||
"supports_reasoning_summaries": true,
|
||||
"tool_calling": true,
|
||||
"upstream_id": "gpt-5.4-mini"
|
||||
},
|
||||
"evidence": {
|
||||
"codex_runtime": {
|
||||
"comp_hash": "2911",
|
||||
"context_window": 272000,
|
||||
"input_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"multi_agent_version": null,
|
||||
"reasoning": {
|
||||
"default": "medium",
|
||||
"levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh"
|
||||
],
|
||||
"summaries": true
|
||||
},
|
||||
"service_tiers": [],
|
||||
"slug": "gpt-5.4-mini",
|
||||
"structured_output": false,
|
||||
"supported_in_api": true,
|
||||
"tools": {
|
||||
"apply_patch_tool_type": "freeform",
|
||||
"parallel_tool_calls": true,
|
||||
"search": true,
|
||||
"shell_type": "shell_command",
|
||||
"tool_mode": null
|
||||
},
|
||||
"visibility": "hide"
|
||||
},
|
||||
"verified_fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
]
|
||||
}
|
||||
},
|
||||
"codex_chatgpt_builtin__gpt_5_5": {
|
||||
"catalog": {
|
||||
"agent_compatible": true,
|
||||
"availability": "current-other",
|
||||
"capability_confidence": "codex-0.149.0-baseline",
|
||||
"context_window": 272000,
|
||||
"default_reasoning": "medium",
|
||||
"description": "Previous-generation frontier Codex model retained in the official selector",
|
||||
"display_name": "GPT-5.5 via ChatGPT",
|
||||
"inventory": "openai-codex",
|
||||
"kind": "chat",
|
||||
"maker": "openai",
|
||||
"modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"parallel_tool_calls": true,
|
||||
"reasoning_levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh"
|
||||
],
|
||||
"resource_group": "chatgpt_subscription",
|
||||
"route": "codex_chatgpt_builtin",
|
||||
"source": "openai-codex-client-models",
|
||||
"structured_output": false,
|
||||
"supports_reasoning_summaries": true,
|
||||
"tool_calling": true,
|
||||
"upstream_id": "gpt-5.5"
|
||||
},
|
||||
"evidence": {
|
||||
"codex_runtime": {
|
||||
"comp_hash": "2911",
|
||||
"context_window": 272000,
|
||||
"input_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"multi_agent_version": null,
|
||||
"reasoning": {
|
||||
"default": "medium",
|
||||
"levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh"
|
||||
],
|
||||
"summaries": true
|
||||
},
|
||||
"service_tiers": [
|
||||
{
|
||||
"description": "1.5x speed, increased usage",
|
||||
"id": "priority",
|
||||
"name": "Fast"
|
||||
}
|
||||
],
|
||||
"slug": "gpt-5.5",
|
||||
"structured_output": false,
|
||||
"supported_in_api": true,
|
||||
"tools": {
|
||||
"apply_patch_tool_type": "freeform",
|
||||
"parallel_tool_calls": true,
|
||||
"search": true,
|
||||
"shell_type": "shell_command",
|
||||
"tool_mode": null
|
||||
},
|
||||
"visibility": "list"
|
||||
},
|
||||
"verified_fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
]
|
||||
}
|
||||
},
|
||||
"codex_chatgpt_builtin__gpt_5_6_luna": {
|
||||
"catalog": {
|
||||
"agent_compatible": true,
|
||||
"availability": "recommended",
|
||||
"capability_confidence": "codex-0.149.0-baseline",
|
||||
"context_window": 272000,
|
||||
"default_reasoning": "medium",
|
||||
"description": "Fast built-in Codex participant for clear repeatable work",
|
||||
"display_name": "GPT-5.6 Luna via ChatGPT",
|
||||
"inventory": "openai-codex",
|
||||
"kind": "chat",
|
||||
"maker": "openai",
|
||||
"modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"parallel_tool_calls": true,
|
||||
"reasoning_levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh",
|
||||
"max"
|
||||
],
|
||||
"resource_group": "chatgpt_subscription",
|
||||
"route": "codex_chatgpt_builtin",
|
||||
"source": "openai-codex-client-models",
|
||||
"structured_output": false,
|
||||
"supports_reasoning_summaries": true,
|
||||
"tool_calling": true,
|
||||
"upstream_id": "gpt-5.6-luna"
|
||||
},
|
||||
"evidence": {
|
||||
"codex_runtime": {
|
||||
"comp_hash": "3000",
|
||||
"context_window": 272000,
|
||||
"input_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"multi_agent_version": "v1",
|
||||
"reasoning": {
|
||||
"default": "medium",
|
||||
"levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh",
|
||||
"max"
|
||||
],
|
||||
"summaries": true
|
||||
},
|
||||
"service_tiers": [
|
||||
{
|
||||
"description": "1.5x speed, increased usage",
|
||||
"id": "priority",
|
||||
"name": "Fast"
|
||||
}
|
||||
],
|
||||
"slug": "gpt-5.6-luna",
|
||||
"structured_output": false,
|
||||
"supported_in_api": true,
|
||||
"tools": {
|
||||
"apply_patch_tool_type": "freeform",
|
||||
"parallel_tool_calls": true,
|
||||
"search": true,
|
||||
"shell_type": "shell_command",
|
||||
"tool_mode": "code_mode_only"
|
||||
},
|
||||
"visibility": "list"
|
||||
},
|
||||
"verified_fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
]
|
||||
}
|
||||
},
|
||||
"codex_chatgpt_builtin__gpt_5_6_sol": {
|
||||
"catalog": {
|
||||
"agent_compatible": true,
|
||||
"availability": "recommended",
|
||||
"capability_confidence": "codex-0.149.0-baseline",
|
||||
"context_window": 272000,
|
||||
"default_reasoning": "low",
|
||||
"description": "Highest-capability built-in Codex participant for difficult open-ended work",
|
||||
"display_name": "GPT-5.6 Sol via ChatGPT",
|
||||
"inventory": "openai-codex",
|
||||
"kind": "chat",
|
||||
"maker": "openai",
|
||||
"modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"parallel_tool_calls": true,
|
||||
"reasoning_levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh",
|
||||
"max",
|
||||
"ultra"
|
||||
],
|
||||
"resource_group": "chatgpt_subscription",
|
||||
"route": "codex_chatgpt_builtin",
|
||||
"source": "openai-codex-client-models",
|
||||
"structured_output": false,
|
||||
"supports_reasoning_summaries": true,
|
||||
"tool_calling": true,
|
||||
"upstream_id": "gpt-5.6-sol"
|
||||
},
|
||||
"evidence": {
|
||||
"codex_runtime": {
|
||||
"comp_hash": "3000",
|
||||
"context_window": 272000,
|
||||
"input_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"multi_agent_version": "v2",
|
||||
"reasoning": {
|
||||
"default": "low",
|
||||
"levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh",
|
||||
"max",
|
||||
"ultra"
|
||||
],
|
||||
"summaries": true
|
||||
},
|
||||
"service_tiers": [
|
||||
{
|
||||
"description": "1.5x speed, increased usage",
|
||||
"id": "priority",
|
||||
"name": "Fast"
|
||||
}
|
||||
],
|
||||
"slug": "gpt-5.6-sol",
|
||||
"structured_output": false,
|
||||
"supported_in_api": true,
|
||||
"tools": {
|
||||
"apply_patch_tool_type": "freeform",
|
||||
"parallel_tool_calls": true,
|
||||
"search": true,
|
||||
"shell_type": "shell_command",
|
||||
"tool_mode": "code_mode_only"
|
||||
},
|
||||
"visibility": "list"
|
||||
},
|
||||
"verified_fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
]
|
||||
}
|
||||
},
|
||||
"codex_chatgpt_builtin__gpt_5_6_terra": {
|
||||
"catalog": {
|
||||
"agent_compatible": true,
|
||||
"availability": "recommended",
|
||||
"capability_confidence": "codex-0.149.0-baseline",
|
||||
"context_window": 272000,
|
||||
"default_reasoning": "medium",
|
||||
"description": "Balanced built-in Codex flagship participant for everyday engineering",
|
||||
"display_name": "GPT-5.6 Terra via ChatGPT",
|
||||
"inventory": "openai-codex",
|
||||
"kind": "chat",
|
||||
"maker": "openai",
|
||||
"modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"parallel_tool_calls": true,
|
||||
"reasoning_levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh",
|
||||
"max",
|
||||
"ultra"
|
||||
],
|
||||
"resource_group": "chatgpt_subscription",
|
||||
"route": "codex_chatgpt_builtin",
|
||||
"source": "openai-codex-client-models",
|
||||
"structured_output": false,
|
||||
"supports_reasoning_summaries": true,
|
||||
"tool_calling": true,
|
||||
"upstream_id": "gpt-5.6-terra"
|
||||
},
|
||||
"evidence": {
|
||||
"codex_runtime": {
|
||||
"comp_hash": "3000",
|
||||
"context_window": 272000,
|
||||
"input_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"multi_agent_version": "v2",
|
||||
"reasoning": {
|
||||
"default": "medium",
|
||||
"levels": [
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"xhigh",
|
||||
"max",
|
||||
"ultra"
|
||||
],
|
||||
"summaries": true
|
||||
},
|
||||
"service_tiers": [
|
||||
{
|
||||
"description": "1.5x speed, increased usage",
|
||||
"id": "priority",
|
||||
"name": "Fast"
|
||||
}
|
||||
],
|
||||
"slug": "gpt-5.6-terra",
|
||||
"structured_output": false,
|
||||
"supported_in_api": true,
|
||||
"tools": {
|
||||
"apply_patch_tool_type": "freeform",
|
||||
"parallel_tool_calls": true,
|
||||
"search": true,
|
||||
"shell_type": "shell_command",
|
||||
"tool_mode": "code_mode_only"
|
||||
},
|
||||
"visibility": "list"
|
||||
},
|
||||
"verified_fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"models_sha256": "582577ba4d39d61b9d12ff7be2349453c06d909ffc178086df32fc4fbd2ac8c5",
|
||||
"schema_version": 8,
|
||||
"sources": {
|
||||
"openai-codex-client-models": "https://raw.githubusercontent.com/openai/codex/rust-v0.149.0/codex-rs/models-manager/models.json",
|
||||
"openai-codex-models": "https://developers.openai.com/codex/models"
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,219 @@
|
||||
{
|
||||
"adapter": "zai_coding_docs_models_join",
|
||||
"as_of": "2026-08-16",
|
||||
"captures": [
|
||||
{
|
||||
"response_sha256": "3d1468829bc484e53d087182d5ce3ce2dade52d9a35a736c6406f19a4729c4d2",
|
||||
"retrieved_at": "2026-08-16T08:08:00Z",
|
||||
"source": "zai-coding-plan"
|
||||
},
|
||||
{
|
||||
"response_sha256": "960881863cc44246a1d25c3925964afc3c78f25fe24455e16da6706d88220fc4",
|
||||
"retrieved_at": "2026-08-16T08:08:00Z",
|
||||
"source": "zai-coding-models"
|
||||
},
|
||||
{
|
||||
"response_sha256": "cf623bb0507c21b807c377f0975b4fc897e68231157c125a770c3f3f8c9cac47",
|
||||
"retrieved_at": "2026-08-16T08:08:00Z",
|
||||
"source": "zai-codex-integration"
|
||||
},
|
||||
{
|
||||
"response_sha256": "b07051b257be57a382efdb536a72c12ddff7d332ac0776bb92db849da0423143",
|
||||
"retrieved_at": "2026-08-16T08:08:00Z",
|
||||
"source": "zai-core-parameters"
|
||||
}
|
||||
],
|
||||
"discovery": {
|
||||
"endpoint": "https://api.z.ai/api/coding/paas/v4/models"
|
||||
},
|
||||
"dynamic": false,
|
||||
"fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
],
|
||||
"inventory": "zai-coding-plan",
|
||||
"models": {
|
||||
"zai_coding_openai_chat__glm_4_7": {
|
||||
"catalog": {
|
||||
"agent_compatible": true,
|
||||
"availability": "current",
|
||||
"availability_source": "zai-coding-plan",
|
||||
"capability_confidence": "documented",
|
||||
"capability_source": "zai-core-parameters",
|
||||
"context_window": 204800,
|
||||
"default_reasoning": "none",
|
||||
"description": "Z.AI Coding Plan coding-agent model binding",
|
||||
"display_name": "GLM-4.7 Coding Plan",
|
||||
"inventory": "zai-coding-plan",
|
||||
"kind": "chat",
|
||||
"maker": "zai",
|
||||
"max_output_tokens": 131072,
|
||||
"modalities": [
|
||||
"text"
|
||||
],
|
||||
"output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"parallel_tool_calls": true,
|
||||
"reasoning_levels": [
|
||||
"none"
|
||||
],
|
||||
"resource_group": "zai_coding_plan",
|
||||
"route": "zai_coding_openai_chat",
|
||||
"source": "zai-coding-plan",
|
||||
"structured_output": true,
|
||||
"supports_reasoning_summaries": true,
|
||||
"tool_calling": true,
|
||||
"upstream_id": "glm-4.7"
|
||||
},
|
||||
"evidence": {
|
||||
"source_captures": [
|
||||
"zai-coding-plan",
|
||||
"zai-coding-models",
|
||||
"zai-core-parameters"
|
||||
],
|
||||
"verified_fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
]
|
||||
}
|
||||
},
|
||||
"zai_coding_openai_chat__glm_5_turbo": {
|
||||
"catalog": {
|
||||
"agent_compatible": true,
|
||||
"availability": "current",
|
||||
"availability_source": "zai-coding-plan",
|
||||
"capability_confidence": "documented",
|
||||
"capability_source": "zai-core-parameters",
|
||||
"context_window": 204800,
|
||||
"default_reasoning": "none",
|
||||
"description": "Z.AI Coding Plan coding-agent model binding",
|
||||
"display_name": "GLM-5-Turbo Coding Plan",
|
||||
"inventory": "zai-coding-plan",
|
||||
"kind": "chat",
|
||||
"maker": "zai",
|
||||
"max_output_tokens": 131072,
|
||||
"modalities": [
|
||||
"text"
|
||||
],
|
||||
"output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"parallel_tool_calls": true,
|
||||
"reasoning_levels": [
|
||||
"none"
|
||||
],
|
||||
"resource_group": "zai_coding_plan",
|
||||
"route": "zai_coding_openai_chat",
|
||||
"source": "zai-coding-plan",
|
||||
"structured_output": true,
|
||||
"supports_reasoning_summaries": true,
|
||||
"tool_calling": true,
|
||||
"upstream_id": "glm-5-turbo"
|
||||
},
|
||||
"evidence": {
|
||||
"source_captures": [
|
||||
"zai-coding-plan",
|
||||
"zai-coding-models",
|
||||
"zai-core-parameters"
|
||||
],
|
||||
"verified_fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
]
|
||||
}
|
||||
},
|
||||
"zai_coding_responses__glm_5_3": {
|
||||
"catalog": {
|
||||
"agent_compatible": true,
|
||||
"availability": "current",
|
||||
"availability_source": "zai-coding-plan",
|
||||
"capability_confidence": "documented",
|
||||
"capability_source": "zai-codex-integration",
|
||||
"context_window": 1048576,
|
||||
"default_reasoning": "max",
|
||||
"description": "Flagship orchestration, architecture, and difficult reasoning model",
|
||||
"display_name": "GLM-5.3",
|
||||
"inventory": "zai-coding-plan",
|
||||
"kind": "chat",
|
||||
"maker": "zai",
|
||||
"max_output_tokens": 131072,
|
||||
"modalities": [
|
||||
"text"
|
||||
],
|
||||
"output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"parallel_tool_calls": true,
|
||||
"reasoning_levels": [
|
||||
"low",
|
||||
"high",
|
||||
"max"
|
||||
],
|
||||
"resource_group": "zai_coding_plan",
|
||||
"route": "zai_coding_responses",
|
||||
"source": "zai-codex-integration",
|
||||
"structured_output": true,
|
||||
"supports_reasoning_summaries": true,
|
||||
"tool_calling": true,
|
||||
"upstream_id": "glm-5.3"
|
||||
},
|
||||
"evidence": {
|
||||
"source_captures": [
|
||||
"zai-coding-plan",
|
||||
"zai-coding-models",
|
||||
"zai-codex-integration",
|
||||
"zai-core-parameters"
|
||||
],
|
||||
"verified_fingerprint_fields": [
|
||||
"upstream_id",
|
||||
"canonical_slug",
|
||||
"pricing",
|
||||
"limits",
|
||||
"modalities",
|
||||
"tools",
|
||||
"reasoning",
|
||||
"structured_output",
|
||||
"supported_parameters",
|
||||
"deprecation",
|
||||
"endpoint_metadata"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"models_sha256": "1da052161686e3761a70b785fc67bdb152faedf3ed350f1b0b4a61dd55b9373d",
|
||||
"schema_version": 8,
|
||||
"sources": {
|
||||
"zai-codex-integration": "https://docs.z.ai/devpack/tool/codex.md",
|
||||
"zai-coding-models": "https://docs.z.ai/devpack/latest-model.md",
|
||||
"zai-coding-plan": "https://docs.z.ai/devpack/overview.md",
|
||||
"zai-core-parameters": "https://docs.z.ai/guides/overview/concept-param.md"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
schema_version = 8
|
||||
default_profile = "adaptive-engineering"
|
||||
base_codex_home = "~/.codex"
|
||||
auth_link_mode = "shared"
|
||||
gateway_host = "127.0.0.1"
|
||||
gateway_port_min = 42000
|
||||
gateway_port_max = 51999
|
||||
gateway_start_timeout_seconds = 15
|
||||
gateway_idle_timeout_seconds = 3600
|
||||
job_retention_days = 14
|
||||
session_retention_days = 14
|
||||
codex_bin = "codex"
|
||||
switchyard_bin = "switchyard-server"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,118 @@
|
||||
# Release acceptance
|
||||
|
||||
A Codex MMO release is acceptable only when the source tree, generated archives, and cleanly extracted archive trees all represent the same package.
|
||||
|
||||
## Required gates
|
||||
|
||||
Run from the project root:
|
||||
|
||||
```bash
|
||||
make clean-bytecode
|
||||
make test
|
||||
make validate
|
||||
make release
|
||||
```
|
||||
|
||||
`make test` runs the offline integration suite. `make validate` checks required files, source syntax, configuration syntax, profile semantics, model inventories, output contracts, package safety, catalog reproducibility, and the integration suite. `make release` repeats validation, builds deterministic tar.gz and ZIP artifacts, verifies every archived file against the staged file manifest, and performs a second build to confirm byte-for-byte reproducibility.
|
||||
|
||||
A successful source release must satisfy all of the following:
|
||||
|
||||
- Every bundled profile resolves and compiles into a content-addressed snapshot.
|
||||
- The installed Codex binary is exactly 0.149.0 and its 401 generated experimental schema files
|
||||
match the pinned digest, including Unix host, thread/goal/turn/history/control, pending-input,
|
||||
elicitation, and approval request/response contracts.
|
||||
- Every role receives the snapshot's exact generated `AGENTS.md`. Spawn- or control-capable roles receive the registered orchestration `SKILL.md`; true leaf homes do not materialize it, and leaf config layers explicitly disable the inherited skill. Leaf guidance declares the no-spawn/no-control boundary, and changed guidance changes snapshot identity.
|
||||
- The bundled set is exactly eight featured workload profiles plus four documented experimental labs; retired IDs have no aliases, compatibility readers, or install-time migration.
|
||||
- Every profile has a nonredundant purpose, a justified exact route-qualified model assignment, and a matching smoke/evaluation scenario.
|
||||
- Native-agent, Agent-MCP, and hybrid capabilities are represented where their enforcement trade-offs fit; no pure-native profile is required merely to exercise a mode.
|
||||
- Tool MCP definitions remain operator-owned, profile grants can only narrow finite tool allowlists, snapshots contain no credential values, and generated native-role overlays do not widen authority.
|
||||
- Low-trust roles are mechanically restricted.
|
||||
- Native-only roles do not claim strict contract enforcement, and no profile documents an advisory native control as an MCP guarantee.
|
||||
- Roots use dependency-only waiting instructions; spawn- or control-capable generated guidance describes
|
||||
inspect/trace/steer/interrupt/pause/continue/detach/stop/finalize/compact/respond/effort/fork controls;
|
||||
write-heavy MCP work has at most one active worker writer or read-only workers; and bounded
|
||||
nesting cannot exceed declared depth/budget.
|
||||
- Every root and Agent-MCP worker uses a non-ephemeral Unix app-server thread, authoritative event
|
||||
and terminal history, goal/turn lifecycle, warning-only stall policy, bounded explicit
|
||||
finalization, exact-thread recovery, `suspended` continuation, and partial-result retention.
|
||||
- Mutating controls serialize revision assignment through delivery; a lost reply after delivery
|
||||
begins is retained as `delivery_unknown`, not rewritten as a definite rejection or failure.
|
||||
- Interactive and noninteractive roots use the same app-server host; the TUI is a remote client.
|
||||
An external harness wall limit detaches while work continues, and pause, continue, detach,
|
||||
graceful full-stop, and immediate cancel have distinct tests.
|
||||
- Positive decomposable evaluation tasks require their intended reachable MCP roles, while negative controls remain free to reject unnecessary delegation.
|
||||
- Required multimodal roles preserve image input through the complete compiled transport, and the visual evaluation carries a real reference image.
|
||||
- The release-time OpenRouter, OpenCode Zen/Go, Z.AI, Z.AI Coding Plan, and Codex inventories match their hash-checked common snapshots and the generated catalog baseline.
|
||||
- Python, shell, TOML, and JSON files parse successfully.
|
||||
- No credentials, private keys, unresolved template markers, development markers, bytecode, or symlinks are present.
|
||||
- The release tree includes the runtime, CLI, catalog, profiles, documentation, evaluations, tests, installer, and uninstaller.
|
||||
- Tar and ZIP member names, exact permission modes, sizes, and content hashes match the staged release tree.
|
||||
- Archive member-count and expanded-byte ceilings are enforced progressively during streaming tar inspection/extraction and bounded ZIP extraction.
|
||||
- Repeated builds with the same source epoch produce identical archive hashes.
|
||||
- Remote TUI launch passes the PTY color, foreground-process-group, live-resize, and host-retention regression test.
|
||||
- Every external selected model has an exact startup catalog slug and validated context, reasoning, modality, and patch-tool metadata.
|
||||
- Mixed built-in/external native-agent processes preserve the active Codex bundled catalog while adding generated external route rows.
|
||||
- Every featured suite provides matched configured-root, strongest-single, accessible-service, root-plus-worker, full-profile, and complete worker-ablation evidence; release evaluation uses five trials per task.
|
||||
- Worker results use explicit read/accept/reject/integrate states, and delegated writer patches cannot reach the canonical workspace without accepted, audited integration.
|
||||
- OpenRouter policy and telemetry distinguish the aggregator route from the actual serving provider, and incomplete route identity fails featured promotion.
|
||||
- API dollars, API-equivalent estimates, subscription units, and local resource time remain separate ledgers with no aggregate score.
|
||||
- Every MMO-owned document uses exact schema 8 and, where applicable, exact package/profile version
|
||||
8.0.0. Other generations are rejected at load time without mutation, compatibility inspection,
|
||||
alternate host, fallback, or migration. No redundant lifecycle-version field exists. Exactly one
|
||||
immutable run exists per session and terminal sessions do not resume.
|
||||
- A generation-8 immutable snapshot whose resolved profile is not generation 8 is rejected before
|
||||
its policy can be interpreted by the current runtime.
|
||||
|
||||
## Archive verification
|
||||
|
||||
The release builder writes a generation-8 `PACKAGE-MANIFEST.json` inside each archive. The manifest records every packaged source file, normalized mode, size, SHA-256 digest, aggregate file count, and total payload bytes. The verifier compares exact permission bits rather than only executable status and calculates a deterministic tree digest over paths, modes, sizes, and bytes.
|
||||
|
||||
Verify externally supplied artifacts with:
|
||||
|
||||
```bash
|
||||
sha256sum -c codex-multimodel-orchestrator-8.0.0-SHA256SUMS.txt
|
||||
```
|
||||
|
||||
Inspect member counts:
|
||||
|
||||
```bash
|
||||
tar -tzf codex-multimodel-orchestrator-8.0.0-linux.tar.gz | wc -l
|
||||
unzip -Z1 codex-multimodel-orchestrator-8.0.0-linux.zip | wc -l
|
||||
```
|
||||
|
||||
Extract each archive into a separate empty directory and run:
|
||||
|
||||
```bash
|
||||
make test
|
||||
make validate
|
||||
```
|
||||
|
||||
The extracted package must pass the same gates as the source tree.
|
||||
|
||||
## Live acceptance
|
||||
|
||||
Offline acceptance proves package integrity and orchestration semantics without consuming provider quota. Deployment acceptance additionally requires credentials and reachable endpoints:
|
||||
|
||||
```bash
|
||||
codex-mmo validate --all-profiles
|
||||
codex-mmo tool-mcp validate
|
||||
codex-mmo catalog verify --remote --codex
|
||||
codex-mmo doctor --profile adaptive-engineering --live --probe
|
||||
codex-mmo profile smoke adaptive-engineering --cwd "$PWD"
|
||||
```
|
||||
|
||||
Repeat live smoke tests for every profile intended for production. A provider account may expose a different live inventory from the release-time baseline; refresh catalog overlays only after reviewing the discovered models and capabilities.
|
||||
|
||||
## Acceptance evidence
|
||||
|
||||
For a formal release, retain:
|
||||
|
||||
- Test and validation output.
|
||||
- The generated validation JSON report.
|
||||
- Both release archives.
|
||||
- The checksum file.
|
||||
- Archive member counts.
|
||||
- A clean-extraction comparison report.
|
||||
- Live doctor and smoke results for deployed profiles.
|
||||
|
||||
Live provider checks are deployment evidence and are not substituted by offline tests.
|
||||
@@ -0,0 +1,458 @@
|
||||
# Architecture
|
||||
|
||||
## Application purpose
|
||||
|
||||
Codex MMO is a single-user, host-local CLI and process supervisor for running immutable,
|
||||
provider-neutral multi-agent Codex profiles. Its normal path is: resolve a profile, compile an
|
||||
immutable snapshot, create or resume one durable root session, and coordinate native Codex agents
|
||||
and isolated Agent-MCP workers while retaining evidence and enforcing the profile's lineage,
|
||||
resource, workspace, and lifecycle policy.
|
||||
|
||||
The application owns orchestration state, generated Codex homes, local worker processes, immutable
|
||||
profile snapshots, and optional local Switchyard processes. Codex app-server owns model-thread and
|
||||
turn execution; providers own inference; operator-defined Tool MCP servers own their external tools.
|
||||
The deployment target is one workstation and a bounded number of local processes—not a distributed,
|
||||
multi-tenant scheduler. Correct recovery, explicit ownership, inspectable state, and useful root
|
||||
productivity matter more than horizontal scale or a general workflow language.
|
||||
|
||||
## Design objective
|
||||
|
||||
Codex MMO provides a small generic execution kernel for many model compositions. The runtime owns process and state mechanics; profiles own intelligence policy. No scheduler branch depends on a particular model or provider name.
|
||||
|
||||
```text
|
||||
static profile pack
|
||||
|
|
||||
v
|
||||
profile resolver + capability validation
|
||||
|
|
||||
v
|
||||
content-addressed immutable snapshot
|
||||
|
|
||||
+-- compiled per-role AGENTS.md and profile orchestration SKILL.md
|
||||
+-- generated Codex homes and native role files
|
||||
+-- generated Switchyard routes when required
|
||||
+-- resolved spawn/control graphs, contracts, resources, and trust
|
||||
|
|
||||
v
|
||||
immutable root app-server host + persistent thread
|
||||
|
|
||||
+-- detachable stock Codex TUI/noninteractive client
|
||||
+-- native Codex participant threads
|
||||
|
|
||||
+-- Agent MCP supervisor
|
||||
|
|
||||
+-- isolated persistent Unix app-server workers
|
||||
```
|
||||
|
||||
## Layer 1: runtime kernel
|
||||
|
||||
The kernel implements only generic mechanisms:
|
||||
|
||||
- Session and job lifecycle.
|
||||
- Content-addressed snapshot lookup.
|
||||
- Agent lineage and authenticated caller identity.
|
||||
- Directed spawn- and control-edge admission.
|
||||
- Depth, active-child, per-role, session, writer, and resource limits.
|
||||
- Workspace and attachment containment.
|
||||
- Write-scope conflict detection.
|
||||
- Asynchronous app-server hosting, same-thread recovery, live control, result retrieval, graceful stop, and immediate cancellation.
|
||||
- Codex goal-token lifecycle, turn-mode execution without a task clock, warning-only stalls, and bounded explicit terminal serialization.
|
||||
- Output-contract extraction and validation.
|
||||
- Durable audit, headerless app-server protocol event trace, stderr, partial/final result, patch, and metadata storage.
|
||||
- Credential filtering and generated Codex homes.
|
||||
- Content-addressed gateway lifecycle shared by equivalent route sets.
|
||||
|
||||
The kernel does not decide that a task is “Qwen work,” “GLM work,” or “DeepSeek work.” A profile maps task categories and role instructions to models.
|
||||
|
||||
### Code ownership
|
||||
|
||||
The implementation is organized around that execution path:
|
||||
|
||||
- `mmo_profiles.py` resolves composition policy and profile lifecycle; `mmo_catalog_data.py` owns
|
||||
catalog records and overlays, while `mmo_catalog.py` owns discovery and inventory operations.
|
||||
- `mmo_snapshot.py` compiles immutable resolved artifacts. `mmo_codex_home.py` projects those
|
||||
artifacts into generated Codex homes, native-role files, and process environments.
|
||||
- `mmo_runtime.py` is the orchestration center: admission, root/session commands, lineage, resource
|
||||
and write-scope policy, result disposition, and stop/cancel decisions live there.
|
||||
- `mmo_state.py` owns durable state paths, strict record readers, canonical session/job publication,
|
||||
session/run mirroring, and persisted session capabilities. Callers do not cache bearer tokens or
|
||||
write `session.json`/`metadata.json` directly. `mmo_workspace.py` owns Git worktree and patch
|
||||
mechanics.
|
||||
- `mmo_app_server.py` owns the exact Codex 0.149.0 Unix WebSocket transport, protocol gate,
|
||||
persistent-thread requests, server-request validation, bounded local control framing, control
|
||||
delivery, and partial-evidence extraction. `root_runner.py` and `worker_runner.py` own their
|
||||
respective session/job lifecycle, recovery, authoritative history, goal state, action semantics,
|
||||
and terminal publication around that shared transport. Each runner has one concrete lifetime owner;
|
||||
they deliberately do not share a base class.
|
||||
- `mmo_gateway.py` owns Switchyard lifecycle, route availability, and interpretation of provider/router
|
||||
telemetry. Generic worker execution records the resulting observation but does not parse
|
||||
OpenRouter- or Switchyard-specific event shapes.
|
||||
- `mmo_diagnostics.py` composes profile validation, doctor, MCP handshake, and live smoke workflows.
|
||||
`mmoctl.py` parses commands and renders results; it does not own those workflows.
|
||||
|
||||
These are concrete ownership boundaries, not replaceable service interfaces. Direct module calls are
|
||||
intentional because MMO is one local application, not a plugin container or distributed service.
|
||||
|
||||
## Layer 2: composition profiles
|
||||
|
||||
A profile defines:
|
||||
|
||||
- One root agent.
|
||||
- Any number of participant roles.
|
||||
- Model binding and reasoning effort per role.
|
||||
- Native, MCP, or hybrid execution per participant.
|
||||
- Directed `can_spawn` edges and per-target action-specific `controls` grants.
|
||||
- Task-kind admission.
|
||||
- Trust and verification policy.
|
||||
- Read-only or workspace-write permission.
|
||||
- Required input modalities.
|
||||
- Output contract and enforcement level.
|
||||
- Per-role concurrency, child, task-size, goal-token/stall/finalization, reasoning-effort, and resource limits.
|
||||
- Session-wide depth, spawn, writer, and active-agent limits.
|
||||
- Coordination and contradiction policy.
|
||||
- Role-specific instructions and smoke tests.
|
||||
- Per-role grants to operator-defined third-party Tool MCP servers.
|
||||
|
||||
Profile packs are static and auditable. They cannot include executable code.
|
||||
|
||||
## Layer 3: model and route catalog
|
||||
|
||||
The global catalog keeps model maker/upstream identity separate from executable route identity. Routes record API operator, access product, wire protocol, billing mode, endpoint, credential names, driver, transport behavior, and resource pool. Models record capabilities, costs where documented, inventory membership, context/output limits, reasoning levels, modalities, tools, and availability.
|
||||
|
||||
Externally sourced built-in inventories are catalog-ready records under `config/inventory-snapshots/`. OpenRouter, OpenCode Zen/Go, Z.AI, and Codex use the same schema, provenance envelope, and canonical model-record digest regardless of inventory size. `scripts/generate_catalog.py` validates and aggregates those snapshots offline, derives aggregate metadata from them, and runtime inventory verification compares their exact normalized records with the catalog. Local deployment entries remain explicit curated configuration.
|
||||
|
||||
Profiles refer to exact catalog keys rather than embedding endpoint configuration. Every model key begins with its exact route key plus `__`, so a binding identifies its access path without consulting a hidden default. Operators can add or override generation-8 catalog entries under:
|
||||
|
||||
```text
|
||||
~/.config/codex-mmo/catalog.d/*.toml
|
||||
```
|
||||
|
||||
Fragments are applied in lexical order. Profile-local `catalog.toml` fragments are applied after global overlays. Fragments from any other MMO generation and old model keys are rejected without migration or lookup fallback.
|
||||
|
||||
## Layer 4: Tool MCP registry
|
||||
|
||||
Machine-specific third-party MCP commands and endpoints live under
|
||||
`~/.config/codex-mmo/tool-mcp.d/*.toml`, outside installable profile packs. The registry defines a
|
||||
finite tool and approval-policy ceiling. Profiles reference stable server IDs and may narrow that
|
||||
ceiling per role but cannot change transport or operator policy. Referenced normalized definitions
|
||||
are part of snapshot identity; credential values are not.
|
||||
|
||||
At generated-config time, standalone Agent-MCP workers receive only relevant definitions. A process
|
||||
with native roles receives the union needed by that native tree, with an explicit enable/disable
|
||||
matrix per role because Codex custom-agent tables merge recursively. `mmo_mesh` remains a reserved
|
||||
runtime-owned Agent MCP identity.
|
||||
|
||||
## Layer 5: credentials
|
||||
|
||||
Credentials are never stored in profile packs or snapshots.
|
||||
|
||||
- Built-in ChatGPT/Codex access links or explicitly copies file-backed `auth.json` from the configured base `CODEX_HOME`; keyring-only state cannot cross Codex 0.149's home-scoped keyring namespace.
|
||||
- Switchyard receives the credentials declared by the snapshot’s selected routes; conventional unrelated credential-like names are filtered from its inherited environment.
|
||||
- MCP worker shells receive the same name-based filter, with only a direct selected route's declared credential/header variables admitted explicitly. Unconventionally named secrets cannot be identified mechanically.
|
||||
- A direct `codex_custom` route is the explicit exception: its required credential must be supplied to the Codex process that calls it.
|
||||
- Environment variables referenced by selected Tool MCP definitions are supplied explicitly. Native
|
||||
roles share these values with their parent process; Agent-MCP workers retain process isolation.
|
||||
|
||||
## Snapshot compilation
|
||||
|
||||
`codex-mmo profile compile PROFILE` performs:
|
||||
|
||||
1. Static profile-pack tree validation.
|
||||
2. Global catalog load and deterministic overlay merge.
|
||||
3. Operator Tool MCP registry resolution and grant validation.
|
||||
4. Provider, resource, and model validation.
|
||||
5. Runtime model rebinding.
|
||||
6. Agent capability, modality, trust, backend, and contract validation.
|
||||
7. Spawn-graph and coordination validation.
|
||||
8. Logical hash calculation excluding profile installation location.
|
||||
9. Deterministic per-role `AGENTS.md` and profile orchestration `SKILL.md` compilation.
|
||||
10. Snapshot identity calculation over resolved semantics and exact generated guidance bytes.
|
||||
11. Snapshot materialization under the state root.
|
||||
12. Deterministic Switchyard route generation if required.
|
||||
13. Read-only permission application.
|
||||
|
||||
Snapshots live under:
|
||||
|
||||
```text
|
||||
~/.local/state/codex-mmo/snapshots/SHA256/
|
||||
```
|
||||
|
||||
Every session records the profile ID, profile version, logical hash, physical snapshot hash, and resolved Codex executable. Descendants and resumed runs use the same snapshot and pinned executable even if PATH, the active profile, or future-session settings change. Within the current resolved-profile schema, snapshot loading authenticates historical generated guidance from the exact stored bytes covered by that snapshot's content address; it does not regenerate prose with a newer compiler. A snapshot carrying a retired resolved-profile schema is rejected rather than interpreted under new admission semantics. The remaining manifest semantics and non-guidance payloads are still derived from the stored resolved profile and verified byte-for-byte, so this is immutable verification rather than a migration or compatibility rewrite.
|
||||
|
||||
## Persistent sessions and execution runs
|
||||
|
||||
A root session is a durable logical container for one immutable profile snapshot, pinned working-directory boundary, generated Codex home, protected control identity, one app-server host, and one current top-level Codex thread in an ordered lineage. Plain `codex-mmo` records `session_kind = "interactive"`; `codex-mmo exec` records `"noninteractive"`. Both use the same canonical execution host. The kind describes the first client, not a backend choice.
|
||||
|
||||
Each session contains exactly one immutable run record. Reattach, pause/continue, controller replacement, and transport recovery preserve its run ID, capabilities, goal token budgets, workers, evidence, and exact thread identities. Detach retains scheduler capacity while its hosts remain active; cold pause and suspension release scheduler admission capacity, and continuation performs fresh admission before starting replacement controllers for those same threads. Gateway leases are deliberately separate: a preserved live root or worker app-server still consumes its route-set gateway even when its logical session/job no longer reserves scheduler capacity. The stock TUI may deliberately start a fresh top-level context; that advances an explicit root-thread generation inside the same immutable MMO session/run instead of silently creating another MMO session. Every predecessor, successor, timestamp, reason, and in-progress transition is durable, while native child/fork threads remain ineligible. Resume accepts any predecessor ID as a locator but always attaches the current canonical generation.
|
||||
|
||||
Resume first requires the session's exact MMO package-generation marker, then verifies the pinned snapshot and executable, restores its content-addressed gateway before refreshing generated homes, and reconciles dead session workers to evidence-retaining suspension before returning control. It never compiles the current profile, searches rollout files or unrelated Codex history, creates a second session/run, rotates the MMO identity, or adopts another runtime generation. Controller crash recovery may call the pinned app-server's `thread/list` only inside the session-isolated root home to recover a top-level successor already created by the attached TUI. A terminal session cannot be resumed.
|
||||
|
||||
`session detach` disconnects the client while the root host and descendants continue. `session pause` first persists a cold-pause checkpoint and partial evidence, then pauses/interrupts admitted turns and retires fingerprint-matched root and supervised-worker hosts. `session continue` freshly admits and reactivates the same root generation and exact paused worker set; it may increase a goal's total token budget only within the compiled ceiling. `session compact` temporarily hosts the paused root, invokes app-server compaction, and cold-pauses it again. `session stop --grace` requests evidence-only finalization, waits a bounded operator grace, then retires every host. `session cancel` is immediate termination with evidence retention.
|
||||
|
||||
Every MMO-owned runtime record uses schema 8 and exact package version 8.0.0; the redundant lifecycle-version field no longer exists. Job app-server and controller sockets use immutable identity-derived names in a private short runtime directory, so workspace and state-root length cannot disable control. Retired fields, other generations, and additional run inventories are rejected at load time, including inspection. Historical evidence belongs in an external manual archive and can enter a new session only through an explicit bounded handoff.
|
||||
|
||||
## Root sessions
|
||||
|
||||
A root session creates a session-specific generated Codex home containing:
|
||||
|
||||
- Root model/route configuration.
|
||||
- The snapshot's exact root `AGENTS.md`.
|
||||
- A registered `skills/mmo-profile-orchestration/SKILL.md` when the role may spawn or control another agent.
|
||||
- Native custom-agent files reachable from the root.
|
||||
- An Agent MCP server definition when the root has MCP-reachable children.
|
||||
- Root-granted Tool MCP definitions plus disabled definitions needed by reachable native roles.
|
||||
- Links to built-in Codex authentication when required.
|
||||
- A copy/link of the model catalog metadata needed by Codex.
|
||||
|
||||
Every root runner starts or reconnects to one isolated `codex app-server --listen unix://…` host and persists its current thread generation, ordered lineage, Unix socket, event stream, terminal history, goal state, pending requests, and process identity. Interactive use starts the stock Codex TUI with `--remote unix://… resume THREAD_ID`; the TUI never owns the durable host. A `thread/started` notification can advance the canonical generation only while the fingerprinted attached TUI is alive, the prior canonical thread matches the host, the prior root has no active turn, and the candidate is a persistent top-level thread in the pinned working directory. The transition is staged durably before publication; a replacement controller reconciles an interrupted transition from the isolated app-server thread index. A fresh generation receives the ongoing interactive goal before its first accepted turn is activated.
|
||||
|
||||
Noninteractive execution is another client of the same lifecycle. An external `--wall-timeout` detaches that client while the app-server goal continues; it never interrupts or deletes work. Pending user input, MCP elicitation, and non-auto-denied approval requests remain durably inspectable and cause the noninteractive caller to detach rather than fabricate a response.
|
||||
|
||||
A retryable failed turn, including a provider usage limit or transient transport failure, is durable thread history. While the fingerprinted TUI remains attached, the root controller retains partial evidence and keeps the app-server connection alive so the user can submit a later turn after recovery. Headless execution suspends and retires the host; explicitly detaching after an attached retryable failure does the same. Resume starts a replacement controller against the same thread rather than replaying or deleting the failed turn.
|
||||
|
||||
A root is published as completed only when its terminal history contains a readable agent result. If Codex reports completion but terminal history cannot supply that result, MMO reports failure and retains the available event evidence as partial output; it never converts an empty or unreadable result into success.
|
||||
|
||||
`root_execution_host` has one valid value: `app_server`. Client detach, TUI exit, terminal suspension, or controller replacement does not rewrite it. This keeps process ownership and observability truthful across every interface.
|
||||
|
||||
## Interactive terminal ownership
|
||||
|
||||
The stock remote TUI is launched without an intermediate terminal emulator or output proxy. The wrapper creates a dedicated client process group, assigns it as the terminal foreground owner, and waits with job-control awareness. Resize, interrupt, suspend, and input reach the TUI directly; the independently hosted app-server remains under MMO lifecycle control. On completion or suspension, the wrapper restores foreground ownership and saved terminal attributes before returning control to the shell, then records a client detach if the session remains active.
|
||||
|
||||
Root/worker runners and their app-server hosts use isolated process groups so full stop/cancel can retire exact descendants without targeting an unrelated shell group. Noninteractive callers are disposable clients; host lifetime is independent from them.
|
||||
|
||||
## Codex startup model catalogs
|
||||
|
||||
A generated provider route ID is also the model slug Codex sees. For every process containing a non-built-in model, the compiler writes `models.json` inside that process's isolated `CODEX_HOME` and points `model_catalog_json` at it. Rows are derived from the resolved model/provider binding and include exact slug, context window, reasoning presets, patch-tool transport, modalities, parallel-tool capability, and compaction headroom.
|
||||
|
||||
A custom startup catalog replaces Codex's normal startup catalog. If one native-agent process mixes built-in Codex models with external participants, the runtime therefore queries the active binary with `codex debug models --bundled`, caches the result by binary fingerprint, and merges the generated external route rows into those exact rows. Discovery occurs before the global runtime admission lock so a slow binary probe cannot serialize unrelated sessions.
|
||||
|
||||
## Native participants
|
||||
|
||||
Native participants are generated as Codex custom-agent TOML files. They inherit the current Codex workspace and sandbox environment. Their role file pins the model, provider, reasoning effort, description, instructions, and optional MCP server access.
|
||||
|
||||
Tool MCP visibility is role-scoped, but native roles share one operating-system process environment.
|
||||
The compiler warns when a native role's Tool MCP uses environment-backed authentication.
|
||||
|
||||
Native root-to-participant delegation is not visible to the external supervisor at every tool boundary. Therefore native limits are partly advisory. The default hybrid policy does not permit unobserved recursive native trees: a native participant that can delegate receives Agent MCP access for its children.
|
||||
|
||||
## Agent MCP participants
|
||||
|
||||
Agent MCP is a local STDIO server with up to twenty-four generic tools. Each generated server exposes exactly the subset authorized for that caller. A controller without spawn authority receives only its action-granted observation/control and result-read tools, not lineage cancellation, spawn, disposition, or patch-integration authority.
|
||||
|
||||
The server implements the MCP `2025-06-18` initialization lifecycle over newline-delimited UTF-8 STDIO and validates the JSON-RPC 2.0 request envelope. Calls other than `ping` are unavailable until a valid `initialize` request and `notifications/initialized` notification complete. If a client requests an unsupported protocol revision, the server returns its supported revision rather than echoing the unknown value.
|
||||
|
||||
- `agent_spawn`
|
||||
- `agents_spawn`
|
||||
- `agent_status`
|
||||
- `agents_wait`
|
||||
- `agent_result`
|
||||
- `agent_result_accept`
|
||||
- `agent_result_reject`
|
||||
- `agent_patch_integrate`
|
||||
- `agent_cancel`
|
||||
- `agent_list`
|
||||
- `agent_inspect`
|
||||
- `agent_trace`
|
||||
- `agent_trace_record`
|
||||
- `agent_steer`
|
||||
- `agent_interrupt`
|
||||
- `agent_pause`
|
||||
- `agent_continue`
|
||||
- `agent_detach`
|
||||
- `agent_stop`
|
||||
- `agent_finalize`
|
||||
- `agent_compact`
|
||||
- `agent_respond`
|
||||
- `agent_set_effort`
|
||||
- `agent_fork`
|
||||
|
||||
Each spawn starts a detached runner that privately hosts `codex app-server --listen unix://…`. Controller connections are disposable; the non-ephemeral Codex thread plus authoritative event/terminal history is the durable worker identity. A worker has:
|
||||
|
||||
- A minimal generated `CODEX_HOME`.
|
||||
- A pinned model/provider/reasoning role.
|
||||
- A bounded prompt and compiled `goal` or `turn` lifecycle.
|
||||
- A sandbox no stronger than the role maximum.
|
||||
- Optional validated attachments.
|
||||
- Optional explicit write scopes.
|
||||
- A role output contract plus command/artifact correlation when declared.
|
||||
- App-server `turn/start.outputSchema` constrained generation when a strict contract can be projected onto Codex's supported schema subset and the resolved model advertises structured-output support. The full original schema remains the independent enforcement authority. One same-thread shape-only repair turn may correct malformed JSON; no separate fixer role may invent evidence.
|
||||
- An isolated Git worktree and patch capture for writable workers.
|
||||
- Durable headerless app-server protocol events, terminal thread history, stderr, partial/final results, usage, and exact thread identity.
|
||||
- A distinct inherited MCP caller capability whose plaintext is not persisted.
|
||||
- Only Tool MCP definitions selected for that worker and its reachable native roles.
|
||||
|
||||
Each runner exposes an owner-only Unix control socket. The app-server boundary owns the shared strict
|
||||
JSON/newline framing and size limit; root and worker runners retain their different action semantics.
|
||||
Agents do not receive a worker's raw app-server socket: the authenticated Agent-MCP kernel resolves
|
||||
an opaque `agent_run_ref`, checks the action-specific `controls` grant, exact session/run identity,
|
||||
role, and target backend, then relays the request with compare-and-swap revision semantics. The same
|
||||
interface covers root, native, and MCP runs. Inspect and paginated trace filter private reasoning while
|
||||
retaining completed messages and empirical tool evidence. An oversized event becomes a checksummed
|
||||
bounded summary with a private `record_cursor`; `agent_trace_record` then returns the exact filtered
|
||||
record in contiguous checksummed pages. Malformed records retain only a bounded diagnostic because
|
||||
their raw content cannot be structurally filtered. Mutations support steering, current-turn
|
||||
interrupt, goal pause, same-thread continue/token extension, client detach, full stop, evidence-only
|
||||
finalization, compaction, exact pending-request response, allowed effort changes, and fork. Mutations
|
||||
serialize revision assignment through delivery. A reply lost after delivery begins is
|
||||
`delivery_unknown`; callers inspect before deciding whether a retry is safe.
|
||||
|
||||
The root and MCP participants receive Agent MCP access only for declared child/action roles. One 0600
|
||||
session capability document is the only runtime source of the root/native identities required by
|
||||
independently hosted detach/recovery across controller processes; public state contains only hashes
|
||||
and no process-global token cache can outlive or mask revocation of that file. It is stable for the
|
||||
immutable run and destroyed when the session becomes terminal. Job capabilities are inherited only
|
||||
by their detached runners. Native role capabilities live only in protected generated role
|
||||
configuration. The supervisor authenticates the exact session/run pair and durable lineage; a caller
|
||||
cannot claim another role, parent, or run.
|
||||
|
||||
## Goal, turn, and finalization model
|
||||
|
||||
Models are never asked to measure time. A `turn` role has no profile task clock. A `goal` role uses Codex's own cumulative token accounting, an initial `goal_token_budget`, and a profile ceiling. An authorized `continue` may raise the total budget within that ceiling. Once the objective and every required integration are complete, the model calls `update_goal(status="complete")` in its terminal turn; a final assistant message alone deliberately does not end an active goal. Goal completion is intent, not a replacement for `turn/completed`: the host remains attached until that terminal turn and its final message are durably recorded. Stall intervals publish operator warnings only; slow providers, long tool calls, pending input, and silence do not erase work or trigger model-blind termination.
|
||||
|
||||
Goal budget exhaustion never fabricates a terminal answer. The host first consumes the authoritative completion of any already-admitted turn. It publishes that turn only when its current-turn identity, successful status, and strict result contract all validate; the result records the limit and completes with a warning. Otherwise the worker suspends recoverably with partial evidence. Explicit finalization or full stop may request one evidence-only terminal serialization bounded by `finalization_grace_seconds`; strict contracts may receive at most one same-thread shape-only repair. Failure to serialize never deletes the authoritative events, complete terminal history, bounded readable partial, or isolated patch.
|
||||
|
||||
If a client transport exits, the runner reconnects to the exact Unix host/thread. If the host itself is lost, recovery starts one replacement host only for that same persisted thread. A turn committed before transport loss is read from authoritative history rather than duplicated. If hosting still fails, the run becomes `suspended`; trace, terminal history, pending requests, partial result, patch, and thread identity remain available for explicit continuation.
|
||||
|
||||
## Lineage and admission
|
||||
|
||||
For each MCP spawn, the runtime validates:
|
||||
|
||||
1. The root session is active and uses the requested snapshot.
|
||||
2. The caller identity and capability token are valid.
|
||||
3. The requested child is present in the caller’s `can_spawn` list.
|
||||
4. The child supports MCP.
|
||||
5. The task kind is allowed.
|
||||
6. Task length and role goal/turn lifecycle are within compiled bounds; callers cannot supply a per-spawn timeout or task wall clock.
|
||||
7. The requested sandbox does not exceed role permissions.
|
||||
8. Required write scopes and attachments are valid and contained.
|
||||
9. Maximum depth and ancestor-role policy are satisfied.
|
||||
10. The child route is available for the current credential/endpoint overlay.
|
||||
11. Active-agent target, caller child limit, role concurrency, global resource capacity, and writer capacity are available. Terminal workers release these leases for later sequential delegation.
|
||||
12. No active writer in any session has an overlapping absolute canonical scope; nested session roots cannot bypass a lease.
|
||||
|
||||
Rejected work is recorded in the session audit log where appropriate.
|
||||
|
||||
## Resource groups
|
||||
|
||||
A resource group represents shared capacity rather than a hard-coded model limit. Examples include a subscription, provider rate pool, one local GPU, or a private endpoint.
|
||||
|
||||
```toml
|
||||
[resources.local_gpu_0]
|
||||
lock_key = "gpu:0"
|
||||
max_active = 1
|
||||
```
|
||||
|
||||
Agents consume `resource_units`; providers or models supply a default resource group. Capacity is enforced across sessions for MCP jobs and root sessions.
|
||||
|
||||
## Write isolation, disposition, and integration
|
||||
|
||||
MCP workspace writers require a Git repository and execute in isolated worktrees pinned to a captured base. `write_scope_required = true` requires explicit canonical scopes; setting it to `false` makes omission mean the whole delegated tree (`.`), never an unscoped write. Overlapping canonical leases are rejected across sessions, including nested workspace roots.
|
||||
|
||||
The runner captures text and binary-safe patch artifacts, base fingerprints, changed paths, file modes, hashes, and declared artifacts. A path outside the authorized scope, unsafe member, symlink/special artifact, or patch-boundary error fails the job and taints the session. The canonical checkout remains unchanged.
|
||||
|
||||
Successful results use a monotonic lifecycle. `agent_result` marks a terminal result read and returns either one complete strict JSON value or contiguous text pages. Callers begin at cursor zero and follow `next_cursor` until null. `agents_wait` returns compact lifecycle state and deterministic per-job progress revisions; supplying the exact observed revision map wakes on the first durable change, while bounded result previews are opt-in. Model-visible MCP responses remove supervisor result/event/stderr/socket locations, and generated guidance requires the lifecycle APIs instead of direct state-file reads. After transport recovery, generated guidance requires discovery and inspection of retained runs before replacement admission. The authorized ancestor must explicitly accept or reject a successfully completed result with a reason. Suspended jobs expose partial evidence without pretending it is final and may be continued. Typed provider-limit, transport, malformed-tool, and terminal-turn failures retain the complete upstream error plus partial evidence on that same lineage. If a dead turn-mode host left an active turn in persisted history, its replacement interrupts and settles that exact turn before starting at most one continuation; a terminal result that wins the recovery race remains authoritative. Stopped, failed, and cancelled jobs remain readable evidence but cannot be dispositioned as successful results. Only an accepted writable result may be passed to `agent_patch_integrate`, which rechecks patch identity, base fingerprints, scope, lease, and `git apply --check` before changing the canonical workspace. Reading is not acceptance, and acceptance is not integration.
|
||||
|
||||
There is no progress-checkpoint tool or model-authored liveness deadline. Durable app-server events are the progress record. An authorized controller inspects state or reads a filtered trace when a real dependency requires it; silence alone is not failure.
|
||||
|
||||
If publishing the integrated lifecycle state fails after `git apply`, the runtime attempts to reverse-apply the exact patch before returning the persistence error. If reverse-apply also fails, it durably taints the session when state storage remains writable and always reports that manual workspace recovery is required. If even the taint marker cannot be persisted, the error says so explicitly; the caller must stop using the session rather than assuming durable containment.
|
||||
|
||||
Native participants do not pass through this worktree/result supervisor. All featured bundled native participants are read-only; third-party profiles with native writers receive warnings and must rely on Codex sandboxing and manual review.
|
||||
|
||||
## Gateway lifecycle
|
||||
|
||||
Snapshots that use one or more `switchyard` providers receive a deterministic `routes.toml`. The gateway identity hashes only the selected clients, targets, and route semantics, so profiles with equivalent transport/model route sets can share one process without sharing their immutable profile snapshots. At runtime:
|
||||
|
||||
- The gateway bind setting is restricted to an unbracketed IPv4 or IPv6 loopback literal. Switchyard 0.2.0 accepts an IP address, not a hostname, and the generated ingress has no client-authentication layer.
|
||||
- A port is selected deterministically from the configured range using the gateway hash.
|
||||
- Credentials required by the snapshot are loaded from `credentials.env` or the process environment.
|
||||
- Switchyard starts in a separate process group with declared provider secrets and the filtered non-credential environment.
|
||||
- The gateway state records the exact executable version. Reuse requires that version to still match the configured binary; changing it recycles the gateway and its dependent root app-server so version-scoped transport behavior cannot drift underneath a durable thread.
|
||||
- Health and route advertisement can be checked by `doctor --live`.
|
||||
- Sessions whose snapshots have the same gateway hash reuse the gateway, even when their profile or snapshot hashes differ.
|
||||
- Different gateway hashes use independent instances and may run concurrently.
|
||||
- Per-session injected route faults are immutable supervisor admission overlays. They do not alter route discovery, restart the shared process, or affect another session using that gateway.
|
||||
- `gateway stop-idle` uses side-effect-free validated state reads and retains an instance for every
|
||||
live root controller, root app-server, worker runner, or worker app-server, including detached and
|
||||
recoverable suspended hosts. It starts the configured idle interval at the latest canonical
|
||||
consumer release or gateway use timestamp; filesystem modification time is not a lifecycle clock.
|
||||
- Route telemetry is interpreted at this integration boundary from Codex events and the exact
|
||||
Switchyard routing log. A worker records only the normalized observation.
|
||||
|
||||
A profile containing only built-in Codex or direct custom providers starts no gateway.
|
||||
|
||||
## Durable state
|
||||
|
||||
Default XDG paths:
|
||||
|
||||
```text
|
||||
~/.config/codex-mmo/
|
||||
active-profile
|
||||
credentials.env
|
||||
settings.toml
|
||||
catalog.d/
|
||||
profiles.d/
|
||||
tool-mcp.d/
|
||||
|
||||
~/.local/state/codex-mmo/
|
||||
snapshots/
|
||||
sessions/
|
||||
jobs/
|
||||
gateways/
|
||||
evaluations/
|
||||
backups/
|
||||
```
|
||||
|
||||
State uses atomic file replacement and a process-shared file lock for critical admissions. Append-only JSONL records hold an advisory lock across all short-write retries so concurrent records cannot interleave. Job, root, test, and gateway processes use process groups so cancellation and normal lifecycle completion retire descendants. State enumeration validates and returns records without reconciling processes or changing lifecycle fields; runtime-owned inspection/admission paths explicitly reconcile when that behavior is required.
|
||||
|
||||
Within a logical session, `session.json` is the canonical projection of the session and its one
|
||||
immutable run. Exactly one `runs/<run-id>/run.json` mirrors run fields through the same locked
|
||||
persistence operation; additional run inventory is invalid. Runtime admission/control code decides
|
||||
session and admission transitions; the state module validates and publishes the exact records.
|
||||
The root host updates root-owned lifecycle fields, and each isolated worker owns execution
|
||||
transitions only for its own `jobs/<job-id>/metadata.json` plus append-only trace/audit artifacts.
|
||||
Configuration and immutable snapshots are never used as mutable runtime state.
|
||||
|
||||
## Architectural invariants
|
||||
|
||||
- `mmo_runtime.py` owns admission and orchestration policy; providers and runners do not choose
|
||||
delegation, routing, trust, or resource policy.
|
||||
- `mmo_state.py` owns the only canonical session/job publication functions and side-effect-free
|
||||
validated record readers. `mmo_runtime.py` owns stale-host reconciliation and all session lifecycle
|
||||
policy. Session/run mirror ordering remains explicit at the transition site because terminal
|
||||
retirement must mirror the active run before clearing its ID.
|
||||
- One `WorkerRunner` owns each worker process's stop signal and app-server client. Root and worker
|
||||
action implementations remain separate because their lifecycle and native-thread responsibilities
|
||||
differ.
|
||||
- Local control transport owns framing only. Authorization and compare-and-swap admission happen in
|
||||
the runtime before a runner executes an action.
|
||||
- Provider-specific telemetry interpretation stays in the gateway/provider integration boundary;
|
||||
the scheduling kernel remains model- and provider-name agnostic.
|
||||
- Immutable profile snapshots and generated homes are inputs to a session, never mutable runtime
|
||||
state or an alternate session registry.
|
||||
|
||||
## Architectural non-goals
|
||||
|
||||
Codex MMO is deliberately not a distributed scheduler, multi-tenant service, arbitrary workflow
|
||||
language, general plugin runtime, provider catalog replacement, or persistence platform. Profile
|
||||
packs are data, not executable extensions. Provider and Tool MCP seams exist because multiple real
|
||||
integrations are supported; internal runtime components use direct calls rather than speculative
|
||||
factories, dependency containers, event buses, or remote boundaries.
|
||||
|
||||
## Failure model
|
||||
|
||||
- A worker host that exits without a terminal update is marked `suspended`; partial evidence is materialized and the persisted thread may be continued.
|
||||
- A root TUI/client disconnect detaches from the still-live app-server host; descendants and the root goal continue. A controller that exits with a recorded thread becomes recoverable, while a host that exits before any thread identity is recorded fails admission truthfully.
|
||||
- Worker transport recovery resumes only its exact recorded thread. Root recovery resumes the current canonical generation or records a newer eligible top-level generation already created by the attached TUI; it never scans rollout files, searches outside the session-isolated app-server index, or substitutes a new MMO session/run. A replaced turn-mode worker interrupts and observes an orphaned in-progress turn before launching one continuation, while a concurrently completed turn remains deliverable.
|
||||
- Resume restores a stopped content-addressed gateway before rewriting generated homes and reconciles dead current-run workers to suspended partial evidence before returning control.
|
||||
- Graceful stop preserves results when agents finish in time and otherwise retains partials before bounded termination. Immediate cancellation remains distinct and is never described as graceful finalization.
|
||||
- Provider startup failures preserve logs and a failed gateway record.
|
||||
- Contract, command-correlation, artifact-correlation, and literal-evidence failures produce a failed job in strict mode and a warning in advisory mode.
|
||||
- Unavailable optional worker routes produce typed admission failures without creating jobs; an unavailable root route blocks session creation.
|
||||
- A completed isolated writer cannot affect the canonical workspace until explicit accepted integration succeeds.
|
||||
- Output, completed messages, bounded empirical tool observations, and isolated patches are retained when available even without a valid final result.
|
||||
- Profile changes cannot affect active snapshots.
|
||||
- Live model inventory changes are reported, not silently interpreted.
|
||||
|
||||
## Performance model
|
||||
|
||||
The design favors bounded process isolation and durable correctness over an in-process agent framework. Native agents are available when lower launch overhead matters. MCP is used when enforcement and isolation have greater value. Profiles can combine both rather than forcing every task through the more expensive path.
|
||||
+222
@@ -0,0 +1,222 @@
|
||||
# Route and model catalog
|
||||
|
||||
The generation-8 catalog separates what earlier designs collapsed into “provider.” A model binding identifies both a model and an executable access route; it does not rely on a hidden default provider.
|
||||
|
||||
## Identity model
|
||||
|
||||
A route records:
|
||||
|
||||
- `api_operator`: who operates the API;
|
||||
- `access_product`: the subscription, API, gateway, or local service used for access;
|
||||
- `wire_protocol`: the protocol sent on the wire;
|
||||
- `billing_mode`: subscription, API, local, or catalog-only;
|
||||
- `base_url` and `credential_envs` when applicable;
|
||||
- driver, resource group, transport modalities, and tool behavior.
|
||||
|
||||
A model records its `maker`, `route`, and exact `upstream_id` independently. Its catalog key is:
|
||||
|
||||
```text
|
||||
<exact-route-key>__<normalized-upstream-id>
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
opencode_go_openai_chat__deepseek_v4_pro
|
||||
opencode_zen_openai_chat__deepseek_v4_pro
|
||||
openrouter_openai_chat__deepseek_deepseek_v4_pro
|
||||
zai_coding_responses__glm_5_3
|
||||
```
|
||||
|
||||
Those are different bindings even when two routes reach the same maker/model. Validation requires the key prefix to match `model.route` and rejects duplicate `(route, upstream_id)` identities. Profile binding accepts only the exact, case-sensitive key. It never accepts a raw upstream ID, a prior key spelling, or a search alias.
|
||||
|
||||
Catalogs from any other MMO generation and obsolete provider-qualified keys are invalid. There is no migration reader, lookup fallback, automatic rewrite, or compatibility alias.
|
||||
|
||||
OpenRouter itself publishes some exact upstream IDs beginning with `~` for rolling targets. MMO inventories those IDs literally and records their actual maker; it does not make them aliases for another MMO key. Selecting one still requires its exact route-qualified catalog key and accepts the provider-owned rolling semantics.
|
||||
|
||||
## Release baseline
|
||||
|
||||
The reviewed baseline is current through **2026-08-23** (each inventory retains
|
||||
its own exact review date):
|
||||
|
||||
| Inventory | Records | Notes |
|
||||
|---|---:|---|
|
||||
| Installed Codex/ChatGPT catalog | 6 | Built-in models; account visibility remains authoritative |
|
||||
| OpenCode Go | 29 | Complete public listing joined to reviewed capability evidence |
|
||||
| OpenCode Zen | 64 | Complete public listing; unsupported Google-native transport stays catalog-only |
|
||||
| OpenRouter | 422 | Complete reviewed text-input/text-output inventory |
|
||||
| Z.AI General API | 35 | Includes chat, media, OCR, and hosted-service entries |
|
||||
| Z.AI Coding Plan | 3 | Current Coding Plan records, including executable GLM-5.3 Responses |
|
||||
| Local deployment | 1 | Project-capped Qwen3.5-9B llama.cpp route |
|
||||
| **Total** | **560** | **21 routes, 9 resource groups, 462 agent-compatible models** |
|
||||
|
||||
“Supported by an inventory” is not the same as “agent compatible.” Media generators, hosted services, transports the installed driver cannot preserve, and announced-but-unavailable models remain visible without being executable agent bindings.
|
||||
|
||||
The canonical source is [config/upstream-inventory.json](../config/upstream-inventory.json). Every external inventory—large or small—uses the same envelope under [config/inventory-snapshots](../config/inventory-snapshots): source captures, fingerprint fields, exact record set, normalized digest, review date, and discovery policy. OpenRouter's reviewed model-to-endpoint selections are part of that declarative discovery policy; the adapter contains no portfolio-specific endpoint map. `scripts/generate_catalog.py` regenerates [config/catalog.toml](../config/catalog.toml) deterministically; release validation fails on drift.
|
||||
|
||||
OpenCode Go transport selection uses the explicit provider endpoint table as its authority and Models.dev for capability metadata. The 2026-08-23 sources disagree for four Qwen entries: Models.dev selects its OpenAI-compatible default adapter while the Go table explicitly assigns `/messages` and `@ai-sdk/anthropic`. Each record retains both values, the selected value, authority, and disagreement flag in `protocol_resolution`; that evidence participates in the fingerprint. Live verification reads OpenCode's current documentation branch while the reviewed snapshot retains an immutable commit capture, so later convergence or divergence cannot be hidden by the pinned artifact.
|
||||
|
||||
OpenCode currently publishes separate peak and off-peak rates for DeepSeek V4 Flash and Pro. Both rows and labels are retained in snapshot evidence. The scalar catalog cost fields are omitted for those bindings because selecting either time tier as a universal price would be false. The same rule applies to context tiers. A complete dash-valued provider row, such as `ox-alpha-free`, is retained as explicit unpriced evidence rather than converted to zero. Models.dev pricing is not substituted for an absent, dash-valued, or tiered provider billing row. Five live Go IDs (`glm-5`, `kimi-k2.5`, `mimo-v2-omni`, `mimo-v2-pro`, and `qwen3.5-plus`) currently have Models.dev capability evidence but no row in the provider endpoint/pricing table; those gaps remain explicit instead of being filled by inferred provider claims.
|
||||
|
||||
## Inventory evidence and uncertainty
|
||||
|
||||
The snapshot fingerprint covers IDs, canonical slugs, pricing, limits, modalities, tools, reasoning, structured output, supported parameters, deprecation, and endpoint metadata. Discovery does not infer a capability from a new model name.
|
||||
|
||||
Known release-time uncertainties remain explicit:
|
||||
|
||||
- the exact authenticated Codex account catalog must be checked on the target installation;
|
||||
- OpenCode Go's `hy3-preview` live ID has only listing evidence and remains non-agent-compatible;
|
||||
- five live OpenCode Go IDs lack a current provider endpoint/pricing-table row and therefore use only the transport/capability evidence actually available; `ox-alpha-free` has a complete provider row whose price cells are dashes and is deliberately unpriced;
|
||||
- four current Go Qwen protocol records have a captured provider-source disagreement, and all Go source/protocol metadata can drift independently of the listing;
|
||||
- two live OpenCode Zen IDs (`deepseek-v4-flash-free` and `laguna-s-2.1-free`) are present in the live listing and Models.dev but absent from the provider documentation table; they remain executable with `live-undocumented` availability and no inferred scalar price;
|
||||
- Z.AI Coding Plan availability requires the user’s Coding Plan credential;
|
||||
- role-specific performance claims require the bundled live evaluations, not catalog metadata.
|
||||
|
||||
## Maker, route, gateway, and serving provider
|
||||
|
||||
These dimensions answer different questions:
|
||||
|
||||
| Dimension | Example | What it tells you |
|
||||
|---|---|---|
|
||||
| Maker | `deepseek` | Who created the model family |
|
||||
| API operator | `opencode` | Who receives the API request |
|
||||
| Access product | `opencode_go` | Which plan or service is consumed |
|
||||
| Wire protocol | `openai_chat` | Which request/response contract is used |
|
||||
| Gateway | Switchyard | Which local adapter translated Codex traffic |
|
||||
| Serving provider | `parasail/fp8` | Which downstream inference host actually served an aggregator request, when reported |
|
||||
|
||||
Model diversity, access diversity, and infrastructure diversity must therefore be evaluated separately. Sending two requests to OpenRouter is not evidence of independent serving infrastructure unless routing policy and response telemetry establish that fact.
|
||||
|
||||
## OpenRouter policy and telemetry
|
||||
|
||||
An OpenRouter model may declare a typed `route_policy`:
|
||||
|
||||
```toml
|
||||
[models.openrouter_openai_chat__deepseek_deepseek_v4_pro.route_policy]
|
||||
only = ["parasail/fp8"]
|
||||
allow_fallbacks = false
|
||||
require_parameters = true
|
||||
data_collection = "deny"
|
||||
zdr = true
|
||||
quantizations = ["fp8"]
|
||||
```
|
||||
|
||||
The compiler emits the policy mechanically and opts the route into OpenRouter's current `X-OpenRouter-Metadata: enabled` response metadata. Runtime telemetry keeps the requested route/model and policy separate from any selected serving provider or endpoint that survives the full OpenRouter → Switchyard → Codex event path. It ignores unselected provider candidates. OpenRouter's router `attempt` is recorded as a fallback attempt; transport retries are counted only from explicit retry fields. Missing provider, endpoint, retry, or fallback observations remain unknown rather than being inferred from the model maker or retry count. Bundled OpenRouter roles pin an allowlist and disable fallback; route failure is reported instead of silently changing infrastructure.
|
||||
|
||||
Switchyard 0.2.0 preserves buffered same-format response extensions but its streaming translation does not expose arbitrary provider metadata as a durable routing-log field. Authenticated end-to-end inference must therefore prove which metadata reaches the installed Codex JSON event stream. Until it does, `route_telemetry.complete = false` and the release evaluation gate fails; a requested endpoint policy is not reported as an observed serving endpoint.
|
||||
|
||||
## Z.AI routes
|
||||
|
||||
Z.AI General API and Z.AI Coding Plan are distinct access products with distinct credentials:
|
||||
|
||||
```text
|
||||
zai_general_openai_chat -> ZAI_API_KEY
|
||||
zai_coding_responses -> ZAI_CODING_API_KEY
|
||||
```
|
||||
|
||||
The bundled GLM-5.3 roles bind exactly `zai_coding_responses__glm_5_3`. That route uses the Coding Plan's dedicated Codex/Responses base `https://api.z.ai/api/v1`; Switchyard appends `/responses`. The separate OpenAI-compatible Chat route uses `https://api.z.ai/api/coding/paas/v4`. There is no fallback between those paths or to the General API. The General API remains catalogued for operators who independently have that access, including catalog-only media/service endpoints. The Coding Plan Anthropic-compatible route remains catalog-only because the installed Switchyard transport cannot emit its documented Bearer-auth form faithfully.
|
||||
|
||||
## Transport conservatism
|
||||
|
||||
Model capability and complete-route capability are both required. A model may support images upstream while a particular adapter path is text-only. Such a route advertises only the modalities proven through the complete path. The visual profile uses a ChatGPT-backed Codex route whose image transport is validated end to end; rebinding it to a text-only Go, Zen, OpenRouter, or Z.AI path is rejected.
|
||||
|
||||
Reasoning levels are likewise route evidence, not guesses. The internal `none` value means “omit the selector and use upstream defaults”; it does not claim that the model performs no reasoning.
|
||||
|
||||
## Layering and local extensions
|
||||
|
||||
Catalog resolution order is:
|
||||
|
||||
1. bundled `config/catalog.toml`;
|
||||
2. user fragments in `~/.config/codex-mmo/catalog.d/*.toml`, lexically ordered;
|
||||
3. an optional profile-local `catalog.toml`;
|
||||
4. exact `--bind ROLE=MODEL_KEY` overrides.
|
||||
|
||||
Later fragments may replace exact keys, after which the complete catalog is revalidated. A fragment may override only a resource, or define full generation-8 route/model records. For example:
|
||||
|
||||
```toml
|
||||
schema_version = 8
|
||||
|
||||
[resources.private_pool]
|
||||
lock_key = "private:engineering"
|
||||
max_active = 2
|
||||
description = "Private inference capacity"
|
||||
|
||||
[routes.private_openai_chat]
|
||||
name = "Private OpenAI-compatible service"
|
||||
api_operator = "example"
|
||||
access_product = "private_endpoint"
|
||||
wire_protocol = "openai_chat"
|
||||
billing_mode = "api"
|
||||
driver = "switchyard"
|
||||
base_url = "https://models.example.test/v1"
|
||||
credential_envs = ["PRIVATE_MODELS_API_KEY"]
|
||||
resource_group = "private_pool"
|
||||
transport_modalities = ["text"]
|
||||
tool_calling = true
|
||||
supports_custom_tools = true
|
||||
parallel_tool_calls = false
|
||||
max_retries = 1
|
||||
|
||||
[models.private_openai_chat__example_coder]
|
||||
display_name = "Example Coder"
|
||||
description = "Reviewed private coding model"
|
||||
maker = "example"
|
||||
route = "private_openai_chat"
|
||||
upstream_id = "example-coder"
|
||||
kind = "chat"
|
||||
availability = "current"
|
||||
agent_compatible = true
|
||||
context_window = 131072
|
||||
modalities = ["text"]
|
||||
output_modalities = ["text"]
|
||||
reasoning_levels = ["none"]
|
||||
default_reasoning = "none"
|
||||
tool_calling = true
|
||||
parallel_tool_calls = false
|
||||
structured_output = false
|
||||
supports_reasoning_summaries = false
|
||||
resource_group = "private_pool"
|
||||
```
|
||||
|
||||
Do not set `agent_compatible = true` until tool behavior, output limits, modality preservation, and the exact driver/protocol path have been tested.
|
||||
|
||||
`tool_calling` describes ordinary schema-defined function tools. OpenAI Responses
|
||||
custom tools are a distinct capability: set `supports_custom_tools = false` for
|
||||
an otherwise tool-capable binding whose endpoint rejects them. With Codex 0.149,
|
||||
that keeps function-based shell and MCP tools available while omitting the
|
||||
free-form `apply_patch` tool. The field defaults to `tool_calling`; it cannot be
|
||||
true when `tool_calling` is false.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
codex-mmo catalog summary
|
||||
codex-mmo catalog routes
|
||||
codex-mmo catalog models --agent-compatible
|
||||
codex-mmo catalog models --inventory opencode-go
|
||||
codex-mmo catalog models --inventory opencode-zen
|
||||
codex-mmo catalog models --inventory openrouter
|
||||
codex-mmo catalog model zai_coding_responses__glm_5_3
|
||||
codex-mmo catalog resources
|
||||
codex-mmo catalog inventory
|
||||
codex-mmo catalog verify
|
||||
```
|
||||
|
||||
Compare the release fingerprint with live sources and the authenticated Codex installation:
|
||||
|
||||
```bash
|
||||
codex-mmo catalog verify --remote --codex
|
||||
```
|
||||
|
||||
Refresh observations and optionally install a conservative Codex overlay:
|
||||
|
||||
```bash
|
||||
codex-mmo catalog refresh
|
||||
codex-mmo catalog refresh --install-codex-overlay
|
||||
```
|
||||
|
||||
Known IDs retain reviewed metadata. Newly observed IDs are reported but not made agent compatible. Stored credentials are sent automatically only to reviewed official origins; custom discovery URLs require explicitly supplied credentials and cross-origin redirects are rejected.
|
||||
|
||||
## Cost metadata
|
||||
|
||||
Costs are informational and route-specific. When the source supports it, the catalog records base input, cache read, cache creation/write, and output prices independently. Dynamic, tiered, fixed-request, media, or separate reasoning charges are not flattened into a misleading scalar price. Runtime evaluation maintains separate ledgers for subscription usage, local resource usage, and API cost; it does not add unlike units into one “total cost.”
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
# Command-line interface
|
||||
|
||||
Codex MMO has one Python command surface exposed through two installed entrypoints:
|
||||
|
||||
- `codex-mmo` is the primary user command. With no control command it creates and attaches to a new interactive MMO session. It also accepts every control command documented below.
|
||||
- `codex-mmoctl` exposes the same control commands but never inserts an implicit `run`. It is useful when a script should fail instead of accidentally starting an interactive session.
|
||||
|
||||
Run `codex-mmo --help` for the complete top-level map and `codex-mmo COMMAND --help` for command-specific options. Help and version reporting do not initialize profiles, gateways, providers, or session state.
|
||||
|
||||
## Command groups
|
||||
|
||||
The command tree follows the runtime concepts users operate:
|
||||
|
||||
| Command | Purpose |
|
||||
|---|---|
|
||||
| `run`, `resume`, `exec` | Start, reattach to, or noninteractively submit root work |
|
||||
| `profile` | List, inspect, validate, compile, select, install, clone, remove, diagnose, or smoke-test profiles |
|
||||
| `tool-mcp` | Inspect and validate operator-owned third-party Tool MCP definitions |
|
||||
| `catalog` | Inspect model/route inventory and run optional live discovery or verification |
|
||||
| `gateway` | Inspect and control local Switchyard gateway processes |
|
||||
| `session` | Inspect and control durable MMO session lifecycle |
|
||||
| `jobs` | Inspect workers and read losslessly paged evidence |
|
||||
| `eval` | Validate, run, inspect, and compare evaluation suites |
|
||||
| `doctor`, `validate` | Diagnose one profile or validate the complete local configuration |
|
||||
| `clean` | Delete only terminal state older than explicit or configured retention thresholds |
|
||||
| `auth` | Run built-in Codex authentication in the configured base Codex home |
|
||||
|
||||
`version` and `--version` print package version 8.0.0. `prompt` prints the reusable orchestration prompt.
|
||||
|
||||
Global `--json`, `--quiet`, and `--debug` flags may appear before or after a control command, provided they occur before an explicit `--` passthrough separator. Everything after that separator belongs to Codex:
|
||||
|
||||
```bash
|
||||
codex-mmo --quiet profile validate adaptive-engineering
|
||||
codex-mmo session list --json
|
||||
codex-mmo run --profile visual-engineering -- --image screenshot.png
|
||||
```
|
||||
|
||||
Abbreviated option names are rejected. A close command or option typo may receive a conservative suggestion.
|
||||
|
||||
## Human and machine output
|
||||
|
||||
Structured commands adapt to stdout:
|
||||
|
||||
- on a terminal, they render compact human-readable tables or labeled details;
|
||||
- in a pipe or redirected file, they emit strict JSON;
|
||||
- `--json` forces strict JSON even on a terminal.
|
||||
|
||||
JSON is deterministic, UTF-8, and rejects non-finite numbers. Human tables may omit optional columns when the terminal is narrow, but identifiers and other required values are never truncated; a labeled block layout is used when necessary. Codex MMO's control output does not use color or animation. The attached stock Codex TUI retains its own terminal behavior.
|
||||
|
||||
Some interfaces are deliberately raw rather than structured. `run` and `resume` attach a TUI, `gateway logs` forwards log text, and `auth` forwards the Codex login interface; these commands reject `--json`. Without `--json`, `exec` prints only the root result. With `--json`, it prints the complete execution record. Scalar commands retain concise text by default and return named JSON objects when `--json` is explicit.
|
||||
|
||||
Examples for automation:
|
||||
|
||||
```bash
|
||||
# JSON is automatic because stdout is a pipe.
|
||||
codex-mmo session list | jq -r '.[].session_id'
|
||||
|
||||
# Force JSON while inspecting it interactively.
|
||||
codex-mmo jobs status JOB_ID --json | jq .
|
||||
|
||||
# Keep requested data separate from diagnostics.
|
||||
codex-mmo catalog verify --remote >report.json 2>diagnostics.log
|
||||
```
|
||||
|
||||
Requested results are written to stdout. Warnings, progress, usage guidance, and errors are written to stderr. Stable, line-oriented stage progress appears only when stderr is an interactive terminal; `--quiet` suppresses that progress without suppressing requested results, warnings, or failures. Redirected and piped commands receive no spinner frames, carriage-return animation, or ANSI decoration.
|
||||
|
||||
Machine-readable failures requested with `--json` are JSON objects on stderr. They contain a stable `error_type` category and message, plus a hint when one is available. `--debug` adds the Python exception type and traceback for unexpected diagnosis; it never changes the exit status or move errors to stdout.
|
||||
|
||||
## Exit status
|
||||
|
||||
| Status | Meaning |
|
||||
|---:|---|
|
||||
| `0` | The requested operation succeeded |
|
||||
| `1` | Validation, execution, lookup, provider, or runtime operation failed |
|
||||
| `2` | Command syntax or an option combination was invalid |
|
||||
| `130` | The command was interrupted with Ctrl-C/SIGINT |
|
||||
| `128 + signal` | A directly represented child process ended from a signal; for example SIGTERM is `143` |
|
||||
| `141` | A downstream pipeline consumer closed stdout (broken pipe/SIGPIPE convention) |
|
||||
|
||||
Validation commands return nonzero when their report does not pass. `jobs wait` returns `1` while requested jobs remain unfinished after its caller wait. A release-mode evaluation returns `1` when an eligible promotion gate fails. The CLI never reports success merely because it was able to print a failure report.
|
||||
|
||||
## Interactive and noninteractive input
|
||||
|
||||
`exec` accepts either one positional prompt or non-empty UTF-8 text from noninteractive stdin:
|
||||
|
||||
```bash
|
||||
codex-mmo exec --profile incident-hypothesis-triage "Analyze this incident"
|
||||
printf '%s\n' "Analyze this incident" | codex-mmo exec --profile incident-hypothesis-triage
|
||||
```
|
||||
|
||||
When stdin is a terminal, omitting the prompt is an immediate usage error; `exec` never waits on an invisible prompt. In CI, cron, and pipelines, an empty or whitespace-only stdin is also an error.
|
||||
|
||||
`--wall-timeout` is an external caller boundary, not a model instruction or execution budget. If reached, the noninteractive client detaches while the durable root thread and evidence remain resumable.
|
||||
|
||||
## Session and worker lifecycle
|
||||
|
||||
Use the lifecycle verbs according to their exact meaning:
|
||||
|
||||
- `session detach` disconnects the client while admitted work continues live.
|
||||
- `session pause` checkpoints evidence, interrupts admitted turns, retires hosts, and releases execution capacity.
|
||||
- `session continue` re-admits and resumes the same paused root and worker threads.
|
||||
- `session compact` compacts the same paused root and returns it to cold pause.
|
||||
- `session stop` requests evidence finalization, waits the operator grace, and retires every host.
|
||||
- `session cancel` terminates immediately while retaining available evidence.
|
||||
|
||||
`resume` reattaches only to a resumable generation-8 session. Supply exactly one session/thread identifier or `--last`; `--all` is valid only with `--last`. `run resume` is rejected because `run` means a deliberately new session.
|
||||
|
||||
Worker results are losslessly paged. Start with cursor zero and follow `next_cursor` until it is null:
|
||||
|
||||
```bash
|
||||
codex-mmo jobs result JOB_ID --json
|
||||
codex-mmo jobs result JOB_ID --cursor NEXT_CURSOR --json
|
||||
```
|
||||
|
||||
Reading evidence does not accept a result or integrate a patch.
|
||||
|
||||
## Configuration and precedence
|
||||
|
||||
Installed defaults come from the runtime's `config/settings.toml`. The user file at `~/.config/codex-mmo/settings.toml` (or the install manifest's configured root) overlays those defaults and rejects unknown or invalid keys. Settings are static inputs to newly compiled state; existing sessions retain their immutable snapshot.
|
||||
|
||||
Profile selection follows this order:
|
||||
|
||||
1. an explicit command `--profile`;
|
||||
2. the profile written by `codex-mmo profile use` in `active-profile`;
|
||||
3. `default_profile` in merged settings.
|
||||
|
||||
For commands that query the installed Codex model catalog, the executable follows explicit `--codex-bin`, then `MMO_CODEX_BIN`, then `codex_bin` in settings. The base home follows explicit `--codex-home`, then `CODEX_HOME`, then `base_codex_home` in settings.
|
||||
|
||||
Live provider-discovery URL overrides follow explicit CLI options, then their corresponding environment variables, then the reviewed catalog endpoint: `MMO_OPENCODE_MODELS_URL`, `MMO_OPENCODE_ZEN_MODELS_URL`, `MMO_OPENROUTER_MODELS_URL`, `MMO_ZAI_MODELS_URL`, and `MMO_ZAI_CODING_MODELS_URL`. Provider credentials are loaded from the protected credentials file and launch environment according to the security contract; they are never printed by normal or debug CLI output.
|
||||
|
||||
The installer follows absolute `XDG_DATA_HOME`, `XDG_CONFIG_HOME`, and `XDG_STATE_HOME` values, plus the project-specific `XDG_BIN_HOME`. Empty or relative values use documented per-user fallbacks. See [Installation](INSTALLATION.md) for paths and [Troubleshooting](TROUBLESHOOTING.md) for recovery guidance.
|
||||
|
||||
## Destructive operations
|
||||
|
||||
`clean` preserves its established noninteractive semantics and never prompts. It only considers terminal jobs and sessions older than the selected thresholds; active work is not eligible. Preview the exact thresholds and eligible counts first:
|
||||
|
||||
```bash
|
||||
codex-mmo clean --dry-run --json
|
||||
codex-mmo clean --job-days 30 --session-days 90 --json
|
||||
```
|
||||
|
||||
Profile removal, job/session cancellation, stop, installer replacement, and uninstaller purge remain explicit commands or flags. Scripts should inspect their exit status and stderr rather than parsing human prose.
|
||||
@@ -0,0 +1,186 @@
|
||||
# Profile evaluation
|
||||
|
||||
The generation-8 evaluation format asks whether a composition improves a workload, not whether workers merely returned. Every variant starts from the same fixture and hidden trial mutation, is scored by topology-neutral outcomes, and records coordination diagnostics separately.
|
||||
|
||||
Suites from any other MMO generation are rejected. There is no evaluation migration or compatibility reader.
|
||||
|
||||
## Bundled suites
|
||||
|
||||
| Suite | Profile | Distinguishing workload |
|
||||
|---|---|---|
|
||||
| `adaptive-change` | `adaptive-engineering` | Decomposable versus tightly coupled changes and delegation selectivity |
|
||||
| `codex-harness` | `codex-harness-team` | Native context isolation, homogeneous peers, and a fresh supervised critic |
|
||||
| `debugging-confidence` | `high-confidence-debugging` | Misleading symptoms, independent reproduction, hidden regressions, correction cycles |
|
||||
| `research-currentness` | `research-backed-engineering` | Conflicting current sources, authority, re-fetching, and source-to-code fidelity |
|
||||
| `security-assurance` | `secure-change` | Vulnerability recall, false positives, empirical reproduction, remediation validation |
|
||||
| `visual-conformance` | `visual-engineering` | Real browser screenshots, responsive behavior, pixel/edge evidence, accessibility |
|
||||
| `contract-refactoring` | `contract-first-refactoring` | Hidden invariants and plausible-but-incompatible implementations |
|
||||
| `incident-triage` | `incident-hypothesis-triage` | Noisy evidence, competing causes, route outage, time to supported diagnosis |
|
||||
| `access-efficient` | `access-efficient-escalation-lab` | Quality versus local/API/scarce-tier consumption |
|
||||
| `route-resilience` | `route-resilience-lab` | Typed credential-loss, rate-limit, and timeout faults; serving-provider identity; fallback behavior |
|
||||
| `bounded-research` | `bounded-research-organization-lab` | Flat versus bounded hierarchical research and handoff loss |
|
||||
| `competing-implementations` | `competing-implementations-lab` | Two empirical candidates versus one after integration effort and cost |
|
||||
|
||||
There is no shared “engineering core” that accidentally rewards one topology. Each suite is workload-specific and declares its own neutral outcome contract.
|
||||
|
||||
## Matched variants
|
||||
|
||||
Every suite contains, as applicable:
|
||||
|
||||
- the configured root with delegation disabled;
|
||||
- the strongest task-specific single-agent candidate;
|
||||
- one single-agent candidate for each available access product;
|
||||
- the configured root plus its highest-value worker;
|
||||
- the complete profile;
|
||||
- one `full_without_worker` ablation for every reachable worker.
|
||||
|
||||
Unavailable access-product controls are recorded as unavailable rather than rebound to a different route. Each pruned topology removes spawn and control authority to unreachable roles, regenerates its immutable guidance/tool surface, and recomputes exact weighted-resource ceilings through the same derivation used by profile resolution. A comparison is meaningful only when fixture, hidden mutation, task, trial index, and release baseline match.
|
||||
|
||||
Each task runs three development trials by default and five release trials with `--trial-mode release`. Negative controls test whether the root avoids delegation when launch cost exceeds expected value. Difficulty bands distinguish easy tasks from the hard cases a profile claims to improve.
|
||||
|
||||
## Hidden work
|
||||
|
||||
Suite directories may include:
|
||||
|
||||
```text
|
||||
fixture/ public starting repository
|
||||
holdout/TASK_ID/ validation-only files installed for that task
|
||||
mutations/TASK_ID/*.patch
|
||||
```
|
||||
|
||||
For a given task/trial, the evaluator deterministically chooses and applies the same hidden mutation to every matched variant. Holdout files are copied only into the trial workspace and are hash-recorded in the run. Symlinks, special files, escaping paths, and non-patch mutation members fail validation.
|
||||
|
||||
This is concealment from the evaluated prompt, not a claim of cryptographic secrecy from a user who owns the package.
|
||||
|
||||
## Suite shape
|
||||
|
||||
```toml
|
||||
schema_version = 8
|
||||
id = "example"
|
||||
profile = "adaptive-engineering"
|
||||
name = "Example workload"
|
||||
description = "A matched, measurable task."
|
||||
fixture = "fixture"
|
||||
development_trials = 3
|
||||
release_trials = 5
|
||||
|
||||
[promotion]
|
||||
primary_metric = "success_rate"
|
||||
direction = "higher"
|
||||
strongest_success_tolerance = 0.02
|
||||
minimum_relative_improvement = 0.10
|
||||
minimum_absolute_improvement = 0.05
|
||||
worker_minimum_success_contribution = 0.02
|
||||
worker_minimum_metric_contribution = 0.10
|
||||
no_regression_higher_metrics = []
|
||||
no_regression_lower_metrics = []
|
||||
require_complete_api_cost = true
|
||||
|
||||
[[variants]]
|
||||
id = "configured-root"
|
||||
purpose = "Configured root with delegation disabled."
|
||||
topology = "root_only"
|
||||
comparison_class = "configured_root_alone"
|
||||
|
||||
[[variants]]
|
||||
id = "full-profile"
|
||||
purpose = "Complete composition."
|
||||
topology = "full"
|
||||
comparison_class = "full_profile"
|
||||
|
||||
[[tasks]]
|
||||
id = "repair"
|
||||
description = "Repair one bounded defect."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "hard"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 1800
|
||||
validation_timeout_seconds = 120
|
||||
prompt = "Repair the defect and end with EVAL_DONE."
|
||||
images = ["reference.png"]
|
||||
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_DONE"]
|
||||
forbidden_patterns = ["UNVERIFIED"]
|
||||
validation_commands = ["python -m unittest -v"]
|
||||
|
||||
[tasks.orchestration_assertions]
|
||||
min_peak_mcp_workers = 1
|
||||
max_jobs = 3
|
||||
min_result_acceptance_rate = 1.0
|
||||
max_contract_failures = 0
|
||||
max_observed_mcp_wait_ratio = 0.8
|
||||
```
|
||||
|
||||
`outcome_assertions` are valid for every topology. `orchestration_assertions` are diagnostics for variants where the referenced roles exist; the compiler rejects unknown roles and impossible declarations rather than encoding a general expression language.
|
||||
|
||||
`wall_timeout_seconds` is an evaluator boundary, not a model instruction or profile execution budget. If it expires, the noninteractive root first detaches with its durable thread and partial evidence; the evaluator immediately performs a full stop before installing holdouts or running validators, so no detached worker can continue mutating the trial workspace. The task fails visibly and the retained session/job evidence remains available for diagnosis.
|
||||
|
||||
Positive decomposable tasks use `required_agents` for the profile's intended MCP contributions. A role pruned from a matched root-only or ablation variant is reported as not applicable; a reachable required role that was never launched fails the diagnostic. Negative controls retain their workload-specific caps and forbidden roles so generated delegation guidance does not turn atomic work into mandatory fan-out.
|
||||
|
||||
The finite orchestration fields are `required_agents`, `forbidden_agents`, `min_peak_mcp_workers`, `max_jobs`, `min_result_acceptance_rate`, `max_contract_failures`, and `max_observed_mcp_wait_ratio`.
|
||||
|
||||
## Promotion gates
|
||||
|
||||
Labs always report `experimental_lab`; they are never assigned a superiority verdict. They still publish `hypothesis_passed` and every underlying check, so experimental maturity is not an exemption from measurement.
|
||||
|
||||
A featured full profile passes only when all configured checks pass:
|
||||
|
||||
- it has nonzero task success;
|
||||
- its success rate is within two percentage points of the strongest executed root/single-agent/access-product control;
|
||||
- it improves the declared primary metric over its declared configured-root or strongest-single baseline by at least 10% relative or five absolute percentage points;
|
||||
- it has no write-scope conflict, contract failure, or incomplete route telemetry;
|
||||
- every observed API cost category required by policy is complete;
|
||||
- configured safety/quality metrics do not regress;
|
||||
- every reachable worker has an ablation and contributes at least two success points or 10% relative improvement in the target metric;
|
||||
- at least one accessible service-specific control actually executes.
|
||||
|
||||
A composition is not promoted because it is cheaper while failing. Conversely, a large council is not promoted merely because it uses more models.
|
||||
|
||||
Relative improvement from an exact zero baseline has no finite percentage. Evaluation records `relative = null` and `relative_unbounded = true` for a positive improvement from zero, applies the relative gate explicitly, and never writes non-standard JSON `Infinity`.
|
||||
|
||||
## Recorded evidence
|
||||
|
||||
Run records include:
|
||||
|
||||
- final patterns, validation commands, numeric outcome metrics, exit status, and elapsed time;
|
||||
- exact profile/snapshot/model/route identities and typed route-fault injections;
|
||||
- worker lineage, task kinds, retries, failures, cancellations, contradictions, and contract results;
|
||||
- result reads, accept/reject decisions, patch integrations, integration corrections, and taint;
|
||||
- peak and overlapping MCP execution, first-useful-result timing, explicit wait intervals, and root activity events observed while workers run;
|
||||
- requested OpenRouter policy, serving provider/endpoint when reported, and route-telemetry completeness;
|
||||
- token categories, subscription request units, local resource seconds, actual API dollars, and API-equivalent estimates in separate ledgers;
|
||||
- hidden mutation and holdout hashes.
|
||||
|
||||
Observed root activity is event coverage, not mind reading. The runtime can prove that a root emitted non-wait events during worker execution and can measure explicit MCP waits; it cannot prove that every other second was cognitively productive.
|
||||
|
||||
Cost ledgers are intentionally not summed into one aggregate score. Subscription units, local resource time, actual API charges, and API-equivalent estimates answer different questions. Missing price categories make the relevant estimate incomplete rather than silently zero.
|
||||
|
||||
The access-efficient suite declares exact scarce model keys, counts root and worker requests to those keys, and compares the complete profile with the strongest single-agent control. Its hypothesis requires at least 50% lower scarce-tier use and no more than a two-point success loss. “Scarce” is suite policy for this experiment, not an inferred catalog property.
|
||||
|
||||
Evaluation tasks may declare `route_faults = { route_id = "credential_loss" }`. The only fault values are `credential_loss`, `rate_limit`, and `timeout`; the named route becomes unavailable with that immutable reason for the trial. The retired `disabled_routes` field is invalid. This is a typed evaluation injection, not an automatic failover or workflow language.
|
||||
|
||||
## Visual evaluation
|
||||
|
||||
`visual-conformance` uses a real Playwright browser render at multiple widths. The original reference and generated screenshots travel through the same multimodal attachment path used by normal sessions. Validation combines deterministic behavior/accessibility checks with pixel/edge evidence; success cannot be inferred from HTML text or a synthetic image description.
|
||||
|
||||
The evaluator rejects a profile binding whose complete transport cannot preserve image inputs and tool-produced image results.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
codex-mmo eval suites
|
||||
codex-mmo eval validate adaptive-change
|
||||
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change --dry-run
|
||||
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change
|
||||
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change --trial-mode release
|
||||
codex-mmo eval list
|
||||
codex-mmo eval show RUN_ID
|
||||
codex-mmo eval compare RUN_ID_A RUN_ID_B
|
||||
```
|
||||
|
||||
Invalid suites, failed tasks, and unfinished runs return nonzero status while preserving diagnostic records. A development run may complete while missing promotion; release mode also returns nonzero when an eligible featured profile fails its promotion checks. Inspect `promotion.checks` instead of equating task completion with profile efficacy.
|
||||
|
||||
## Interpretation
|
||||
|
||||
Provider quota, authentication, route outage, and serving drift are operational failures, not evidence about model reasoning quality. Report them separately. Do not compare unlike suites or repository states, and do not treat a single development run as release evidence. Featured release claims require the five-trial mode, authenticated inventory verification, complete ledgers, and retained task-level artifacts.
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,312 @@
|
||||
# Installation and Development Replacement
|
||||
|
||||
## Requirements
|
||||
|
||||
- Linux with `/proc` available for reliable zombie detection.
|
||||
- Bash for the installer and installed command wrappers.
|
||||
- Python 3.11 or newer.
|
||||
- Git.
|
||||
- Codex CLI exactly 0.149.0 with the reviewed experimental app-server schema.
|
||||
- Switchyard 0.2.0 for profiles that bind one or more models through a `switchyard` route.
|
||||
- npm only when using `--install-codex`.
|
||||
- Rust/Cargo 1.96.1 or newer only when using `--install-switchyard` (the
|
||||
Switchyard 0.2.0 crate's declared minimum Rust version).
|
||||
- A local model server when a selected profile uses one.
|
||||
|
||||
## Default per-user paths
|
||||
|
||||
```text
|
||||
Install: ~/.local/share/codex-mmo
|
||||
Config: ~/.config/codex-mmo
|
||||
State: ~/.local/state/codex-mmo
|
||||
Binaries: ~/.local/bin
|
||||
```
|
||||
|
||||
The installer creates `catalog.d/`, `profiles.d/`, and `tool-mcp.d/` beneath the configuration root. Tool MCP definitions are operator configuration and are preserved across a current-generation reinstall.
|
||||
|
||||
The installer follows XDG Base Directory 0.8 for `XDG_DATA_HOME`, `XDG_CONFIG_HOME`, and `XDG_STATE_HOME`: an unset, empty, or relative value is ignored and the documented absolute fallback is used. `XDG_BIN_HOME` is a Codex MMO installer extension, not an XDG Base Directory variable; it follows the same absolute-value rule and otherwise falls back to `~/.local/bin`.
|
||||
|
||||
The 8.0.0 release targets Python 3.11 semantics and was tested locally on Python 3.14.7, Codex 0.149.0, Switchyard 0.2.0, Git 2.55, GNU tar 1.35, and Info-ZIP 6.00. The optional Cargo path pins Switchyard 0.2.0 because generated route syntax is verified against that exact release; building that crate requires its declared Rust 1.96.1 minimum (tested locally with Rust/Cargo 1.97.1). An operator-supplied newer Switchyard must pass `doctor --live` (including its native dry run) before use. Codex is deliberately exact-pinned: every session admission, reattachment, and replacement host checks the reported 0.149.0 version, while `doctor --live` additionally requires 401 generated experimental schema files with digest `fcfeaf23728b96ab73916a21302eb7a16629e67ee99f7ee47b60fad6b6e5ee1a`. Any other Codex version is rejected; this development release has no compatibility reader or migration path. Provider services remain authoritative for live model/account availability.
|
||||
|
||||
## Verify the archive before installation
|
||||
|
||||
From beside an extracted source tree, verify both downloaded formats when available:
|
||||
|
||||
```bash
|
||||
python3 codex-multimodel-orchestrator-8.0.0/scripts/verify_release.py \
|
||||
--source-tree codex-multimodel-orchestrator-8.0.0 \
|
||||
codex-multimodel-orchestrator-8.0.0-linux.tar.gz \
|
||||
codex-multimodel-orchestrator-8.0.0-linux.zip
|
||||
```
|
||||
|
||||
The command rejects incomplete archives, extra or missing files, implicit or extra directory entries, digest mismatches, file/directory mode drift, traversal, links, special files, and manifest/source differences.
|
||||
|
||||
## Standard installation
|
||||
|
||||
Inspect installer options without changing the system:
|
||||
|
||||
```bash
|
||||
./install.sh --help
|
||||
```
|
||||
|
||||
```bash
|
||||
tar -xzf codex-multimodel-orchestrator-8.0.0-linux.tar.gz
|
||||
cd codex-multimodel-orchestrator-8.0.0
|
||||
./install.sh
|
||||
```
|
||||
|
||||
Install optional tools when missing:
|
||||
|
||||
```bash
|
||||
./install.sh --install-codex --install-switchyard
|
||||
```
|
||||
|
||||
Custom paths and binary names:
|
||||
|
||||
```bash
|
||||
./install.sh \
|
||||
--install-root /opt/user/codex-mmo \
|
||||
--config-root "$HOME/.config/codex-mmo" \
|
||||
--state-root "$HOME/.local/state/codex-mmo" \
|
||||
--bin-dir "$HOME/bin" \
|
||||
--codex-bin /usr/local/bin/codex \
|
||||
--switchyard-bin "$HOME/.cargo/bin/switchyard-server"
|
||||
```
|
||||
|
||||
The installer first requires every declared payload file/directory to exist and rejects source symlinks and special files. It then stages and validates the payload before an atomic replacement. A nonempty existing install root is replaced only when its generation-8 ownership manifest identifies that same canonical path; an unrelated or corrupt tree is left untouched. Existing owned installs are moved to timestamped state backups. Existing settings and credentials are preserved only when they already satisfy generation 8. Optional npm/Cargo installers receive a filtered environment, and a zero package-manager exit is accepted only when the requested Codex or Switchyard executable can then be resolved.
|
||||
|
||||
Finish, gracefully stop, or cancel active sessions before replacing the installed development build. A detached run
|
||||
whose workers are still active is not quiescent. Process identity, app-server control sockets,
|
||||
thread-host metadata, and runtime documents are build-specific. Generation 8 deliberately does not
|
||||
adopt older process trees or load older records. Preserve evidence outside the active MMO state root,
|
||||
remove the obsolete runtime state, and start one new generation-8 session from an explicit handoff.
|
||||
|
||||
## PATH
|
||||
|
||||
The installer warns on stderr when the selected binary directory is not already on `PATH`. Ensure
|
||||
the default directory is available in future shells:
|
||||
|
||||
```bash
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
```
|
||||
|
||||
After installation, use `codex-mmo --help` for the user-facing command map. The
|
||||
`codex-mmoctl` entrypoint exposes the same control commands but intentionally fails when no command
|
||||
is given instead of starting an interactive session. See [Command-line interface](CLI.md).
|
||||
|
||||
## Credentials
|
||||
|
||||
Edit:
|
||||
|
||||
```bash
|
||||
${EDITOR:-vi} ~/.config/codex-mmo/credentials.env
|
||||
```
|
||||
|
||||
Bundled routes may use:
|
||||
|
||||
```dotenv
|
||||
ZAI_CODING_API_KEY=
|
||||
OPENCODE_API_KEY=
|
||||
OPENROUTER_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
ANTHROPIC_API_KEY=
|
||||
# Optional only for an explicitly selected Z.AI General API route:
|
||||
ZAI_API_KEY=
|
||||
```
|
||||
|
||||
Unused values should remain empty. File permissions are `0600`.
|
||||
|
||||
Third-party Tool MCP credentials use the same file, but their variable names come from the operator registry rather than a profile pack. See [Tool MCP servers](TOOL_MCP.md) for the registry and per-role grant format.
|
||||
|
||||
OpenCode uses one `OPENCODE_API_KEY` for both Zen and Go. OpenRouter uses `OPENROUTER_API_KEY`. Every bundled GLM-5.3 role uses the Z.AI Coding Plan Responses route and exactly `ZAI_CODING_API_KEY`; `ZAI_API_KEY` is a separate General API credential and is never a fallback. Obsolete credential names are not recognized or rewritten.
|
||||
|
||||
## Built-in ChatGPT/Codex authentication
|
||||
|
||||
```bash
|
||||
codex-mmo auth login
|
||||
codex-mmo auth status
|
||||
```
|
||||
|
||||
The base Codex home defaults to `~/.codex` and is configurable in `settings.toml`. It must expand to an absolute path; relative values are rejected so authentication and the Codex keyring namespace cannot vary with the caller's working directory. Generated homes receive file-backed `auth.json` using `auth_link_mode = "shared"` (the default symlink), `"copy"` (an explicit `0600` per-home copy), or `"none"`. Authentication is never copied into profile packs or snapshots.
|
||||
|
||||
Codex 0.149 scopes keyring entries to the canonical `CODEX_HOME` path. A login stored only in the base home's keyring is therefore not visible from an isolated generated home. Configure `cli_auth_credentials_store = "file"` in the base Codex configuration and log in again before using a built-in profile. `codex-mmo doctor` distinguishes a valid-but-keyring-only base login from file auth that the generated home can actually use.
|
||||
|
||||
## Local model example
|
||||
|
||||
The bundled Qwen profile expects an OpenAI-compatible llama.cpp server on `127.0.0.1:8001` with the stable model alias `qwen3.5-9b`. For a current llama.cpp installation, adapt this command to the local model path:
|
||||
|
||||
```bash
|
||||
llama-server \
|
||||
--model /path/to/qwen3.5-9b.gguf \
|
||||
--alias qwen3.5-9b \
|
||||
--ctx-size 32768 \
|
||||
--jinja \
|
||||
--reasoning off \
|
||||
--no-mmproj \
|
||||
--host 127.0.0.1 \
|
||||
--port 8001
|
||||
```
|
||||
|
||||
The upstream Qwen3.5-9B model card documents a native 262,144-token context, multimodal input, thinking, and tool use. The bundled `llama_cpp_local_openai_chat__qwen3_5_9b` entry is intentionally a smaller deployment contract: text-only, 32,768 total context, 8,192 advertised output, and no reasoning control. The explicit flags above make the current llama.cpp server match that resource-bounded contract; an operator who provisions more context or modalities should define a separately validated catalog entry.
|
||||
|
||||
Keep the endpoint on loopback unless it is separately authenticated and protected. llama.cpp flags can vary by version; its own help output is authoritative. The profile compiler validates declared capabilities, but the live smoke test is what proves the selected GGUF chat template and tool calling are usable.
|
||||
|
||||
## Initial validation
|
||||
|
||||
```bash
|
||||
codex-mmo version
|
||||
codex-mmo profile list
|
||||
codex-mmo catalog inventory
|
||||
codex-mmo tool-mcp validate
|
||||
codex-mmo validate --all-profiles
|
||||
```
|
||||
|
||||
Check the selected profile and provider path:
|
||||
|
||||
```bash
|
||||
codex-mmo doctor --profile adaptive-engineering
|
||||
codex-mmo doctor --profile adaptive-engineering --live
|
||||
codex-mmo doctor --profile adaptive-engineering --live --probe
|
||||
```
|
||||
|
||||
`--probe` performs a real root-model call and may consume quota. For a goal-mode root, the bounded probe exercises the real `update_goal(status="complete")` lifecycle before accepting the exact terminal marker.
|
||||
|
||||
## Live smoke test
|
||||
|
||||
Run from a small disposable Git repository or a repository you are comfortable exposing to the selected providers:
|
||||
|
||||
```bash
|
||||
codex-mmo profile smoke adaptive-engineering --cwd "$PWD"
|
||||
```
|
||||
|
||||
For a hybrid profile, smoke tasks explicitly select native and MCP paths where needed.
|
||||
|
||||
## Switching profiles
|
||||
|
||||
Changing the default affects new sessions only:
|
||||
|
||||
```bash
|
||||
codex-mmo profile use adaptive-engineering
|
||||
codex-mmo
|
||||
```
|
||||
|
||||
Run a different profile once:
|
||||
|
||||
```bash
|
||||
codex-mmo run --profile high-confidence-debugging
|
||||
```
|
||||
|
||||
Running sessions remain pinned to their original immutable snapshot.
|
||||
|
||||
## Installing a profile pack
|
||||
|
||||
```bash
|
||||
codex-mmo profile install ./my-profile
|
||||
codex-mmo profile install ./my-profile.tar.gz
|
||||
codex-mmo profile install ./my-profile.zip
|
||||
```
|
||||
|
||||
Profile archives must contain exactly one `profile.toml`. Unsafe paths, symlinks, hard links, devices, scripts, unsupported files, and excessive nesting are rejected.
|
||||
|
||||
The profile must declare the exact active package version (`8.0.0`). Reinstalling identical content
|
||||
is a no-op. Different current-generation content is rejected unless `--replace` is supplied; replacement
|
||||
is staged, validated, and atomically selected. Profile installation has no upgrade, downgrade, or
|
||||
migration path while this project is in active development.
|
||||
|
||||
A profile pack may refer to a Tool MCP server ID, but it cannot carry a server command or endpoint. Define and review that server first under `~/.config/codex-mmo/tool-mcp.d/`; profile validation rejects unresolved grants.
|
||||
|
||||
Clone a bundled profile before editing:
|
||||
|
||||
```bash
|
||||
codex-mmo profile clone adaptive-engineering my-engineering-team
|
||||
${EDITOR:-vi} ~/.config/codex-mmo/profiles.d/my-engineering-team/profile.toml
|
||||
codex-mmo profile validate my-engineering-team
|
||||
codex-mmo profile use my-engineering-team
|
||||
```
|
||||
|
||||
## Catalog overlays
|
||||
|
||||
Add local models or endpoint overrides without modifying installed files:
|
||||
|
||||
```text
|
||||
~/.config/codex-mmo/catalog.d/20-local.toml
|
||||
```
|
||||
|
||||
Fragments use MMO schema 8 and are merged lexically. A later fragment can override an exact route, model, or resource entry.
|
||||
|
||||
Discover account-specific Codex models:
|
||||
|
||||
```bash
|
||||
codex-mmo catalog refresh --install-codex-overlay
|
||||
```
|
||||
|
||||
The generated overlay is:
|
||||
|
||||
```text
|
||||
~/.config/codex-mmo/catalog.d/90-codex-discovered.toml
|
||||
```
|
||||
|
||||
Remove it to return to the bundled baseline.
|
||||
|
||||
## Replacing the active development build
|
||||
|
||||
Run the current source installer with the same paths only after retiring active work. The installer:
|
||||
|
||||
1. Preserves user settings and credentials.
|
||||
2. Validates the staged release.
|
||||
3. Moves the old install to a state backup.
|
||||
4. Atomically activates the new install.
|
||||
5. Rewrites launchers to the new install path.
|
||||
|
||||
All MMO-owned documents use schema 8 and profiles require version `8.0.0`. The installer does not
|
||||
rename `active-profile`, add retired aliases, parse earlier configuration or runtime records,
|
||||
rewrite model keys, or migrate runtime state. Incompatible configuration must be recreated directly
|
||||
in generation 8. If the preserved selection names a profile that no longer exists, choose a current
|
||||
ID explicitly:
|
||||
|
||||
```bash
|
||||
codex-mmo profile use adaptive-engineering
|
||||
```
|
||||
|
||||
After replacement:
|
||||
|
||||
```bash
|
||||
codex-mmo validate --all-profiles
|
||||
codex-mmo catalog verify --remote --codex
|
||||
```
|
||||
|
||||
Profile snapshots are addressed by content and package metadata. Any runtime document outside the
|
||||
active schema/package generation is rejected at load time, including inspection and resume. Keep
|
||||
historical evidence in an external/manual archive, not in the active sessions, jobs, snapshots, or
|
||||
gateway directories. Canonical current sessions contain exactly one immutable run; terminal sessions
|
||||
cannot be resumed.
|
||||
|
||||
## Uninstallation
|
||||
|
||||
Remove installed binaries and payload while preserving configuration and state:
|
||||
|
||||
```bash
|
||||
codex-mmo-uninstall
|
||||
```
|
||||
|
||||
Remove configuration:
|
||||
|
||||
```bash
|
||||
codex-mmo-uninstall --purge-config
|
||||
```
|
||||
|
||||
Remove state:
|
||||
|
||||
```bash
|
||||
codex-mmo-uninstall --purge-state
|
||||
```
|
||||
|
||||
Remove all three:
|
||||
|
||||
```bash
|
||||
codex-mmo-uninstall --purge-config --purge-state
|
||||
```
|
||||
|
||||
The package never removes Codex, Switchyard, npm, Cargo, provider accounts, or local model files.
|
||||
|
||||
Recursive removal requires the install's valid generation-8 ownership manifest. Explicit configuration, state, or binary roots must exactly match that manifest. When configuration or state is retained, the uninstaller writes a protected ownership record there so a later explicit purge remains possible after the payload itself is gone. A retained record is considered only while the recorded install path is absent; a successful fresh installation removes retained records because its live generation-8 payload manifest becomes authoritative. If the install path has been recreated independently, that live tree must carry its own valid manifest. An arbitrary, corrupt, or unrelated replacement `--install-root` is refused before any deletion.
|
||||
@@ -0,0 +1,237 @@
|
||||
# Optimization and Performance Policy
|
||||
|
||||
Codex MMO optimizes for completed, validated work—not maximum agent count. A model or worker is useful only when it improves at least one of these dimensions without imposing greater integration cost:
|
||||
|
||||
- Wall-clock latency through genuine parallelism.
|
||||
- Capability through specialization or stronger reasoning.
|
||||
- Context isolation for noisy or independent branches.
|
||||
- Confidence through independent verification.
|
||||
- Access-route cost or local-resource efficiency.
|
||||
- Operational control through durable, cancellable jobs.
|
||||
|
||||
## The delegation decision
|
||||
|
||||
The root should evaluate five variables together:
|
||||
|
||||
1. **Difficulty** — how much reasoning or implementation capability is required?
|
||||
2. **Independence** — can the branch proceed without blocking or duplicating root work?
|
||||
3. **Criticality** — is it on the immediate critical path?
|
||||
4. **Verifiability** — how cheaply can the result be checked from primary evidence?
|
||||
5. **Execution economics** — launch overhead, provider cost, latency, resource pressure, and integration burden.
|
||||
|
||||
A strong default decision tree is:
|
||||
|
||||
```text
|
||||
Is the task an immediate critical-path blocker?
|
||||
├── yes: root usually handles it directly
|
||||
└── no
|
||||
├── difficult/high-consequence and independent: flagship peer
|
||||
├── substantive bounded engineering: implementation specialist
|
||||
├── cheap, literal, easily verified evidence: low-cost scout
|
||||
└── no meaningful benefit from delegation: root handles it
|
||||
```
|
||||
|
||||
Difficulty alone does not determine delegation. A hard independent review is an excellent flagship-worker task; a hard architectural decision that blocks all progress normally belongs in the root thread.
|
||||
|
||||
## Root productivity
|
||||
|
||||
The root is an active engineer and integrator. Its generated policy requires:
|
||||
|
||||
1. Identify the immediate critical path.
|
||||
2. Identify independent side work.
|
||||
3. Spawn only useful branches.
|
||||
4. Continue non-overlapping root work immediately.
|
||||
5. Consume results as they become relevant.
|
||||
6. Wait only at a real dependency barrier.
|
||||
7. Reconcile contradictions from primary evidence.
|
||||
8. Review material changes and run integrated validation.
|
||||
|
||||
The runtime records explicit Agent MCP wait events and root events emitted during worker intervals. Evaluation reports wait seconds/ratio, event coverage, worker overlap, and time to first useful result; event coverage is not proof of continuous cognition. A profile that causes the root to spawn and immediately idle is still misconfigured even if worker utilization looks high.
|
||||
|
||||
## Native versus MCP execution
|
||||
|
||||
Native Codex agents and Agent MCP solve different performance problems.
|
||||
|
||||
Use native agents when:
|
||||
|
||||
- Launch latency matters.
|
||||
- The task is small or read-heavy.
|
||||
- Tight `/agent` interaction is useful.
|
||||
- Built-in Codex authentication and native behavior should remain direct.
|
||||
- External scope/contract enforcement is not essential.
|
||||
|
||||
Use MCP when:
|
||||
|
||||
- Exact model/route pinning must be auditable.
|
||||
- The worker is low-trust or inconsistent.
|
||||
- The task is long-running or asynchronous.
|
||||
- Cancellation, durable results, or operator inspection matter.
|
||||
- Nested delegation needs enforceable lineage and budgets.
|
||||
- Workspace writes need isolated worktrees, explicit scopes, and audited patch integration.
|
||||
- Structured output must be validated.
|
||||
|
||||
Hybrid profiles use native first-level branches for lower latency and MCP for strict or nested work. This avoids paying process-launch overhead universally without giving up enforcement where it has value.
|
||||
|
||||
## Weak-model economics
|
||||
|
||||
A cheap model is not automatically efficient. Its net value is:
|
||||
|
||||
```text
|
||||
saved strong-model work
|
||||
- launch and prompt cost
|
||||
- verification cost
|
||||
- contradiction-resolution cost
|
||||
- rework caused by incorrect output
|
||||
```
|
||||
|
||||
A weak or conflict-prone model should therefore receive only tasks that are:
|
||||
|
||||
- Narrow.
|
||||
- Low ambiguity.
|
||||
- Read-only.
|
||||
- Mechanically checkable.
|
||||
- Cheap to repeat or discard.
|
||||
- Unlikely to contaminate architecture or final decisions.
|
||||
|
||||
The bundled Qwen role is restricted to evidence gathering for precisely this reason. The same trust policy can be applied to any model. Low-trust output is evidence to verify, never authority.
|
||||
|
||||
Do not delegate a command the root can execute directly in less time than starting and checking a worker.
|
||||
|
||||
## Parallelism policy
|
||||
|
||||
Concurrency ceilings are limits, not targets.
|
||||
|
||||
Read-heavy work can be parallelized aggressively when branches are independent:
|
||||
|
||||
- Repository mapping.
|
||||
- Independent diagnosis.
|
||||
- Specification extraction.
|
||||
- Review from different perspectives.
|
||||
- Test and log analysis.
|
||||
|
||||
Write-heavy work requires stronger partitioning:
|
||||
|
||||
- Assign disjoint files or components.
|
||||
- Require MCP write scopes and isolated Git worktrees where enforceability matters.
|
||||
- Avoid logically coupled edits even when paths differ.
|
||||
- Keep one root responsible for reading, accepting/rejecting, explicitly integrating, inspecting, and testing patches.
|
||||
- Cancel duplicate branches once one result makes the other unnecessary.
|
||||
|
||||
Too many agents increase context duplication, provider queuing, file conflicts, and synthesis time. Start with two or three useful branches; increase only when evaluation shows a wall-clock or quality benefit.
|
||||
|
||||
## Resource groups
|
||||
|
||||
Resource groups model shared capacity rather than model names. Examples include:
|
||||
|
||||
- A ChatGPT subscription.
|
||||
- An OpenCode Go account.
|
||||
- A Z.AI plan.
|
||||
- A direct API rate pool.
|
||||
- One local GPU.
|
||||
|
||||
Each active MCP role consumes `resource_units`. Admission is atomic across sessions. This prevents a six-agent profile from launching six copies against a one-slot local model server or exhausting one provider while other capacity sits idle.
|
||||
|
||||
Tune group capacity in a user catalog fragment based on measured provider and hardware behavior. Do not encode a physical capacity assumption in profile instructions.
|
||||
|
||||
## Context isolation
|
||||
|
||||
Worker prompts include the delegated objective, role instructions, output contract, workspace context, and parent lineage—not the entire root transcript by default. This reduces token duplication and keeps independent branches independent.
|
||||
|
||||
Good briefs state:
|
||||
|
||||
- Exact scope.
|
||||
- Expected artifact or decision.
|
||||
- Relevant constraints.
|
||||
- Required evidence and validation.
|
||||
- What the worker must not decide.
|
||||
|
||||
Bad briefs paste the complete conversation, ask an agent to “solve everything,” or omit success criteria.
|
||||
|
||||
## Structured results
|
||||
|
||||
Strict contracts reduce synthesis cost by making important outputs predictable. They are particularly valuable for:
|
||||
|
||||
- Low-trust evidence collectors.
|
||||
- Review findings.
|
||||
- Implementation summaries.
|
||||
- Proposal comparisons.
|
||||
- Test reports.
|
||||
|
||||
Keep contracts small. Requiring a large nested schema can consume more model effort than it saves. Full logs and diffs remain as artifacts; the structured result should contain the decision-relevant subset.
|
||||
|
||||
## Contradiction handling
|
||||
|
||||
Never use model voting as a substitute for verification. When agents disagree:
|
||||
|
||||
1. Identify the exact conflicting propositions.
|
||||
2. Locate source code, specifications, tests, logs, or reproducible commands that discriminate between them.
|
||||
3. Run or inspect that evidence.
|
||||
4. Ask a specialist only when the evidence itself requires expert interpretation.
|
||||
5. Record residual uncertainty.
|
||||
|
||||
A stronger model is not automatically correct; a majority is not automatically correct; an independent result is valuable only when its assumptions and evidence are visible.
|
||||
|
||||
## Gateway reuse
|
||||
|
||||
Switchyard runs per resolved transport/model route set. Sessions with equivalent gateway identity can reuse one gateway; different route semantics remain isolated. This gives:
|
||||
|
||||
- Stable model IDs.
|
||||
- No route collisions during profile switching.
|
||||
- Credential minimization.
|
||||
- Reuse without globally exposing unused routes.
|
||||
- Independent shutdown of idle gateways.
|
||||
|
||||
Profiles using only direct Codex routes avoid gateway startup entirely.
|
||||
|
||||
## Reasoning levels
|
||||
|
||||
Use the highest reasoning setting only where its marginal value exceeds latency and token cost.
|
||||
|
||||
Recommended defaults:
|
||||
|
||||
- Root or flagship architect: high to maximum supported level.
|
||||
- Substantive implementation: high.
|
||||
- Mechanical evidence collector: reasoning disabled or lowest reliable mode.
|
||||
- Reviewer: high when subtle correctness is material.
|
||||
- Plan mode: explicitly configured so the client does not silently fall back to a lower preset.
|
||||
|
||||
A profile binding is validated against declared model and route reasoning metadata. Labels are not assumed to be semantically identical across access products; catalog entries record the selector accepted by that exact path.
|
||||
|
||||
## Evaluation-driven tuning
|
||||
|
||||
Use matched variants within the workload-specific suite:
|
||||
|
||||
```bash
|
||||
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change
|
||||
codex-mmo eval run --profile adaptive-engineering --suite adaptive-change --trial-mode release
|
||||
codex-mmo eval compare RUN_ID_A RUN_ID_B
|
||||
```
|
||||
|
||||
Compare:
|
||||
|
||||
- Success rate, workload metrics, and objective validation.
|
||||
- Wall-clock time.
|
||||
- Observed root MCP wait ratio, peak workers, worker overlap, and time to first result.
|
||||
- Actual worker-role/model/route/serving routing and result-acceptance rate.
|
||||
- Worker completion/failure rate.
|
||||
- Result acceptance and contract failures.
|
||||
- Retry and cancellation counts.
|
||||
- Conflicts and out-of-scope warnings.
|
||||
- Token usage plus separate actual-API, API-equivalent, subscription, and local-resource ledgers.
|
||||
- Integration corrections.
|
||||
|
||||
Optimization changes should be justified by these results, not by intuition about model rank.
|
||||
|
||||
## Common anti-patterns
|
||||
|
||||
- Spawning every available model on every task.
|
||||
- Delegating the critical path and immediately waiting.
|
||||
- Giving weak models ambiguous or high-consequence work.
|
||||
- Letting multiple writers edit overlapping logic.
|
||||
- Using the same prompt for every role.
|
||||
- Allowing unbounded nested delegation.
|
||||
- Retrying a failed branch repeatedly without changing context or approach.
|
||||
- Treating structured output as proof of correctness.
|
||||
- Running both Codex Ultra’s autonomous orchestration and an external mesh without explicit accounting.
|
||||
- Routing built-in Codex models through an unnecessary translation layer.
|
||||
- Measuring utilization instead of completed validated work.
|
||||
@@ -0,0 +1,163 @@
|
||||
# Orchestration Backends
|
||||
|
||||
Codex MMO supports three profile-level orchestration modes:
|
||||
|
||||
```toml
|
||||
[coordination]
|
||||
orchestration = "mcp" # or "native" or "hybrid"
|
||||
```
|
||||
|
||||
Each non-root agent declares one or more executable backends compatible with the profile mode.
|
||||
|
||||
## Agent MCP
|
||||
|
||||
Agent MCP runs each participant through an isolated detached runner that privately hosts a persistent Codex app-server thread. Callers use the authenticated local `mmo_mesh` STDIO supervisor; they never connect to app-server or a worker control socket directly.
|
||||
|
||||
### Mechanically enforced properties
|
||||
|
||||
- Exact role, model, route, and reasoning binding.
|
||||
- Caller authentication and durable lineage.
|
||||
- Directed `can_spawn` edges and exact per-target/action `controls` grants.
|
||||
- Maximum depth and ancestor-role rejection.
|
||||
- Reusable active-agent and active-child limits.
|
||||
- Per-role concurrent `max_active` limits; terminal workers release their admission slots for later sequential work.
|
||||
- Global weighted resource limits.
|
||||
- Sandbox ceiling.
|
||||
- Task-kind, task-size, goal/turn lifecycle, token ceilings, warning-only stalls, finalization grace, reasoning-effort allowlist, and attachment admission.
|
||||
- Isolated writer worktrees, explicit scope leasing, patch capture, and conflict rejection.
|
||||
- Structured output-contract, command-evidence, artifact-hash, and literal-task validation.
|
||||
- Explicit result read/accept/reject disposition and root-controlled patch integration.
|
||||
- Result visibility.
|
||||
- Opaque-run compare-and-swap inspect, filtered trace, steer, interrupt, pause, same-thread continue/token extension, detach, full stop, evidence-only finalize, compact, pending-request response, allowed effort changes, and fork.
|
||||
- Codex goal token accounting or clock-free turns; stall intervals warn without interruption and the model never tracks time.
|
||||
- Transport/host recovery on the exact thread, recoverable suspension, graceful stop, immediate cancellation, and descendant cascades.
|
||||
- Persistent prompts, headerless app-server protocol events, terminal history, pending requests, stderr, partial/final result, disposition, patch/artifact, route, metadata, and audit logs.
|
||||
|
||||
### Costs
|
||||
|
||||
- A detached runner, private Unix app-server host, and generated home per active worker. A lost host may be replaced only to resume its exact durable thread.
|
||||
- Higher launch latency than a native subagent.
|
||||
- Separate model context rather than a shared native thread.
|
||||
|
||||
### Appropriate work
|
||||
|
||||
Use MCP for:
|
||||
|
||||
- Low-trust or inconsistent models.
|
||||
- Long-running asynchronous work.
|
||||
- Route/model pinning that must be auditable.
|
||||
- Participant-to-participant delegation.
|
||||
- Work requiring isolated writes and explicit integration.
|
||||
- Strict result contracts.
|
||||
- Jobs that may need live steering, interruption, continuation, compaction, finalization, or independent forking.
|
||||
- Slow or long-horizon jobs whose useful evidence must survive transport/host failure.
|
||||
- Independent branches where context isolation is useful.
|
||||
|
||||
## Codex native subagents
|
||||
|
||||
Native agents are generated as Codex custom-agent files and launched through Codex’s own subagent tools.
|
||||
|
||||
### Advantages
|
||||
|
||||
- Lower launch overhead.
|
||||
- Natural `/agent` inspection plus app-server-backed steering/interrupt/pause/continue/detach/stop once a native run is observed.
|
||||
- Tight integration with the root Codex session.
|
||||
- Effective for fast read-heavy parallel investigations.
|
||||
- Built-in Codex models retain their native provider behavior and authentication.
|
||||
|
||||
### Enforcement boundary
|
||||
|
||||
Codex MMO can generate a native role with a pinned model, provider, reasoning level, description, and instructions. It cannot intercept every native spawn and tool action. Consequently:
|
||||
|
||||
- Directed graph and child limits are conveyed to the model but are not mechanically intercepted for a pure native nested tree.
|
||||
- Per-job isolated worktrees, write-scope leasing, result disposition, and patch integration are unavailable.
|
||||
- Output contracts are advisory (`contract_enforcement = "warn"`) rather than supervisor-validated; the compiler rejects `strict` on a native-only role.
|
||||
- Native writers share the workspace and require disjoint assignments plus root diff review. All featured bundled native roles are therefore read-only.
|
||||
- Native subagents inherit relevant Codex session sandbox/approval behavior.
|
||||
- Native spawn events may not be completely represented in noninteractive JSON event streams; the live smoke test uses a behavioral completion marker.
|
||||
|
||||
### Appropriate work
|
||||
|
||||
Use native agents for:
|
||||
|
||||
- Small read-only investigations.
|
||||
- Independent code review with no writes.
|
||||
- Tight, low-latency work under a built-in Codex root.
|
||||
- Single-model context specialization.
|
||||
- Work where `/agent` UX and shared root-host context matter more than strict per-job isolation.
|
||||
|
||||
## Hybrid mode
|
||||
|
||||
Hybrid mode makes both paths available:
|
||||
|
||||
```toml
|
||||
[coordination]
|
||||
orchestration = "hybrid"
|
||||
native_nested_delegation = false
|
||||
```
|
||||
|
||||
The recommended topology is:
|
||||
|
||||
```text
|
||||
root
|
||||
├── native participant fast first-level branch
|
||||
│ └── MCP participant enforceable nested branch
|
||||
└── MCP participant durable/strict branch
|
||||
```
|
||||
|
||||
With `native_nested_delegation = false`, any child reachable from a native participant must support MCP. The generated native role receives an authenticated `mmo_mesh` MCP server containing only its allowed children. This gives the first-level native path low overhead while keeping nested depth, resources, scopes, and contracts enforceable.
|
||||
|
||||
Setting `native_nested_delegation = true` allows a profile to instruct native participants to spawn native children directly. Profile compilation emits a warning because those nested edges are advisory. Use this only when native latency is more important than mechanical policy enforcement.
|
||||
|
||||
## Backend selection in hybrid profiles
|
||||
|
||||
A participant can support both:
|
||||
|
||||
```toml
|
||||
[agents.expert]
|
||||
backends = ["mcp", "native"]
|
||||
```
|
||||
|
||||
The root instructions provide role descriptions and available paths. The default guidance is:
|
||||
|
||||
- Native for fast, read-heavy, tightly coupled work.
|
||||
- MCP for durable, strict, nested, write-scoped, or low-trust work.
|
||||
- After client or transport recovery, list and inspect retained work before spawning replacements;
|
||||
continue the same suspended job when its objective remains useful.
|
||||
- Treat a stock-TUI fresh context as a host-owned root generation inside the immutable MMO
|
||||
session/run, not as permission to start a replacement root or session.
|
||||
|
||||
Smoke tests must specify a backend when a role supports both:
|
||||
|
||||
```toml
|
||||
[[tasks]]
|
||||
agent = "expert"
|
||||
backend = "native"
|
||||
```
|
||||
|
||||
An unspecified hybrid smoke task defaults to MCP because MCP produces stronger observable guarantees.
|
||||
|
||||
## Root productivity
|
||||
|
||||
Every backend uses the same root policy:
|
||||
|
||||
1. Determine the immediate critical path.
|
||||
2. Identify genuinely independent side work.
|
||||
3. Spawn only work that saves time, isolates noise, adds specialization, or improves confidence.
|
||||
4. Continue useful non-overlapping root work immediately.
|
||||
5. Poll or inspect results as needed.
|
||||
6. Wait only when the next action depends on unfinished output.
|
||||
7. Reconcile results from primary evidence.
|
||||
8. Explicitly accept or reject material results, integrate accepted patches, and run canonical validation.
|
||||
|
||||
An agent is not useful merely because capacity is available. The configured maximum is a ceiling, not a target.
|
||||
|
||||
## Contradictions
|
||||
|
||||
Profile contradiction policies control generated instructions:
|
||||
|
||||
- `primary_evidence` — resolve from repository, tests, logs, and specifications.
|
||||
- `designated_judge` — one role adjudicates after examining evidence.
|
||||
- `root_adjudication` — the root makes the final determination.
|
||||
|
||||
No policy uses model majority vote as a substitute for evidence.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Profile portfolio coverage
|
||||
|
||||
This document maps the coordination and workload properties intentionally demonstrated by the implemented portfolio. A check records design intent; it does not claim that a live efficacy gate has already passed.
|
||||
|
||||
## Workload and coordination coverage
|
||||
|
||||
| Profile | General engineering | Debugging | Research | Security | Visual | Refactor/contracts | Incident analysis | Cost/access | Route resilience |
|
||||
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| `adaptive-engineering` | ✓ | | | | | | | | |
|
||||
| `codex-harness-team` | ✓ | | | | | | | | |
|
||||
| `high-confidence-debugging` | | ✓ | | | | | | | |
|
||||
| `research-backed-engineering` | | | ✓ | | | | | | |
|
||||
| `secure-change` | | | | ✓ | | | | | |
|
||||
| `visual-engineering` | | | | | ✓ | | | | |
|
||||
| `contract-first-refactoring` | | | | | | ✓ | | | |
|
||||
| `incident-hypothesis-triage` | | | | | | | ✓ | | |
|
||||
| `access-efficient-escalation-lab` | | | | | | | | ✓ | |
|
||||
| `route-resilience-lab` | | | | | | | | | ✓ |
|
||||
| `bounded-research-organization-lab` | | | ✓ | | | | | | |
|
||||
| `competing-implementations-lab` | ✓ | | | | | | | | |
|
||||
|
||||
## Agentic pattern coverage
|
||||
|
||||
| Profile | Heterogeneous makers | Homogeneous control | Dynamic delegation | Bounded nesting | Parallel work | Adversarial/critic | Independent verification | Weak-model containment | Isolated competing writes |
|
||||
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| `adaptive-engineering` | ✓ | | ✓ | | ✓ | ✓ | ✓ | | |
|
||||
| `codex-harness-team` | | ✓ | ✓ | | ✓ | ✓ | ✓ | | |
|
||||
| `high-confidence-debugging` | ✓ | | | | ✓ | ✓ | ✓ | | |
|
||||
| `research-backed-engineering` | ✓ | | ✓ | | ✓ | | ✓ | | |
|
||||
| `secure-change` | ✓ | | ✓ | | ✓ | ✓ | ✓ | | |
|
||||
| `visual-engineering` | | ✓ | ✓ | | ✓ | ✓ | ✓ | | |
|
||||
| `contract-first-refactoring` | ✓ | | | | ✓ | | ✓ | | |
|
||||
| `incident-hypothesis-triage` | ✓ | | ✓ | | ✓ | ✓ | ✓ | | |
|
||||
| `access-efficient-escalation-lab` | ✓ | | ✓ | | ✓ | | ✓ | ✓ | |
|
||||
| `route-resilience-lab` | ✓ | | | | ✓ | | ✓ | | |
|
||||
| `bounded-research-organization-lab` | ✓ | | ✓ | ✓ | ✓ | | ✓ | | |
|
||||
| `competing-implementations-lab` | ✓ | | | | ✓ | | ✓ | | ✓ |
|
||||
|
||||
`high-confidence-debugging` overlaps diagnosis with independent reproduction, but its repair and verification barriers are sequential. A check under parallel work therefore means useful independent activity exists, not that the entire graph is fan-out.
|
||||
|
||||
## Access and backend coverage
|
||||
|
||||
| Profile | Codex/ChatGPT | OpenCode Go | OpenCode Zen | Z.AI Coding Plan | OpenRouter | Local | Native agents | Agent MCP | Hybrid |
|
||||
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| `adaptive-engineering` | ✓ | ✓ | ✓ | | | | ✓ | ✓ | ✓ |
|
||||
| `codex-harness-team` | ✓ | | | | | | ✓ | ✓ | ✓ |
|
||||
| `high-confidence-debugging` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||||
| `research-backed-engineering` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||||
| `secure-change` | | ✓ | ✓ | ✓ | | | | ✓ | |
|
||||
| `visual-engineering` | ✓ | | | | | | ✓ | ✓ | ✓ |
|
||||
| `contract-first-refactoring` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||||
| `incident-hypothesis-triage` | | ✓ | | ✓ | ✓ | | | ✓ | |
|
||||
| `access-efficient-escalation-lab` | ✓ | ✓ | | ✓ | | ✓ | | ✓ | |
|
||||
| `route-resilience-lab` | ✓ | ✓ | | | ✓ | | | ✓ | |
|
||||
| `bounded-research-organization-lab` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||||
| `competing-implementations-lab` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||||
|
||||
The matrix distinguishes access coverage from maker diversity. A ChatGPT-only profile is valuable as a harness control; a route-resilience profile intentionally uses the same DeepSeek model through different access products. Neither should be mislabeled as the other kind of diversity.
|
||||
|
||||
## Mechanical guarantees
|
||||
|
||||
The runtime, not profile prose, enforces the following for Agent-MCP work:
|
||||
|
||||
- exact route/model/reasoning binding and per-route availability;
|
||||
- authenticated lineage, directed spawn edges, per-target action grants, reusable active capacity, and depth;
|
||||
- task-kind admission and structured `literal_task` input for low-trust roles;
|
||||
- sandbox ceilings, attachment containment, resource units, goal token ceilings or clock-free turns, warning-only stalls, bounded explicit finalization, and cancellation cascades;
|
||||
- persistent Unix app-server hosts and exact thread identity, complete event/terminal history, bounded partial evidence, same-thread transport/host recovery, and continuation;
|
||||
- opaque-run compare-and-swap inspect/trace/steer/interrupt/pause/continue/detach/stop/finalize/compact/respond/effort/fork controls limited by exact per-target action grants across root, native, and MCP runs;
|
||||
- isolated Git worktrees for delegated writers, binary-safe patch capture, artifact/hash correlation, and canonical integration controlled by the root;
|
||||
- monotonic `read`, `accept`, `reject`, and `integrate` result states with audit events;
|
||||
- strict JSON result shape plus command/artifact correlation where declared;
|
||||
- route-policy emission and requested/actual serving telemetry;
|
||||
- separate API, subscription, and local-resource ledgers.
|
||||
|
||||
The runtime does not mechanically prove:
|
||||
|
||||
- semantic truth of a valid JSON result;
|
||||
- that a native role obeyed every prompt-level limit;
|
||||
- that every non-wait second was productive;
|
||||
- independent infrastructure merely from different model names;
|
||||
- model-role superiority without repeated live evaluation.
|
||||
|
||||
All featured native participants are read-only. Native output contracts are advisory and native spawn telemetry remains Codex-owned. Profiles use Agent MCP whenever strict scopes, lifecycle, result contracts, external routes, or bounded nesting are essential.
|
||||
|
||||
## Result and write safety
|
||||
|
||||
Every isolated writer works outside the canonical checkout. A successful worker produces a patch/artifact manifest and a clean or tainted audit state; it does not modify the root workspace. The root must read the result, choose `accept` or `reject`, and call integration explicitly. Integration rechecks patch identity and applies it to the canonical worktree under a lease. Concurrent writers are bundled only in `competing-implementations-lab`, where the worktrees are independent and only one candidate is selected.
|
||||
|
||||
## Evaluation coverage
|
||||
|
||||
The twelve suite IDs map one-to-one to the twelve profiles. Schema 8 provides neutral outcomes, optional orchestration diagnostics, configured-root and strongest-single controls, accessible-service controls, root-plus-worker variants, complete worker ablations, three development trials, five release trials, negative controls, difficulty bands, deterministic hidden mutations, and holdout files.
|
||||
|
||||
The evaluator records success, objective validation, wall time, root wait/activity evidence, overlap, acceptance/rejection/integration, contradictions, retries, route/serving identity, tokens, and non-interchangeable cost ledgers. It publishes no aggregate quality/cost score.
|
||||
|
||||
The visual suite uses Playwright screenshots and an image-preserving transport. Security, research, debugging, and contract suites include workload-specific evidence assertions. Route and incident suites inject typed credential-loss, rate-limit, and timeout faults without silently rebinding affected roles.
|
||||
|
||||
## Tool MCP boundary
|
||||
|
||||
Tool MCP is not Agent MCP. Operators own a finite, machine-specific server registry; profiles can grant only existing server/tool IDs and can only narrow the registry’s approval ceiling. Credentials and endpoints never enter installable profile packs. A role’s read-only workspace permission does not make a third-party MCP tool read-only in its external system.
|
||||
|
||||
## Honest release boundary
|
||||
|
||||
Offline validation can establish schema, safety, transport construction, lifecycle, and fixture behavior. It cannot establish current authenticated availability or composition efficacy. A release claim remains blocked until live catalog fingerprints, provider smoke tests, five-trial featured evaluations, complete cost/route telemetry, and all ablation/promotion gates are retained as evidence.
|
||||
|
||||
The 2026-08-23 external audit repinned the execution contract to Codex 0.149.0 alone and refreshed the public inventories to 29 Go, 64 Zen, and 422 OpenRouter records. Go has one listing-only non-agent record (`hy3-preview`); the Muse contributor route now has a complete executable evidence join. Zen has 58 executable records plus six Google-native catalog-only records; two executable records are explicitly `live-undocumented` because the live listing and Models.dev agree while the provider documentation table omits them. Authenticated Z.AI Coding Plan discovery still establishes the three selected bindings among nine advertised IDs, not support for the six unselected IDs. A no-retry, eight-token probe completed through built-in Luna and through OpenRouter Chat with Mistral Nemo; every Go/Zen protocol pairing returned upstream HTTP 403 and both Z.AI Coding pairings returned upstream HTTP 429. The older pinned DeepSeek OpenRouter endpoint failure also remains unresolved: success on a different reviewed model is not a fallback or evidence for that endpoint. Current provider availability, selected-route endpoint health, updated Z.AI document review, complete route telemetry, and repeated efficacy/ablation trials therefore remain release blockers rather than inferred successes.
|
||||
@@ -0,0 +1,184 @@
|
||||
# Bundled profile portfolio
|
||||
|
||||
The bundled portfolio is a set of workload-specific compositions, not a ranking of model vendors. It deliberately separates four kinds of diversity:
|
||||
|
||||
- model maker and model family;
|
||||
- API operator and access product;
|
||||
- wire protocol and billing route;
|
||||
- actual serving provider when an aggregator reports it.
|
||||
|
||||
Those are not interchangeable. An OpenRouter route, for example, identifies an aggregator until the serving provider is recorded in telemetry. GLM-5.3 roles in the bundled profiles always use the Z.AI Coding Plan Responses route and `ZAI_CODING_API_KEY`; they never fall back to the Z.AI General API.
|
||||
|
||||
The 8.0.0 portfolio belongs to one breaking development generation. Every MMO-owned document uses schema 8 and every profile declares exact version 8.0.0. Old profile IDs, model keys, flat control lists, active-work fields, lifetime-spawn fields, rollout state, and records from any other generation fail validation. There are no aliases, migration readers, alternate hosts, automatic rewrites, compatibility profiles, or independent profile-version upgrade paths.
|
||||
|
||||
Every role receives profile-derived guidance in its generated Codex home. For spawn-capable roles, the compiled `AGENTS.md` requires a delegation decision within three substantive task calls. Every spawn- or control-capable role receives `mmo-profile-orchestration/SKILL.md` with its exact spawn/action graph, backend choices, goal/turn lifecycle, warning-only stalls, task-brief requirements, root/native/MCP controls, result lifecycle, and root-productivity workflow. A nontrivial decomposable task launches an eligible branch early; an atomic task may remain single-agent only with a concrete reason. A control-only role is told to inspect or intervene only at a real dependency/risk boundary and is not given spawn or result-disposition authority. True leaf guidance states the no-spawn/no-control boundary and disables the inherited skill. Every root and supervised MCP worker owns a canonical Unix app-server host and durable thread; native participants are durable child threads inside the root host, with advisory guarantees where Codex exposes no interception hook.
|
||||
|
||||
## Choosing a profile
|
||||
|
||||
| Profile | Choose it for | Main trade-off |
|
||||
|---|---|---|
|
||||
| `adaptive-engineering` | General, decomposable engineering | Selective external calls and integration overhead |
|
||||
| `codex-harness-team` | ChatGPT-only access or a homogeneous control | Less provider diversity and less mechanical visibility for native roles |
|
||||
| `high-confidence-debugging` | Ambiguous bugs where reproduction and falsification matter | Sequential evidence gates increase latency |
|
||||
| `research-backed-engineering` | Changes whose correctness depends on current external sources | Source retrieval and audit cost |
|
||||
| `secure-change` | Security-sensitive changes and vulnerability remediation | Expensive, deliberately skeptical review path |
|
||||
| `visual-engineering` | UI work driven by images and browser output | Requires an intact multimodal transport and browser fixture |
|
||||
| `contract-first-refactoring` | Compatibility-sensitive refactors | Contract discovery happens before implementation |
|
||||
| `incident-hypothesis-triage` | Read-only incident diagnosis | Produces hypotheses and evidence, never remediation writes |
|
||||
|
||||
Use a lab only to run its stated experiment. Do not select a lab merely because its topology is novel.
|
||||
|
||||
## Featured profiles
|
||||
|
||||
### `adaptive-engineering`
|
||||
|
||||
The Sol root owns diagnosis, the canonical workspace, integration, and final validation. It can launch independent work early while continuing the critical path:
|
||||
|
||||
```text
|
||||
Sol root (Codex/ChatGPT, writer)
|
||||
├── Luna repository scout (native, read-only)
|
||||
├── DeepSeek V4 Pro implementation specialist (Go, isolated writer)
|
||||
└── Opus 5 adversarial reviewer (Zen, read-only)
|
||||
```
|
||||
|
||||
The specialist receives one disjoint, objectively testable branch. Its Git patch is not canonical until the root reads and accepts the result and explicitly integrates the patch. The reviewer is for material architecture or post-change risk, not a mandatory council call. Maximum depth is one and each worker has one spawn.
|
||||
|
||||
Why it exists: it is the broad default and demonstrates selective hybrid delegation, economical implementation, cross-maker review, and an active flagship root without forcing every task through a council.
|
||||
|
||||
### `codex-harness-team`
|
||||
|
||||
This is the portfolio’s homogeneous-maker control:
|
||||
|
||||
```text
|
||||
Sol root (writer)
|
||||
├── Luna repository scout (native, read-only)
|
||||
├── Terra invariant designer (native, read-only)
|
||||
└── fresh Sol critic (Agent MCP, read-only)
|
||||
```
|
||||
|
||||
All roles use ChatGPT-backed Codex access. The value comes from context isolation, parallel reconnaissance, role specialization, and a fresh critic—not maker diversity. Native roles are advisory and read-only; the Agent-MCP critic has strict lineage and a result contract. The root is the sole writer.
|
||||
|
||||
Why it exists: users with only Codex access still get a meaningful team, and evaluations gain a control against the claim that heterogeneous models are always necessary.
|
||||
|
||||
### `high-confidence-debugging`
|
||||
|
||||
```text
|
||||
DeepSeek V4 Pro debug lead (Go, writer)
|
||||
├── Sonnet 5 independent reproducer (Zen, isolated writer)
|
||||
└── Sol fix verifier (Codex Agent MCP, read-only; one follow-up allowed)
|
||||
```
|
||||
|
||||
The reproducer creates a minimal failing test or empirical reproduction independently of the lead’s suspected cause. Repair cannot be accepted merely from a plausible explanation. The verifier starts from a fresh context, attacks the proposed fix, and may be relaunched once after a correction. The root remains productive by investigating likely causes while reproduction runs, but the repair barrier is real.
|
||||
|
||||
Why it exists: independent reproduction and post-fix falsification address correlated debugging errors that a fast single pass often misses.
|
||||
|
||||
### `research-backed-engineering`
|
||||
|
||||
```text
|
||||
MiniMax M3 research/engineering lead (Go, writer)
|
||||
├── Luna primary-source researchers (Codex Agent MCP, read-only; up to two)
|
||||
└── Sonnet 5 source-to-code auditor (Zen, read-only)
|
||||
```
|
||||
|
||||
Researchers return claim-level URLs, publication/access times, and bounded excerpts. The lead re-fetches pivotal claims rather than treating summaries as truth, implements in the repository, and asks the auditor to compare source requirements with the actual change. Unavailable Tool MCP research servers degrade the research branch; they are not silently replaced by a different model or route.
|
||||
|
||||
Why it exists: current-source verification and repository work demand different context management and benefit from independent source-to-code checking.
|
||||
|
||||
### `secure-change`
|
||||
|
||||
```text
|
||||
Opus 5 security lead (Zen, writer)
|
||||
├── GLM-5.3 vulnerability hunter (Z.AI Coding Plan, read-only; up to two passes)
|
||||
└── DeepSeek V4 Pro finding verifier (Go, read-only)
|
||||
```
|
||||
|
||||
The lead frames assets, trust boundaries, and remediation scope. GLM supplies independent candidate findings; DeepSeek must reproduce consequential claims with commands or repository evidence. The root remediates only confirmed findings and owns regression/security validation. A structured contract constrains shape and requires evidence correlation, but no contract makes a model’s security conclusion true.
|
||||
|
||||
Why it exists: it combines deliberately different security-analysis and empirical-verification roles without granting either worker write authority.
|
||||
|
||||
### `visual-engineering`
|
||||
|
||||
```text
|
||||
multimodal Sol visual lead (Codex, writer)
|
||||
├── Terra reference analyst (native, read-only)
|
||||
└── fresh multimodal Sol verifier (Agent MCP, read-only; one follow-up allowed)
|
||||
```
|
||||
|
||||
The root implements directly from supplied references. The verifier receives the original reference and an actual Playwright browser render through an image-preserving route, checks layout and accessibility evidence, and emits a strict visual result. Compilation rejects a model/transport path that loses image modality. Native analysis remains optional and advisory.
|
||||
|
||||
Why it exists: visual correctness is not adequately tested by DOM text or synthetic prose descriptions alone.
|
||||
|
||||
### `contract-first-refactoring`
|
||||
|
||||
```text
|
||||
Sonnet 5 refactor lead (Zen, writer)
|
||||
├── MiniMax M3 invariant miner (Go, read-only)
|
||||
├── Luna contract-test author (Codex Agent MCP, isolated writer)
|
||||
└── DeepSeek V4 Pro compatibility verifier (Go, read-only)
|
||||
```
|
||||
|
||||
The lead freezes public behavior before changing implementation. The invariant miner inspects callers and edge cases; the test author produces a disjoint patch against that contract; the lead explicitly reviews, accepts, and integrates it. Verification then attacks compatibility with economical independent execution.
|
||||
|
||||
Why it exists: separating specification, tests, implementation, and verification materially reduces self-consistent but wrong refactors.
|
||||
|
||||
### `incident-hypothesis-triage`
|
||||
|
||||
```text
|
||||
GLM-5.3 incident lead (Z.AI Coding Plan, read-only)
|
||||
├── DeepSeek V4 Flash evidence runner (Go, read-only; up to two probes)
|
||||
└── Nemotron causal challenger (OpenRouter, read-only)
|
||||
```
|
||||
|
||||
The entire profile is read-only. The lead constructs and ranks falsifiable hypotheses. The runner collects bounded logs and command evidence; the challenger uses a pinned OpenRouter serving policy to attack causal assumptions. Provider routing telemetry records the requested route and any reported serving provider. Evidence, not votes, determines the final hypothesis order.
|
||||
|
||||
Why it exists: fast operational evidence and independent causal challenge improve triage while mechanically preventing an analysis profile from modifying production code.
|
||||
|
||||
## Experimental labs
|
||||
|
||||
### `access-efficient-escalation-lab`
|
||||
|
||||
A Luna writer routes only literal, bounded reconnaissance to local Qwen3.5-9B; routine isolated patches go to DeepSeek V4 Flash; difficult read-only judgment may escalate to GLM-5.3 Coding Plan. The weak local role is MCP-only, read-only, persistent-thread hosted, literal-task constrained, strict-contract bound, and always verified. It cannot architecture, debug ambiguously, write, delegate, control another worker, or adjudicate. Persistence prevents evidence loss; it does not expand the weak role's authority.
|
||||
|
||||
This cannot be represented by `adaptive-engineering`: its object of study is access cost and escalation thresholds, including a mechanically contained weak model.
|
||||
|
||||
### `route-resilience-lab`
|
||||
|
||||
A Terra observer sends matched read-only tasks to DeepSeek V4 Pro through OpenCode Go and through OpenRouter. The OpenRouter route has a typed provider policy with fallback disabled; requested route, actual serving provider, policy, latency, usage, and errors are retained separately.
|
||||
|
||||
This cannot be folded into incident triage: its workload is deliberately controlled so that route behavior—not agent specialization—is the independent variable.
|
||||
|
||||
### `bounded-research-organization-lab`
|
||||
|
||||
A Sol engineering root compares flat source scouts with a Kimi K3 research lead that may run at most two instances of the same MiniMax M3 scout role concurrently, under delegation depth two. All delegated work is read-only. Completed scouts release their slots, allowing later evidence-driven follow-ups without an arbitrary session-lifetime ceiling.
|
||||
|
||||
This cannot be represented by the flat research profile because bounded nested organization is the feature under evaluation.
|
||||
|
||||
### `competing-implementations-lab`
|
||||
|
||||
A Terra judge defines one objective contract, then DeepSeek V4 Pro and Sonnet 5 build independent patches in isolated Git worktrees. Both writers may run concurrently because their branches are isolated. The judge tests both patches, explicitly accepts one result, integrates only that patch, and rejects the other.
|
||||
|
||||
This cannot be represented by a normal implementation profile: matched competition and empirical selection are the experiment, not merely a second reviewer.
|
||||
|
||||
## Availability and failure behavior
|
||||
|
||||
The root route is startup-critical. A missing root credential or unavailable root model prevents session creation. Missing optional worker credentials do not invalidate the whole snapshot: the role is marked unavailable, attempts to spawn it return a typed `route_unavailable` admission error, and no job is created. Profiles must explain how the productive root proceeds without that branch.
|
||||
|
||||
There is no silent model, maker, provider, access-product, or reasoning fallback. OpenRouter fallback is allowed only when a route’s typed policy explicitly permits it; bundled OpenRouter examples pin allowed serving providers and disable fallback.
|
||||
|
||||
MCP worker results follow an explicit lifecycle: `unread` → `read` → `accepted` or `rejected`; accepted isolated-writer results may then become `integrated`. `agents_wait` returns compact job state and per-job `progress_revision` values, returning at the first durable change when those revisions are supplied as `after_revision`; bounded result previews are opt-in. `agent_result` provides lossless cursor pages until `next_cursor` is null, and `agent_trace_record` does the same for a checksummed oversized filtered trace event. Generated guidance requires result/inspect/trace APIs and prohibits direct reads of MMO supervisor files. Transitions are monotonic and audited. Reading is not acceptance, and acceptance is not patch application.
|
||||
|
||||
## Rebinding
|
||||
|
||||
Bindings use exact route-qualified keys:
|
||||
|
||||
```bash
|
||||
codex-mmo run \
|
||||
--profile adaptive-engineering \
|
||||
--bind implementation_specialist=opencode_go_openai_chat__deepseek_v4_pro
|
||||
```
|
||||
|
||||
The compiler revalidates route availability, protocol, modalities, reasoning levels, tool behavior, trust policy, and role constraints. A syntactically valid replacement is not accepted if the complete route cannot perform the role.
|
||||
|
||||
## Evaluation status
|
||||
|
||||
Every profile has a generation-8 smoke test and matched generation-8 evaluation suite. Smoke wall limits belong to the external harness and do not replace profile goal-token limits or warning-only stall policy. Bundling means the composition is implemented and testable; it does not claim universal superiority. Featured status still requires authenticated live inventory evidence and the suite’s release-trial promotion gates. Labs remain labs unless their matched controls and ablations establish the stated benefit.
|
||||
@@ -0,0 +1,320 @@
|
||||
# Profile format: MMO generation 8
|
||||
|
||||
A profile is a static, non-executable composition pack. It declares roles, exact route-qualified model bindings, delegation/action graphs, goal/turn lifecycle, permissions, trust, resources, contracts, Tool MCP grants, and smoke tasks. The runtime supplies Unix app-server hosting, isolated writer worktrees, lineage, admission, root/native/MCP control, result disposition, integration, pause/continue/detach/stop/cancel lifecycle, gateways, and durable history/telemetry.
|
||||
|
||||
Only MMO schema `8` is valid. Profiles, smoke files, catalog overlays, and evaluation definitions use that same active generation. Earlier packs are not migrated, rewritten, normalized, or read through a compatibility layer. A profile's `version` must exactly equal the package version; it does not evolve independently during active development.
|
||||
|
||||
## Pack layout
|
||||
|
||||
```text
|
||||
profile-id/
|
||||
├── profile.toml
|
||||
├── README.md optional
|
||||
├── LICENSE optional
|
||||
├── catalog.toml optional generation-8 route/model fragment
|
||||
├── smoke.toml optional generation-8 live tasks
|
||||
├── agents/*.md optional role instructions
|
||||
└── contracts/*.json optional result contracts
|
||||
```
|
||||
|
||||
Only those top-level members are allowed. Packs reject executables, scripts, symlinks, special files, escaping paths, unsupported suffixes, and missing explicitly named files. Unknown fields and wrong TOML scalar/container types fail; booleans are not accepted as integer schema versions.
|
||||
|
||||
## Minimal profile
|
||||
|
||||
```toml
|
||||
schema_version = 8
|
||||
id = "review-pair"
|
||||
version = "8.0.0"
|
||||
display_name = "Review Pair"
|
||||
description = "One active engineer and one fresh reviewer."
|
||||
tags = ["review"]
|
||||
maturity = "lab"
|
||||
root = "lead"
|
||||
smoke = "smoke.toml"
|
||||
|
||||
[agents.lead]
|
||||
kind = "root"
|
||||
description = "Critical-path engineer and sole canonical writer"
|
||||
model = "codex_chatgpt_builtin__gpt_5_6_terra"
|
||||
reasoning = "high"
|
||||
plan_reasoning = "high"
|
||||
permissions = "workspace-write"
|
||||
can_spawn = ["reviewer"]
|
||||
execution_mode = "goal"
|
||||
goal_token_budget = 600000
|
||||
max_goal_token_budget = 1000000
|
||||
stall_warning_seconds = 3600
|
||||
finalization_grace_seconds = 1800
|
||||
max_active = 1
|
||||
trust = "high"
|
||||
verification = "risk_based"
|
||||
requires_modalities = ["text"]
|
||||
instructions = "agents/lead.md"
|
||||
|
||||
[agents.reviewer]
|
||||
description = "Fresh read-only correctness reviewer"
|
||||
model = "opencode_zen_anthropic_messages__claude_sonnet_5"
|
||||
reasoning = "high"
|
||||
permissions = "read-only"
|
||||
backends = ["mcp"]
|
||||
can_spawn = []
|
||||
execution_mode = "turn"
|
||||
stall_warning_seconds = 1800
|
||||
finalization_grace_seconds = 900
|
||||
max_active = 1
|
||||
allowed_task_kinds = ["review", "verification"]
|
||||
trust = "adversarial_reviewer"
|
||||
verification = "root_adjudication"
|
||||
contract_enforcement = "strict"
|
||||
output_contract = "contracts/review.json"
|
||||
instructions = "agents/reviewer.md"
|
||||
|
||||
[agents.lead.controls.reviewer]
|
||||
actions = ["inspect", "trace", "steer", "interrupt", "pause", "continue", "detach", "stop", "finalize", "compact", "respond", "set_effort", "fork"]
|
||||
|
||||
[coordination]
|
||||
mode = "actor_critic"
|
||||
orchestration = "mcp"
|
||||
max_active_agents = 2
|
||||
max_depth = 1
|
||||
max_children_per_agent = 1
|
||||
max_active_writers = 0
|
||||
reject_ancestor_role = true
|
||||
wait_policy = "dependency_only"
|
||||
write_conflict_policy = "reject"
|
||||
contradiction_policy = "primary_evidence"
|
||||
result_visibility = "ancestors"
|
||||
```
|
||||
|
||||
## Profile fields
|
||||
|
||||
| Field | Required | Meaning |
|
||||
|---|:---:|---|
|
||||
| `schema_version` | yes | Exactly `8` |
|
||||
| `id` | yes | Stable lowercase ID |
|
||||
| `version` | yes | Exactly the active package version; included in snapshot identity |
|
||||
| `display_name` | yes | Product-facing name |
|
||||
| `description` | yes | Practical purpose and composition |
|
||||
| `tags` | no | Search/classification labels |
|
||||
| `maturity` | yes | `featured` or `lab` |
|
||||
| `root` | yes | The one agent whose `kind = "root"` |
|
||||
| `catalog` | no | Relative generation-8 fragment; implicit `catalog.toml` when present |
|
||||
| `smoke` | no | Relative generation-8 smoke file; implicit `smoke.toml` when present |
|
||||
|
||||
## Coordination
|
||||
|
||||
| Field | Default | Contract |
|
||||
|---|---:|---|
|
||||
| `mode` | `rooted_team` | Descriptive policy label, not a scheduler plugin |
|
||||
| `orchestration` | `mcp` | `mcp`, `native`, or `hybrid` |
|
||||
| `max_active_agents` | `5` | Agent-MCP admission ceiling including the root; also bounds compiler feasibility and defaults the separate native thread cap |
|
||||
| `max_depth` | `1` | Root is depth zero |
|
||||
| `max_children_per_agent` | `4` | Simultaneous children per caller unless role-overridden |
|
||||
| `max_active_writers` | `2` | Concurrent MCP writer jobs; each uses an isolated worktree |
|
||||
| `reject_ancestor_role` | `true` | Prevent a role already in the lineage from recurring |
|
||||
| `wait_policy` | `dependency_only` | The only supported policy; historical modes are rejected |
|
||||
| `write_conflict_policy` | `reject` | Overlapping canonical scopes are rejected |
|
||||
| `contradiction_policy` | `primary_evidence` | `primary_evidence`, `designated_judge`, or `root_adjudication` |
|
||||
| `result_visibility` | `ancestors` | `ancestors` or `session` |
|
||||
| `default_result_chars` | `12000` | Default result retrieval ceiling |
|
||||
| `max_result_chars` | `30000` | Hard result retrieval ceiling; full artifacts remain on disk |
|
||||
| `native_max_concurrent_threads` | derived | Codex native ceiling, excluding the root slot |
|
||||
| `native_interrupt_message` | `true` | Codex native completion interruption behavior |
|
||||
| `native_nested_delegation` | `false` | Explicitly opt into advisory native-to-native nesting |
|
||||
|
||||
The compiler rejects declarations that exceed its derived concurrency, writer, resource, and depth ceilings. For every shared resource lock it reserves any root units, expands each reachable role's `max_active` slots at that role's `resource_units` weight, and counts the cheapest combination that can fit. No role, including the root, may request more units than its resource capacity. Evaluation ablations reuse this same derivation after pruning their graph. This is an exact active-resource ceiling, not a promise that every task can realize it: delegation edges, per-caller child limits, provider behavior, and task dependencies can lower observed concurrency. Terminal workers release admission capacity, so sequential delegation does not consume a lifetime quota. `dependency_only` compiles active-root guidance and the evaluator observes waits/activity events, but MMO cannot prove cognitive productivity between events.
|
||||
|
||||
## Agent fields
|
||||
|
||||
| Field | Default | Meaning |
|
||||
|---|---:|---|
|
||||
| `kind` | `participant` | `root` or `participant` |
|
||||
| `description` | empty | Capability exposed to callers |
|
||||
| `model` | required | Exact route-qualified catalog key |
|
||||
| `reasoning` | `high` | Must be supported by the model/route; `none` omits the selector |
|
||||
| `plan_reasoning` | unset | Root planning effort, when applicable |
|
||||
| `permissions` | `read-only` | `read-only` or `workspace-write` |
|
||||
| `can_spawn` | `[]` | Directed child-role allowlist |
|
||||
| `controls` | `{}` | Per-target tables with an exact `actions` allowlist for root, native, or MCP runs |
|
||||
| `max_active` | `1` | Simultaneous instances of the role across the stable profile ID |
|
||||
| `max_children` | coordination default | Simultaneous children for one caller instance |
|
||||
| `write_scope_required` | `true` | Require canonical scope declarations for MCP writers |
|
||||
| `trust` | `normal` | `low`, `normal`, `high`, or `adversarial_reviewer` |
|
||||
| `verification` | `material_changes` | `always`, `material_changes`, `risk_based`, or `root_adjudication` |
|
||||
| `allowed_task_kinds` | `analysis`, `other` | Exact Agent-MCP admission categories |
|
||||
| `min_task_chars` / `max_task_chars` | `12` / `12000` | Free-form task bounds for normal roles |
|
||||
| `execution_mode` | `turn` | `turn`, or durable Codex `goal` for roots and MCP-only participants; native-capable participants must use `turn` because native delegation exposes no mechanically owned goal lifecycle |
|
||||
| `goal_token_budget` | unset | Required for `goal`; 10,000–100,000,000 host-observed total tokens |
|
||||
| `max_goal_token_budget` | unset | Required for `goal`; controller extension ceiling, at least the initial budget |
|
||||
| `stall_warning_seconds` | `1800` | Silence interval before an operator warning; 60–86,400 seconds and never an interrupt |
|
||||
| `finalization_grace_seconds` | `900` | Host boundary for an explicitly requested terminal serialization/repair; 30–3,600 seconds |
|
||||
| `allowed_reasoning_efforts` | initial `reasoning` only | Exact role-specific allowlist for controller changes; every value must be supported by the bound model |
|
||||
| `requires_modalities` | `text` | Required model and route input modalities |
|
||||
| `requires_output_modalities` | `text` | Required model and route output modalities |
|
||||
| `requires_tool_images` | `false` | Require preservation of image-bearing tool results |
|
||||
| `requires_documents` | `false` | Require file/document transport |
|
||||
| `attachments_allowed` | `false` | Permit validated attachments |
|
||||
| `network_access` | `false` | Permit sandboxed shell networking |
|
||||
| `web_search` | `disabled` | `disabled`, `cached`, `indexed`, or `live` |
|
||||
| `output_contract` | unset | Relative `contracts/*.json` member |
|
||||
| `contract_enforcement` | `warn` | `warn` or `strict`; native-only roles cannot claim strict enforcement |
|
||||
| `resource_group` | route/model default | Weighted capacity pool |
|
||||
| `resource_units` | `1` | Units held while active |
|
||||
| `instructions` | unset | Relative `agents/*.md` member |
|
||||
| `approval_policy` | `never` | Codex app-server `untrusted`, `on-request`, or `never`; `never` is autonomous inside the compiled sandbox and denies escalation, while other policies expose approval as controller-answerable pending input |
|
||||
| `backends` | profile-derived | `mcp`, `native`, or both when compatible |
|
||||
| `native_name` | generated | Optional Codex custom-agent name |
|
||||
| `tool_mcp_servers` | `{}` | Grants into the operator-owned Tool MCP registry |
|
||||
|
||||
`max_active` limits simultaneous executing instances of a role, not how many instances may ever be created during a session. A terminal, cold-paused, or suspended worker releases its role, route, resource, child, and writer admission slots, so an authorized caller may launch later follow-up work. Continuation performs current admission again before restoring the same durable thread. Reattaching a detached session preserves its live leases; continuing a paused or suspended session preserves identity and evidence but reacquires leases. A terminal root session cannot be resumed.
|
||||
|
||||
Every session and job records the creating MMO package version, and live runners additionally record
|
||||
their package version plus runtime-file digest. Inspection exposes whether that runner matches the
|
||||
installed artifact. A record from any other schema or package generation is rejected at load time;
|
||||
historical evidence must be kept outside active MMO state rather than adopted or rewritten.
|
||||
|
||||
Every root and supervised MCP worker has a persistent Unix app-server host. `turn` runs have no profile task deadline. Root and MCP-only `goal` runs use Codex's cumulative token accounting and may be continued with a larger total budget only up to `max_goal_token_budget`; compilation rejects goal mode on a native-capable participant because Codex native delegation does not expose an independently owned goal lifecycle. `stall_warning_seconds` emits an operator-visible warning without interruption; silence and provider/model slowness never cause evidence deletion. Models are never instructed to track time or emit clock-based checkpoints. External smoke/evaluation wall limits detach or stop through the normal lifecycle and are not profile execution policy. Native subagents share the root host's app-server history; their strict scope/contract guarantees remain advisory where Codex does not expose an interception hook.
|
||||
|
||||
When a strict Agent-MCP role has an output contract and its resolved model declares
|
||||
structured-output support, MMO projects the shape-compatible portion of that contract into
|
||||
app-server `turn/start.outputSchema`. Closed objects and required properties are preserved.
|
||||
Optional properties are omitted from the transport projection because Codex strict schemas require
|
||||
every advertised property; this avoids forcing a value where the original contract permits
|
||||
omission. Validation-only keywords unsupported by Codex are removed only from the transport
|
||||
projection. The complete original contract remains in the task and is always parsed, validated,
|
||||
and mechanically correlated after generation. If the first result is malformed, the same
|
||||
persisted agent thread receives one tools-discouraged shape-only repair turn. There is no separate
|
||||
JSON-fixer role, and a repair may not manufacture missing evidence.
|
||||
|
||||
## Backend boundary
|
||||
|
||||
- `mcp` roles execute in isolated persistent app-server workers with mechanical admission, live control, transport recovery, scope, contract, goal/turn lifecycle, partial-evidence, and telemetry controls.
|
||||
- `native` roles use Codex custom-agent machinery. Their model/instructions can be generated, but scope leasing, strict contract validation, and complete spawn telemetry are not externally intercepted.
|
||||
- `hybrid` profiles use both. With `native_nested_delegation = false`, any child reachable from a native participant must have an MCP path.
|
||||
|
||||
All featured bundled native roles are read-only. A native-only role with `contract_enforcement = "strict"` is rejected.
|
||||
|
||||
## Control graph
|
||||
|
||||
`controls` is separate from ancestry and `can_spawn`. Each `[agents.<caller>.controls.<target>]` table contains only `actions = [...]`. The target may be the root, a native run, or an MCP run; the runtime resolves an opaque `agent_run_ref` inside the caller's one active session/run and checks the target role plus the requested action. Low-trust roles cannot control anything. Each mutating command carries the current `control_revision` returned by inspection; a stale revision fails before delivery.
|
||||
|
||||
The available actions are inspect, filtered paginated trace, active-turn steer, interrupt, cold goal pause, same-thread continue (optionally extending a goal within its token ceiling), client detach, full stop, evidence-only finalize, durable-thread compaction, response to pending user input/MCP elicitation/approval, reasoning-effort change within `allowed_reasoning_efforts`, and persisted-thread fork. A supervised MCP pause persists partial evidence and retires its host; native pause is logical because native threads share a root host. Mutating controls are serialized per target across compare-and-swap revision assignment and delivery. If delivery began but the reply was lost, durable status is `delivery_unknown`; callers inspect before retrying rather than treating the request as failed. Control authority permits reading and stopping the target, but it does not itself grant successful-result accept/reject or patch integration; those remain lineage/result-lifecycle authorities.
|
||||
|
||||
## Low-trust roles and `literal_task`
|
||||
|
||||
`trust = "low"` requires all of the following mechanically:
|
||||
|
||||
```toml
|
||||
permissions = "read-only"
|
||||
backends = ["mcp"]
|
||||
can_spawn = []
|
||||
controls = {}
|
||||
max_active = 1
|
||||
verification = "always"
|
||||
contract_enforcement = "strict"
|
||||
network_access = false
|
||||
web_search = "disabled"
|
||||
attachments_allowed = false
|
||||
resource_units = 1
|
||||
```
|
||||
|
||||
It also requires a strict output contract, a single-slot resource group, a brief ceiling of 2,500 characters, no control authority, and task kinds drawn only from `locate`, `references`, `extract`, and `summarize_supplied`.
|
||||
|
||||
A caller cannot send low-trust free-form prose. It must submit the typed `literal_task` object advertised by Agent MCP. The runtime validates paths/ranges, reads or hashes the requested input, and synthesizes a non-interpretive prompt. The weak role may return literal evidence; it may not architecture, debug ambiguously, write, delegate, or adjudicate.
|
||||
|
||||
## Tool MCP grants
|
||||
|
||||
Tool MCP is separate from the Agent-MCP backend:
|
||||
|
||||
```toml
|
||||
[agents.researcher.tool_mcp_servers.firecrawl]
|
||||
required = true
|
||||
enabled_tools = ["search", "fetch"]
|
||||
```
|
||||
|
||||
The server ID must already exist under `~/.config/codex-mmo/tool-mcp.d/*.toml`. A grant can only narrow the operator’s finite allowlist. It cannot define or change a command, URL, credential mapping, timeout, transport, or approval policy. `mmo_mesh` is reserved.
|
||||
|
||||
## Model and route validation
|
||||
|
||||
For every role the compiler checks:
|
||||
|
||||
- exact model key and route namespace;
|
||||
- `agent_compatible = true` and tool calling;
|
||||
- reasoning selector support;
|
||||
- model input/output modality plus complete route transport;
|
||||
- tool-image/document preservation when required;
|
||||
- executable driver rather than catalog-only service;
|
||||
- resource group and requested backend;
|
||||
- trust/backend/contract consistency.
|
||||
|
||||
Rebinding reruns the same checks and changes snapshot identity:
|
||||
|
||||
```bash
|
||||
codex-mmo run \
|
||||
--profile adaptive-engineering \
|
||||
--bind implementation_specialist=opencode_go_openai_chat__deepseek_v4_pro
|
||||
```
|
||||
|
||||
No unqualified model ID, old provider key, or alias is accepted.
|
||||
|
||||
## Output contracts
|
||||
|
||||
Contracts use a deliberately small, offline JSON Schema subset:
|
||||
|
||||
```text
|
||||
$schema, $id, title, description
|
||||
type, properties, required, additionalProperties, items
|
||||
enum, const
|
||||
minLength, maxLength, pattern, format
|
||||
minimum, maximum
|
||||
minItems, maxItems, uniqueItems
|
||||
oneOf, anyOf, allOf, not, if, then, else
|
||||
```
|
||||
|
||||
Supported string formats are absolute RFC 3986 `uri`, RFC 3339 `date`, and timezone-bearing RFC 3339 `date-time`. Leap seconds and lowercase `t`/`z` are accepted where RFC 3339 permits them; malformed percent encodings and URI references are rejected. Unsupported keywords fail pack validation. No remote references or vocabularies are loaded.
|
||||
|
||||
Strict enforcement fails the job when JSON extraction, schema validation, or mechanical correlation fails. Objects containing `command` plus `exit_code` must correlate to a parsed captured command event with the same exit code. Objects containing `relative_path` plus `sha256` must name a regular non-symlink file inside the delegated tree whose bytes match the hash. Literal evidence is also correlated to the runtime-generated operation, path/range, and input hash.
|
||||
|
||||
These checks establish shape and evidence identity, not semantic correctness. A valid security verdict or architectural claim still requires independent evidence and root adjudication.
|
||||
|
||||
## Result and patch lifecycle
|
||||
|
||||
An Agent-MCP result begins `unread`. `agent_result` makes it `read`. It returns a complete strict JSON value when that value fits the requested page, otherwise contiguous text identified by `cursor`, `total_chars`, and `next_cursor`; callers must follow `next_cursor` until null. `agents_wait` returns compact lifecycle state, not a result transport. Callers pass the exact observed `progress_revision` map for every requested job as `after_revision` to wake on the first durable change and opt into bounded terminal previews only when useful. A truncated, valid `agent_trace` record carries its exact raw-line `record_cursor`; `agent_trace_record` returns the complete private-reasoning-filtered JSON text in contiguous checksummed pages. Malformed records remain bounded because their raw content cannot be structurally filtered. The authorized caller must then use `agent_result_accept` or `agent_result_reject` with a reason. Accepted isolated-writer output may be passed to `agent_patch_integrate`; integration verifies patch identity, canonical scope, and lease before applying it. States are monotonic and audited; rejection cannot be reversed and reading never applies a patch.
|
||||
|
||||
A controller/transport failure yields `suspended` plus a bounded partial result, authoritative trace/history, and patch; an authorized controller may continue the exact thread. `stopped`, `failed`, and `cancelled` are terminal evidence states and cannot be accepted as successful results. Partial evidence is intentionally labeled non-final and never satisfies a strict output contract.
|
||||
|
||||
Writable workers execute in isolated Git worktrees. Out-of-scope changes, unsafe artifacts, or patch-boundary failures taint the session and fail the job rather than mutating the canonical checkout.
|
||||
|
||||
## Smoke files
|
||||
|
||||
```toml
|
||||
schema_version = 8
|
||||
|
||||
[[tasks]]
|
||||
agent = "reviewer"
|
||||
backend = "mcp"
|
||||
task_kind = "review"
|
||||
mode = "read-only"
|
||||
wall_timeout_seconds = 300
|
||||
required_mcp_tools = ["repository.search"]
|
||||
task = "Inspect the repository and return directly observed evidence."
|
||||
```
|
||||
|
||||
Optional `write_scope`, `attachments`, and `wait_seconds` fields use runtime boundaries. `wall_timeout_seconds` belongs only to the smoke harness: a root first detaches with partial evidence and the smoke runner then fully stops that test session; an unfinished worker is cancelled at the harness boundary, retaining its partial evidence. Neither boundary changes the role's goal token budget or warning-only stall policy. For a goal-mode root, the harness appends an explicit `update_goal(status="complete")` lifecycle requirement after all task evidence is ready; merely emitting a final message would leave Codex's goal continuation active. A low-trust smoke task uses `literal_task = { ... }` instead of `task`/`task_kind`. Root success requires a successful root turn; MCP success requires a completed job; native success requires a native-role behavioral marker. Smoke tests consume live provider access and are distinct from offline tests.
|
||||
|
||||
`required_mcp_tools` is an optional list of `server.tool` names. A name may identify a Tool MCP grant or a runtime-owned `mmo_mesh` tool mechanically available to that task's role. A task passes only when every named tool has a successful direct MCP call in that task's Codex event stream. Contract-valid output, prose claiming that a tool ran, resource discovery, and failed tool calls do not satisfy this assertion.
|
||||
|
||||
## Installation identity
|
||||
|
||||
Installation accepts only the active package/profile version. Reinstalling byte-identical content is an idempotent no-op. Different current-generation content requires `--replace`; it is staged, validated, and atomically swapped. There is no upgrade, downgrade, or migration comparison between profile versions.
|
||||
|
||||
## Compilation
|
||||
|
||||
```bash
|
||||
codex-mmo profile install ./review-pair
|
||||
codex-mmo profile validate review-pair
|
||||
codex-mmo profile compile review-pair
|
||||
codex-mmo profile use review-pair
|
||||
```
|
||||
|
||||
Compilation resolves the generation-8 catalog, route availability contract, Tool MCP grants, model bindings, spawn/action-specific control graph, goal/turn lifecycle, resources, instructions, contracts, native files, generated Codex homes, and Switchyard routes into a content-addressed immutable snapshot. It also derives one exact `AGENTS.md` per role and, whenever any role can spawn or control, a profile-wide `mmo-profile-orchestration/SKILL.md` that documents the complete admitted lifecycle. Profile packs do not carry handwritten copies of these generated files. Their exact bytes participate in snapshot identity. A spawn- or control-capable role's generated home materializes the skill at Codex's standard `skills/mmo-profile-orchestration/SKILL.md` path and enables that exact file through `skills.config`; true leaf layers explicitly disable the inherited path. Credential values never become snapshot content. Current-generation sessions continue using their exact snapshot when profile files are replaced.
|
||||
@@ -0,0 +1,245 @@
|
||||
# Route drivers
|
||||
|
||||
Codex MMO is model-maker neutral, but protocols and access products are not interchangeable. In MMO generation 8, an executable `route` selects one driver and records its API operator, access product, wire protocol, billing mode, transport capabilities, credential names, and resource group.
|
||||
|
||||
The filename remains `PROVIDER_DRIVERS.md` for documentation organization only; the public catalog object is `routes`, not the old provider-shaped schema.
|
||||
|
||||
## Driver matrix
|
||||
|
||||
| Driver | Execution path | Typical use | Managed gateway |
|
||||
|---|---|---|:---:|
|
||||
| `codex_builtin` | Codex reserved provider | ChatGPT-backed Codex models | no |
|
||||
| `codex_custom` | Direct Codex Responses endpoint | Conforming private/third-party Responses API | no |
|
||||
| `codex_oss` | Codex native local integration | Ollama or LM Studio | no |
|
||||
| `switchyard` | Immutable snapshot-specific route | Chat, Responses, Anthropic Messages, compatible local APIs | yes |
|
||||
| `catalog_only` | Metadata only | Media/service endpoint or unsupported transport | no execution |
|
||||
|
||||
## Common route identity
|
||||
|
||||
```toml
|
||||
[routes.example_openai_chat]
|
||||
name = "Example API"
|
||||
api_operator = "example"
|
||||
access_product = "example_api"
|
||||
wire_protocol = "openai_chat"
|
||||
billing_mode = "api"
|
||||
driver = "switchyard"
|
||||
base_url = "https://models.example.test/v1"
|
||||
credential_envs = ["EXAMPLE_API_KEY"]
|
||||
resource_group = "example_pool"
|
||||
transport_modalities = ["text"]
|
||||
transport_output_modalities = ["text"]
|
||||
tool_calling = true
|
||||
parallel_tool_calls = false
|
||||
max_retries = 1
|
||||
```
|
||||
|
||||
Required identity fields are never inferred from a route key. A recognized field that does not apply to the selected driver is rejected instead of retained inertly.
|
||||
|
||||
## `codex_builtin`
|
||||
|
||||
```toml
|
||||
[routes.codex_chatgpt_builtin]
|
||||
name = "Built-in Codex with ChatGPT authentication"
|
||||
api_operator = "openai"
|
||||
access_product = "chatgpt_codex"
|
||||
wire_protocol = "codex_builtin"
|
||||
billing_mode = "chatgpt_subscription"
|
||||
driver = "codex_builtin"
|
||||
provider_id = "openai"
|
||||
auth = "chatgpt"
|
||||
resource_group = "chatgpt_subscription"
|
||||
transport_modalities = ["text", "image"]
|
||||
tool_calling = true
|
||||
parallel_tool_calls = true
|
||||
```
|
||||
|
||||
This preserves Codex’s native authentication, entitlement, tools, images, and active model catalog. A generated home uses the configured base home’s file-backed `auth.json` according to `auth_link_mode`; Codex 0.149 keyring-only state cannot be transferred to another `CODEX_HOME`.
|
||||
|
||||
Account-visible models and capabilities remain authoritative. The bundled catalog is a release baseline, not an entitlement claim.
|
||||
|
||||
## `codex_custom`
|
||||
|
||||
```toml
|
||||
[routes.private_responses]
|
||||
name = "Private Responses"
|
||||
api_operator = "example"
|
||||
access_product = "private_responses"
|
||||
wire_protocol = "openai_responses"
|
||||
billing_mode = "api"
|
||||
driver = "codex_custom"
|
||||
base_url = "https://llm.example.internal/v1"
|
||||
wire_api = "responses"
|
||||
credential_envs = ["PRIVATE_LLM_API_KEY"]
|
||||
transport_modalities = ["text", "image"]
|
||||
preserves_tool_media = true
|
||||
tool_result_modalities = ["text", "image"]
|
||||
|
||||
[routes.private_responses.http_headers]
|
||||
"X-Static-Tenant" = "engineering"
|
||||
|
||||
[routes.private_responses.env_http_headers]
|
||||
"X-Private-Token" = "PRIVATE_HEADER_VALUE"
|
||||
```
|
||||
|
||||
Use direct mode only when the endpoint faithfully implements Responses and direct transport is preferable to translation. The Codex process necessarily receives declared credential/header variables, so place sensitive work in a suitable OS boundary.
|
||||
|
||||
`wire_api` is currently `responses`. Direct retry fields are `request_max_retries`, `stream_max_retries`, and `stream_idle_timeout_ms`. Codex interprets zero as immediate expiry, not “disabled,” so route validation rejects it. Generated homes raise the effective idle floor to at least one hour and twice the role's warning-only stall interval. That value detects a genuinely broken stream; it is not a task deadline, goal budget, or instruction for the model to keep time.
|
||||
|
||||
## `codex_oss`
|
||||
|
||||
```toml
|
||||
[routes.ollama_codex_oss]
|
||||
name = "Codex native Ollama OSS mode"
|
||||
api_operator = "local"
|
||||
access_product = "ollama"
|
||||
wire_protocol = "codex_oss"
|
||||
billing_mode = "local"
|
||||
driver = "codex_oss"
|
||||
provider_id = "ollama"
|
||||
resource_group = "local_gpu_0"
|
||||
transport_modalities = ["text"]
|
||||
```
|
||||
|
||||
Supported `provider_id` values are `ollama` and `lmstudio`. Switchyard variants also exist for compatible local Chat endpoints when common routing and telemetry matter more than native transport.
|
||||
|
||||
## `switchyard`
|
||||
|
||||
Switchyard 0.2.0 supports these route protocols:
|
||||
|
||||
- `openai_chat`;
|
||||
- `openai_responses`;
|
||||
- `anthropic_messages`.
|
||||
|
||||
The compiler emits only routes used by one immutable snapshot. A profile with no Switchyard binding starts no gateway. Managed ingress is restricted to a loopback IP literal because the generated gateway has no client-auth layer.
|
||||
|
||||
Bundled Switchyard routes include:
|
||||
|
||||
- OpenCode Go Chat, Responses, and Anthropic Messages through `OPENCODE_API_KEY`;
|
||||
- OpenCode Zen Chat, Responses, and Anthropic Messages through the same `OPENCODE_API_KEY`;
|
||||
- OpenRouter Chat through `OPENROUTER_API_KEY`;
|
||||
- Z.AI Coding Plan Chat and Responses through `ZAI_CODING_API_KEY`;
|
||||
- Z.AI General API through the separately explicit `ZAI_API_KEY`;
|
||||
- a bundled llama.cpp model binding;
|
||||
- direct OpenAI/Anthropic and LM Studio/Ollama route definitions for explicit
|
||||
user catalog extensions. A route without a model record is not an executable
|
||||
bundled model binding.
|
||||
|
||||
Switchyard receives `extra_headers`, not environment-backed per-header mappings. API authentication uses `credential_envs`. MMO emits `max_retries = 1` by default, meaning one additional attempt; the accepted range is 0–10.
|
||||
|
||||
### Temporary Codex MCP namespace bridge
|
||||
|
||||
Codex 0.149 represents MCP servers as Responses `namespace` tools. Switchyard 0.2.0's Responses decoder retains top-level function and ID tools but does not retain that namespace shape. For a gateway whose recorded executable version is exactly `0.2.0`, MMO therefore asks the same Codex app-server for its granted MCP tool schemas, registers equivalent flat `mmo_mcp__SERVER__TOOL` dynamic functions when a thread is first created, and routes each dynamic call back through Codex's `mcpServer/tool/call`. The MMO-owned prefix is deliberate because Codex rejects dynamic declarations using its reserved native `mcp__` prefix. Codex remains the MCP connection owner and enforces its configured server/tool allowlists; the bridge does not read supervisor files or connect to MCP servers independently. MCP text, image, audio, error state, and non-redundant structured content are preserved; a structured value is omitted only when an existing text block parses to that exact value (including the common single-key result wrapper), avoiding duplicate context without discarding distinct information. Durable Codex threads retain the dynamic tool declarations, while every replacement host reconstructs only the in-memory routing map.
|
||||
|
||||
This is a release-scoped compatibility shim, not a second permanent MCP architecture. It is disabled for every Switchyard version other than `0.2.0`. The baseline/bridge guard test deliberately fails when the reviewed Switchyard baseline changes. Before accepting a newer baseline:
|
||||
|
||||
1. prove with the exact candidate binary that Codex 0.149 namespace tools reach both the Go and Zen model routes and that a real MCP call completes;
|
||||
2. prove new-thread, resumed-thread, worker, allowlist, error, and cancellation behavior without `dynamicTools`;
|
||||
3. remove the bridge constants, dynamic schema discovery/call routing, root/worker injection, and their compatibility tests;
|
||||
4. update this section, the architecture/veracity records, changelog, package manifest, and installed runtime.
|
||||
|
||||
An operator-supplied newer Switchyard is never assigned the 0.2.0 shim merely because the profile uses the `switchyard` driver. Its native namespace path must pass live qualification before production use.
|
||||
|
||||
#### Open removal tracker
|
||||
|
||||
Upstream [Switchyard PR #384](https://github.com/NVIDIA-NeMo/Switchyard/pull/384), merged as commit [`c7beccd4891fa5cfe3a3b94fdd376f5765864507`](https://github.com/NVIDIA-NeMo/Switchyard/commit/c7beccd4891fa5cfe3a3b94fdd376f5765864507) on 2026-08-20, implements native Codex namespace preservation. The current `v0.2.0` tag predates that commit, and no published tag contains it as of 2026-08-23. This tracker remains open until the first official `switchyard-server` release whose tag contains that commit is available.
|
||||
|
||||
The release containing that commit is an upgrade/removal trigger, not permission to retain both paths. The baseline bump must be one atomic change: run the qualification matrix above against the exact released binary, delete the dynamic bridge and its version constant, then update the pin. The existing `test_temporary_switchyard_bridge_must_be_removed_when_baseline_advances` test blocks a baseline-only update. If native qualification fails, keep `0.2.0` pinned and record the failure; do not extend the bridge to the newer release.
|
||||
|
||||
Z.AI Coding Plan uses distinct documented bases by wire protocol: Responses/Codex uses `https://api.z.ai/api/v1`, while OpenAI-compatible Chat uses `https://api.z.ai/api/coding/paas/v4`. Switchyard appends the protocol endpoint suffix. MMO does not retry one route through the other and does not fall back to the separately credentialed General API.
|
||||
|
||||
The Coding Plan Anthropic-compatible endpoint remains catalog-only: its documented Bearer authentication cannot be represented by Switchyard 0.2.0’s fixed Anthropic `x-api-key` behavior. No hidden transport fallback is attempted.
|
||||
|
||||
## `catalog_only`
|
||||
|
||||
Catalog-only records preserve complete inventory information without pretending the endpoint can run a Codex agent. Examples include OCR, image/video generation, audio transcription, hosted application agents, Google-native Zen models without an installed driver, and the incompatible Coding Plan Anthropic route.
|
||||
|
||||
```toml
|
||||
[routes.zai_general_catalog]
|
||||
name = "Z.AI media and specialist API catalog"
|
||||
api_operator = "zai"
|
||||
access_product = "zai_general_api"
|
||||
wire_protocol = "catalog_only"
|
||||
billing_mode = "catalog_only"
|
||||
driver = "catalog_only"
|
||||
transport_modalities = ["text", "image", "video", "audio", "file"]
|
||||
|
||||
[models.zai_general_catalog__glm_image]
|
||||
maker = "zai"
|
||||
route = "zai_general_catalog"
|
||||
upstream_id = "glm-image"
|
||||
kind = "image_generation"
|
||||
agent_compatible = false
|
||||
```
|
||||
|
||||
## Driver-specific fields
|
||||
|
||||
| Field | Drivers | Meaning |
|
||||
|---|---|---|
|
||||
| `api_operator` | all | API operator identity |
|
||||
| `access_product` | all | Plan/service identity |
|
||||
| `wire_protocol` | all | Exact protocol enum |
|
||||
| `billing_mode` | all | `api`, `subscription`, `chatgpt_subscription`, `local`, or `catalog_only` |
|
||||
| `provider_id` | built-in/OSS | Codex-owned provider identifier |
|
||||
| `base_url` | custom/Switchyard/catalog-only | Absolute HTTP(S) API root |
|
||||
| `credential_envs` | custom/Switchyard/catalog-only | Ordered valid environment names |
|
||||
| `wire_api` | custom | Direct Codex API; `responses` |
|
||||
| `http_headers` | custom | Static Codex headers |
|
||||
| `env_http_headers` | custom | Header name to environment name |
|
||||
| `extra_headers` | Switchyard | Static Switchyard headers |
|
||||
| `max_retries` | Switchyard | Additional attempts, 0–10 |
|
||||
| `request_max_retries` | custom | Direct request retries |
|
||||
| `stream_max_retries` | custom | Stream reconnections |
|
||||
| `stream_idle_timeout_ms` | custom | Provider stream-idle floor (at least 1,000 ms); generated app-server configs raise it through the role's execution window plus finalization grace when that is longer |
|
||||
| `transport_modalities` | all | Inputs the complete path preserves |
|
||||
| `transport_output_modalities` | all | Outputs the complete path preserves |
|
||||
| `preserves_tool_media` | all | Whether tool media survives transport |
|
||||
| `tool_result_modalities` | all | Preserved tool-result media types |
|
||||
| `supports_documents` | all | Complete file/document path |
|
||||
| `openrouter_policy` | OpenRouter only | Route-wide typed serving policy |
|
||||
|
||||
There are no singular credential aliases or legacy header-name aliases in the v2 contract. Use exactly `credential_envs`, `http_headers`, `env_http_headers`, and `extra_headers` for their documented drivers.
|
||||
|
||||
`base_url` must be an absolute HTTP(S) root with a nonempty host. Userinfo, control/space characters, backslashes, queries, and fragments are rejected. Header names and values are validated before downstream configuration is generated.
|
||||
|
||||
## Model transport fields
|
||||
|
||||
```toml
|
||||
[models.example_openai_chat__example_coder]
|
||||
maker = "example"
|
||||
route = "example_openai_chat"
|
||||
upstream_id = "example-coder"
|
||||
kind = "chat"
|
||||
agent_compatible = true
|
||||
modalities = ["text"]
|
||||
output_modalities = ["text"]
|
||||
tool_calling = true
|
||||
parallel_tool_calls = false
|
||||
structured_output = true
|
||||
context_window = 200000
|
||||
max_output_tokens = 32000
|
||||
reasoning_levels = ["none", "high"]
|
||||
default_reasoning = "high"
|
||||
resource_group = "example_pool"
|
||||
```
|
||||
|
||||
The compiler checks model capabilities against route capabilities. A maker/model name alone is never evidence that a particular transport preserves tools, images, documents, reasoning selectors, or structured output. `extra_body` is supported only for Switchyard-backed model targets and must contain finite JSON-compatible values.
|
||||
|
||||
## Credentials
|
||||
|
||||
Credential assignments belong in `~/.config/codex-mmo/credentials.env` or the launch environment. Profiles, catalogs, snapshots, and generated static configuration store names, never values. Worker/gateway environments filter conventional unrelated credential names and explicitly re-admit only the selected route and Tool MCP variables.
|
||||
|
||||
That filtering is name-based, not an OS sandbox or secret oracle. Use conventional secret names and sanitize unusual variables before launching untrusted work.
|
||||
|
||||
## Adding a route
|
||||
|
||||
Add a generation-8 fragment under `~/.config/codex-mmo/catalog.d/`, then run:
|
||||
|
||||
```bash
|
||||
codex-mmo catalog verify
|
||||
codex-mmo catalog routes
|
||||
codex-mmo catalog models --route ROUTE_KEY
|
||||
codex-mmo profile validate PROFILE
|
||||
```
|
||||
|
||||
No runtime plugin is needed when an existing driver fits. Executable driver plugins are intentionally outside profile packs.
|
||||
@@ -0,0 +1,260 @@
|
||||
# Security Model
|
||||
|
||||
Codex MMO coordinates tool-using models that may read or modify a workspace and may call external routes. Security depends on boundaries around profile installation, credentials, process execution, delegation authority, isolated writes, network access, and result trust.
|
||||
|
||||
## Threat model
|
||||
|
||||
The platform assumes these may be untrusted or fallible:
|
||||
|
||||
- Downloaded profile packs.
|
||||
- Model-generated tool calls and prose.
|
||||
- Repository content containing prompt injection or malicious instructions.
|
||||
- Low-capability or contradictory worker output.
|
||||
- External route availability, serving identity, and metadata.
|
||||
- Third-party Tool MCP server instructions, schemas, tool results, and side effects.
|
||||
- A worker that exceeds its delegated logical scope.
|
||||
- A local compatible endpoint that does not faithfully implement its advertised protocol.
|
||||
|
||||
The platform does not assume a compromised operating-system account, root-level attacker, or malicious replacement of installed Codex/Switchyard binaries can be contained by application-level policy.
|
||||
|
||||
## Static profile packs
|
||||
|
||||
Profiles are data, not plugins. Installation rejects:
|
||||
|
||||
- Unexpected top-level files.
|
||||
- Executable files.
|
||||
- Python and shell code.
|
||||
- Symlinks.
|
||||
- Paths that escape the pack.
|
||||
- Archive traversal and unsafe members.
|
||||
- Unsupported instruction or contract file types.
|
||||
|
||||
The package validator repeats static safety checks for all bundled profiles. Profile instructions can still influence model behavior, so inspect third-party Markdown and TOML before installation. A profile may grant a Tool MCP server by stable ID, but cannot introduce or modify its command, URL, authentication, headers, timeouts, approval policy, or operator tool allowlist. Those definitions remain in the operator-owned registry outside the profile pack.
|
||||
|
||||
## Credential isolation
|
||||
|
||||
API credentials belong in:
|
||||
|
||||
```text
|
||||
~/.config/codex-mmo/credentials.env
|
||||
```
|
||||
|
||||
or the process environment. Profile, catalog, Tool MCP registry, and snapshot files contain environment variable names, never literal keys.
|
||||
|
||||
For Switchyard-backed models:
|
||||
|
||||
- The managed gateway binds only to an IPv4 or IPv6 loopback literal. Non-loopback, hostname, bracketed, and scoped-address settings are rejected before startup because Switchyard 0.2.0's generated ingress is not authenticated.
|
||||
- The snapshot determines exactly which routes are needed.
|
||||
- The gateway explicitly receives the selected routes' declared credentials; common unrelated credential-like names are filtered.
|
||||
- Worker shell environments filter common credential-like names unless a direct selected route requires them.
|
||||
- Discovery reports never serialize key values.
|
||||
- Catalog discovery rejects cross-origin redirects before a configured or explicit authorization header can be forwarded.
|
||||
- Credentials are preserved across a validated current-generation reinstall and excluded from release archives.
|
||||
|
||||
For third-party Tool MCP servers:
|
||||
|
||||
- Schema version 8 supports environment-backed credentials, not inherited Codex MCP OAuth state.
|
||||
- Generated homes receive only credential variable names; values are selected from `credentials.env` or the launch environment immediately before process execution.
|
||||
- Each Codex process receives only variables referenced by the Tool MCP servers compiled into that process.
|
||||
- Native roles share a parent Codex process, so their Tool MCP credential values are visible to that parent and its in-process native siblings. Use a separately launched Agent-MCP role when credential isolation is required.
|
||||
|
||||
Built-in ChatGPT/Codex authentication remains in the operator's base Codex home. The default `shared` mode gives generated homes a symlink to its file-backed `auth.json`; `copy` deliberately creates a protected `0600` copy, and `none` disables propagation. Codex 0.149 keyring entries are bound to canonical `CODEX_HOME`, so keyring-only auth cannot be inherited by an isolated home and fails the doctor credential gate. MMO never propagates Codex's separate `.credentials.json` MCP OAuth store.
|
||||
|
||||
Logs and prompts can still contain sensitive repository data or model output. Protect the state directory with normal user permissions and apply retention appropriate to the workspace.
|
||||
|
||||
Environment filtering is deliberately name-based so ordinary toolchain, locale, terminal, and path configuration continues to work. It recognizes conventional `KEY`, `TOKEN`, `SECRET`, `PASSWORD`, `CREDENTIAL`, `COOKIE`, `AUTH`, and `PRIVATE` components; no process can infer that an arbitrary variable such as a project-specific URL contains a secret. Sanitize the parent environment or use conventional credential names before launching MMO. This is a documented containment limit, not a claim of complete secret discovery.
|
||||
|
||||
## Process isolation and durable hosting
|
||||
|
||||
Every root and Agent-MCP worker has a separate detached runner. Each privately hosts
|
||||
`codex app-server --listen unix://…` with:
|
||||
|
||||
- A minimal generated `CODEX_HOME`.
|
||||
- A role-pinned model/route.
|
||||
- A sandbox ceiling from the profile and root session.
|
||||
- No Tool MCP servers except explicit grants for that role, and no internal `mmo_mesh` server unless generated for authorized Agent-MCP nesting.
|
||||
- Network and web search disabled unless the role permits them.
|
||||
- A profile-compiled `goal` token budget/ceiling or clock-free `turn` lifecycle.
|
||||
- A warning-only stall interval and bounded explicit terminal-serialization grace.
|
||||
- A separate process group for cancellation.
|
||||
- A non-ephemeral thread plus authoritative terminal history, stderr, headerless app-server protocol events, prompt,
|
||||
partial/final result, pending requests, goal state, metadata, patch, and audit artifacts.
|
||||
|
||||
The app-server host plus persistent thread is execution identity; any TUI/noninteractive/control
|
||||
connection is a detachable client. A runner reconnects to the exact host/thread after transport
|
||||
failure, or starts one replacement host only to resume that exact persisted thread. If hosting
|
||||
cannot recover, work becomes `suspended` with trace, terminal history, partial result, pending
|
||||
requests, and isolated patch retained. Graceful stop asks agents to synthesize from retained
|
||||
evidence; immediate cancellation targets exact process groups. Neither path erases durable evidence.
|
||||
|
||||
MMO records the isolated process group, PID, and Linux process-start token as soon as app-server
|
||||
is spawned, before protocol initialization or `thread/start`. Cleanup rechecks that fingerprint
|
||||
before signalling, so PID reuse cannot redirect cancellation. Worker-runner loss retires that
|
||||
job's app-server/Tool-MCP group before an explicit same-thread continuation, while ordinary root
|
||||
controller loss preserves a separately owned live root host for reattachment. Bootstrap failure,
|
||||
stale terminal transitions, stop, and cancel still retire their exact owned groups, so they do not
|
||||
leave unowned descendants behind. Detach deliberately preserves the host fingerprint and socket while
|
||||
work continues; only full stop/cancel retires them.
|
||||
|
||||
`approval_policy = "never"` permits autonomous work only inside the compiled sandbox and
|
||||
automatically declines app-server escalation requests. `untrusted` and `on-request` keep an
|
||||
approval request pending; only a role with an exact per-target `respond` grant can inspect it and send
|
||||
the method-specific response through `agent_respond`.
|
||||
|
||||
## Delegation authority
|
||||
|
||||
MCP caller identity is capability-based. Root/native bearer values are stored only in a protected `0600` session capability file so a detached host can be controlled after the original client exits; public session/run state contains only SHA-256 digests. The values remain stable for the one immutable run and the private file is destroyed when the session becomes terminal. Every MCP job receives a distinct inherited job capability, and a native participant that may call the mesh additionally receives a role capability in protected generated configuration. Together those capabilities are tied to:
|
||||
|
||||
- Snapshot hash.
|
||||
- Root session.
|
||||
- Execution run.
|
||||
- Agent role.
|
||||
- Parent lineage.
|
||||
- Declared child allowlist.
|
||||
|
||||
The supervisor validates the capabilities against durable session/job state before accepting the caller identity. A caller cannot gain authority by changing `MMO_CALLER_AGENT` or writing another role or parent into tool arguments. These are same-user process capabilities, not an operating-system isolation boundary: a sufficiently privileged process running as the same account may inspect another process or the protected state tree.
|
||||
|
||||
Persistent resume also validates that exactly one run exists, every session-owned Codex home and run record remains contained under its recorded session without symlink redirection, and generated model catalogs still match their recorded SHA-256 digests. MMO-owned `AGENTS.md` and orchestration skills are refreshed from the same integrity-checked snapshot; neither grants authority beyond runtime capabilities. Drift suspends the same run closed rather than substituting current data or creating a replacement.
|
||||
|
||||
Admission checks include:
|
||||
|
||||
- Directed `can_spawn` edge.
|
||||
- Maximum depth and ancestor-role rejection.
|
||||
- Reusable active-agent and per-caller active-child limits.
|
||||
- Role and resource capacity.
|
||||
- Task kind and size.
|
||||
- Task-size, goal/turn mode, token-ceiling, stall-warning, and finalization-grace bounds.
|
||||
- Sandbox ceiling.
|
||||
- Attachment permission and containment.
|
||||
- Write-scope requirements and conflicts.
|
||||
- Per-route credential/endpoint availability and per-role active caps.
|
||||
- Result visibility.
|
||||
|
||||
Pure native nested agents do not pass through this enforcement boundary. Native nesting is disabled by default in hybrid profiles; enabling it is an explicit advisory tradeoff.
|
||||
|
||||
## Tool MCP authority
|
||||
|
||||
Tool MCP is a separate authority boundary from Agent MCP. The internal `mmo_mesh` server controls bounded participant delegation; a third-party Tool MCP server can act on whatever external system, local application, account, or data source that server itself can access.
|
||||
|
||||
Profile grants can only narrow the operator registry's finite `enabled_tools` allowlist. Generated role configuration explicitly disables ungranted servers and tools, including across Codex's recursive native-agent config merge. The operator also owns server-wide and per-tool approval modes. Keep mutating operations out of the allowlist when they are unnecessary, and use prompt approval for consequential operations.
|
||||
|
||||
Codex role declarations do not sandbox the server behind a Tool MCP connection:
|
||||
|
||||
- `permissions = "read-only"` constrains Codex workspace access, not an MCP tool's ability to modify an IDA database, browser session, SaaS account, or remote repository.
|
||||
- `network_access = false` constrains sandboxed shell networking, not an HTTP MCP connection or a local MCP subprocess's own network activity.
|
||||
- `required = false` changes startup-failure handling; it does not reduce a server's authority when the server is available.
|
||||
|
||||
Review the MCP implementation, deployment identity, advertised schemas, and downstream authorization independently. Use separate service credentials and least-privilege accounts where the external system supports them.
|
||||
|
||||
## Workspace writes
|
||||
|
||||
MCP writers execute in isolated Git worktrees pinned to a captured base. Admission compares canonical scopes and rejects overlap even when two sessions use different, nested workspace roots. The runner captures changed paths, modes, base fingerprints, text/binary patch artifacts, and hashes. Out-of-scope changes, symlinks/special artifacts, unsafe patch members, or boundary failures fail the job and taint the session; they do not modify the canonical checkout.
|
||||
|
||||
A successfully completed result starts unread. An authorized ancestor must read it and explicitly accept or reject it. Suspended jobs retain their thread and can be continued; stopped, failed, and cancelled jobs remain readable but cannot be dispositioned as successes. Only an accepted result may be integrated, and integration rechecks patch identity, base fingerprints, scope, lease, and `git apply --check`. Disposition transitions are monotonic and audited.
|
||||
|
||||
Limitations:
|
||||
|
||||
- Patch/hash correlation does not prove semantic correctness.
|
||||
- Disjoint paths can still represent logically coupled state.
|
||||
- Native workers do not use MCP worktrees, scopes, disposition, or integration; all featured bundled native roles are read-only.
|
||||
- A root with workspace-write permission remains powerful.
|
||||
- Integration can conflict with canonical changes made after the worker base; the runtime rejects rather than guesses a merge.
|
||||
|
||||
The root must inspect accepted diffs, integrate deliberately, reconcile coupled changes, and run canonical validation.
|
||||
|
||||
## Low-trust models
|
||||
|
||||
A low-trust role is mechanically constrained to supervised Agent-MCP execution with read-only Codex workspace access, at most one active instance, no descendants, no network or attachments, a single-slot resource, strict structured output, and mandatory verification. A terminal instance releases its slot, but every later invocation is subject to the same constraints. Callers must use a typed `literal_task`; the runtime validates and supplies the literal inputs instead of accepting an interpretive free-form brief. That workspace restriction does not make a granted Tool MCP server read-only; only grant independently reviewed, non-mutating tools to a low-trust role.
|
||||
|
||||
Do not promote low-trust output directly into:
|
||||
|
||||
- Architecture decisions.
|
||||
- Security conclusions.
|
||||
- Concurrency or data-integrity reasoning.
|
||||
- Migrations.
|
||||
- Implementation changes.
|
||||
- Final recommendations.
|
||||
|
||||
Resolve conflicts from primary evidence. A structured evidence document is easier to inspect; it is not proof that the evidence was interpreted correctly.
|
||||
|
||||
Durable app-server events, partial evidence, and contract-shaped worker output remain untrusted
|
||||
claims. MMO filters controller-visible traces, validates final contracts and correlated command
|
||||
or artifact evidence, and preserves interrupted work, but none of those mechanisms proves the
|
||||
worker's interpretation. Only a successfully completed result may enter the accept/reject
|
||||
lifecycle, and only an accepted isolated patch may be considered for audited integration.
|
||||
|
||||
## Prompt injection
|
||||
|
||||
Repository files, logs, web pages, route output, MCP server instructions, and MCP tool results may contain instructions that conflict with the user or role policy. Agents are instructed to treat these as data unless the root explicitly authorizes them.
|
||||
|
||||
Operational controls:
|
||||
|
||||
- Keep web search and shell network access disabled by default.
|
||||
- Grant only reviewed Tool MCP servers and finite tool subsets to roles that need them.
|
||||
- Use read-only reviewers for untrusted content analysis.
|
||||
- Delegate only bounded tasks with explicit success criteria.
|
||||
- Do not expose credentials through prompts or command output.
|
||||
- Review proposed commands that cross trust boundaries.
|
||||
- Use the narrowest workspace and route permissions that satisfy the task.
|
||||
|
||||
## Network and web access
|
||||
|
||||
`network_access` and `web_search` are independent role declarations. Network-enabled work broadens the attack surface and may transmit repository data to external services. Neither declaration disables or confines an explicitly granted Tool MCP server; its transport and its own downstream network behavior are separate.
|
||||
|
||||
Use live web access only when current external information is genuinely required. Prefer cached/read-only evidence roles for research and keep implementation roles offline unless dependency access is necessary.
|
||||
|
||||
## Multimodal data
|
||||
|
||||
Images and documents may contain sensitive information. A profile must explicitly allow attachments and require the modality. The compiler validates the model/transport path, but the operator remains responsible for route/operator data-handling policy.
|
||||
|
||||
Avoid protocol conversions known to flatten image/document blocks into text when visual fidelity is material. Use a native capable path or reject the binding.
|
||||
|
||||
## Catalog and discovery trust
|
||||
|
||||
The release catalog is a dated baseline. Live discovery verifies IDs but does not infer unknown capabilities. A newly observed model remains non-executable until an operator supplies reviewed metadata.
|
||||
|
||||
This prevents a hosted inventory name addition from automatically receiving tool access, workspace permissions, or multimodal claims.
|
||||
|
||||
## State and cleanup
|
||||
|
||||
Default state contains prompts, outputs, logs, snapshots, and evaluation artifacts. Inspect it with:
|
||||
|
||||
```bash
|
||||
codex-mmo session list
|
||||
codex-mmo jobs list
|
||||
codex-mmo gateway list
|
||||
```
|
||||
|
||||
Apply retention with:
|
||||
|
||||
```bash
|
||||
codex-mmo clean --job-days 14 --session-days 14
|
||||
```
|
||||
|
||||
Use `--dry-run` first. Cleanup does not replace organization-specific secure deletion or backup policy.
|
||||
Only terminal sessions become cleanup candidates after their `last_active_at` age exceeds `session_retention_days`. Detached, paused, suspended, and running sessions remain protected regardless of age. Removing a terminal record also removes its generated home and retained thread artifacts, never an external workspace or Tool MCP system.
|
||||
|
||||
## Release integrity
|
||||
|
||||
Release archives include `PACKAGE-MANIFEST.json` with per-file SHA-256 values, normalized modes, and sizes. Verification rejects exact permission drift as well as path, type, size, and digest drift. Tar inspection and extraction are streamed under progressive member-count and expanded-byte ceilings. The release builder emits archive checksums and verifies byte-for-byte reproducibility with an independent second build.
|
||||
|
||||
Before installation:
|
||||
|
||||
```bash
|
||||
sha256sum -c codex-multimodel-orchestrator-8.0.0-SHA256SUMS.txt
|
||||
```
|
||||
|
||||
The package acceptance validator scans for common credential formats, private keys, unresolved placeholders, executable profile files, symlinks, bytecode, syntax errors, and catalog/profile drift.
|
||||
|
||||
## Operational boundary
|
||||
|
||||
Codex MMO is an orchestration and enforcement layer, not a substitute for:
|
||||
|
||||
- Operating-system isolation.
|
||||
- Container or virtual-machine boundaries for hostile repositories.
|
||||
- Provider access controls and billing limits.
|
||||
- Source-control review.
|
||||
- Secret scanning and dependency security tools.
|
||||
- Authorization to test or modify a system.
|
||||
|
||||
Use a disposable container or VM for code that is actively malicious or not trusted to be opened by a tool-using agent.
|
||||
@@ -0,0 +1,168 @@
|
||||
# Tool MCP Servers
|
||||
|
||||
Codex MMO distinguishes two unrelated uses of MCP:
|
||||
|
||||
- **Agent MCP** is the internal `mmo_mesh` server that launches and supervises profile
|
||||
participants.
|
||||
- **Tool MCP** connects Codex agents to third-party tools and context, such as a web-research
|
||||
service or a reverse-engineering application.
|
||||
|
||||
`mmo_mesh` is runtime-owned and reserved. Tool MCP definitions use operator-owned registry files;
|
||||
profile packs contain grants only. This keeps installed profile packs static and prevents one from
|
||||
introducing a local executable or remote endpoint.
|
||||
|
||||
Codex MMO compiles tool definitions to Codex's documented `[mcp_servers.NAME]` configuration.
|
||||
Codex 0.149 supports local STDIO processes and Streamable HTTP endpoints, environment-backed
|
||||
authentication, tool allow/deny lists, startup/tool timeouts, and server/tool approval policy. See
|
||||
the official [Codex MCP documentation](https://developers.openai.com/codex/mcp) and
|
||||
[configuration reference](https://developers.openai.com/codex/config-reference).
|
||||
|
||||
## Operator registry
|
||||
|
||||
Place registry fragments under:
|
||||
|
||||
```text
|
||||
~/.config/codex-mmo/tool-mcp.d/*.toml
|
||||
```
|
||||
|
||||
Fragments load in lexical filename order. When multiple fragments define the same server ID, the
|
||||
later definition replaces the complete earlier entry. Fields are not recursively retained across a
|
||||
transport change.
|
||||
|
||||
This example is illustrative; use the command, URL, and exact tool names documented by the MCP
|
||||
server you install:
|
||||
|
||||
```toml
|
||||
schema_version = 8
|
||||
|
||||
[tool_mcp_servers.firecrawl]
|
||||
transport = "streamable_http"
|
||||
url = "https://mcp.example.invalid/"
|
||||
bearer_token_env_var = "FIRECRAWL_API_KEY"
|
||||
enabled_tools = ["search", "scrape"]
|
||||
default_tools_approval_mode = "writes"
|
||||
startup_timeout_sec = 15
|
||||
tool_timeout_sec = 120
|
||||
|
||||
[tool_mcp_servers.firecrawl.tools.scrape]
|
||||
approval_mode = "prompt"
|
||||
|
||||
[tool_mcp_servers.ida_pro]
|
||||
transport = "stdio"
|
||||
command = "ida-pro-mcp"
|
||||
args = []
|
||||
env_vars = ["IDA_MCP_TOKEN"]
|
||||
enabled_tools = ["list_functions", "decompile"]
|
||||
default_tools_approval_mode = "writes"
|
||||
```
|
||||
|
||||
Every definition requires:
|
||||
|
||||
- A stable lowercase server ID other than `mmo_mesh`.
|
||||
- `transport = "stdio"` plus `command`, or `transport = "streamable_http"` plus `url`.
|
||||
- A non-empty `enabled_tools` array. This is the operator's maximum exposure, including for future
|
||||
profile grants.
|
||||
- An explicit `default_tools_approval_mode`: `auto`, `prompt`, `writes`, or `approve`.
|
||||
|
||||
The STDIO `command` may be a bare executable resolved through `PATH` or an
|
||||
absolute/tilde-expanded path. STDIO definitions may also use `args`, absolute/tilde-expanded `cwd`,
|
||||
non-secret literal `env`, and environment-forwarded `env_vars`. HTTP definitions may use
|
||||
`bearer_token_env_var`, non-secret `http_headers`, and environment-backed `env_http_headers`. Both transports accept
|
||||
`startup_timeout_sec`, `tool_timeout_sec`, `supports_parallel_tool_calls`, and per-tool approval
|
||||
overrides under `tools`.
|
||||
|
||||
The private app-server lifecycle wait is automatically at least 30 seconds longer than the
|
||||
largest enabled `startup_timeout_sec` for that role, and never less than twenty minutes. A valid long
|
||||
MCP startup therefore is not cut off by an unrelated fixed thread-start timeout. The separate
|
||||
`tool_timeout_sec` continues to bound each Tool MCP call; it is not a model-turn deadline.
|
||||
The generated `mmo_mesh` timeout is independently derived from the largest app-server lifecycle
|
||||
reachable through that caller's exact spawn/control grants, plus protocol overhead.
|
||||
|
||||
Schema version 8 rejects OAuth fields, plaintext bearer tokens, recognizable credential-bearing
|
||||
literal environment/header fields, transport-incompatible fields, and unknown fields. OAuth is not
|
||||
copied from the base Codex home because `.credentials.json` can contain credentials for unrelated
|
||||
MCP servers.
|
||||
|
||||
## Credentials
|
||||
|
||||
Put values in the existing operator credential file or the launch environment:
|
||||
|
||||
```bash
|
||||
${EDITOR:-vi} ~/.config/codex-mmo/credentials.env
|
||||
```
|
||||
|
||||
```dotenv
|
||||
FIRECRAWL_API_KEY=replace-me
|
||||
IDA_MCP_TOKEN=replace-me
|
||||
```
|
||||
|
||||
Snapshots and generated TOML contain only environment variable names. At session launch, the
|
||||
runtime forwards only names referenced by a server selected for that Codex process. Missing values
|
||||
are reported by `tool-mcp validate` and `profile doctor`; a profile's `required` policy determines
|
||||
whether Codex treats server startup failure as fatal.
|
||||
|
||||
## Profile grants
|
||||
|
||||
Grant servers independently to each role:
|
||||
|
||||
```toml
|
||||
[agents.researcher.tool_mcp_servers.firecrawl]
|
||||
required = true
|
||||
enabled_tools = ["search"]
|
||||
|
||||
[agents.reverse_engineer.tool_mcp_servers.ida_pro]
|
||||
required = true
|
||||
```
|
||||
|
||||
`required` defaults to `true`. Omitting `enabled_tools` grants the complete operator allowlist;
|
||||
otherwise it must be a non-empty subset. A profile cannot change the command, URL, credentials,
|
||||
headers, timeouts, parallel-call declaration, or approval policy. Omitting a server grant disables
|
||||
that server for the role.
|
||||
|
||||
The compiler stores normalized definitions and grants in the immutable snapshot. Changing a
|
||||
registry definition therefore changes future snapshot identity without changing a running session.
|
||||
|
||||
## Native-role behavior
|
||||
|
||||
Native roles share their parent's Codex process, and Codex recursively merges custom-agent config
|
||||
layers. The generated root config consequently contains the union of servers used by the root and
|
||||
reachable native roles. Each role layer explicitly enables its grants and disables every other
|
||||
server. The operator `enabled_tools` list remains constant, while role-specific narrowing is emitted
|
||||
as `disabled_tools`; this lets a child clear a narrower inherited deny-list without exceeding the
|
||||
operator maximum.
|
||||
|
||||
Tool visibility remains role-scoped, but process environment variables do not. If a native role
|
||||
uses an environment-authenticated MCP, its parent and sibling native agents share the containing
|
||||
process environment. Use a separately launched Agent-MCP participant when strict credential
|
||||
isolation is required.
|
||||
|
||||
## Security boundary
|
||||
|
||||
Codex shell settings do not constrain an external MCP server:
|
||||
|
||||
- `permissions = "read-only"` controls the Codex filesystem sandbox, not whether an MCP tool can
|
||||
mutate an IDA database, browser, SaaS account, or other external state.
|
||||
- `network_access = false` controls sandboxed shell networking, not an HTTP MCP connection or a
|
||||
local MCP process's own networking.
|
||||
- MCP server instructions and tool results are external content and may carry prompt injection.
|
||||
|
||||
Use narrow operator allowlists, conservative approval modes, per-tool `prompt` overrides for
|
||||
mutating operations, role-specific grants, and independent verification of consequential results.
|
||||
|
||||
## Inspection and diagnostics
|
||||
|
||||
These commands validate configuration without launching an MCP server:
|
||||
|
||||
```bash
|
||||
codex-mmo tool-mcp list
|
||||
codex-mmo tool-mcp show firecrawl
|
||||
codex-mmo tool-mcp validate
|
||||
codex-mmo tool-mcp validate firecrawl
|
||||
codex-mmo profile show PROFILE --resolved
|
||||
codex-mmo profile doctor PROFILE
|
||||
```
|
||||
|
||||
Readiness checks report command resolution, working-directory availability, and whether referenced
|
||||
environment variables are present. They never print credential values. Actual protocol/tool
|
||||
behavior remains a live acceptance concern; exercise it through a profile smoke task after reviewing
|
||||
the server and its tool schema.
|
||||
@@ -0,0 +1,374 @@
|
||||
# Troubleshooting
|
||||
|
||||
## Start with deterministic checks
|
||||
|
||||
From the source or extracted release directory:
|
||||
|
||||
```bash
|
||||
make test
|
||||
make validate
|
||||
```
|
||||
|
||||
From an installed environment:
|
||||
|
||||
```bash
|
||||
codex-mmo validate --all-profiles
|
||||
codex-mmo tool-mcp validate
|
||||
codex-mmo doctor --profile PROFILE
|
||||
```
|
||||
|
||||
Add `--live --probe` only after local validation succeeds.
|
||||
|
||||
## A command is rejected or produces unexpected output
|
||||
|
||||
Use command-local help and preserve stderr separately from requested data:
|
||||
|
||||
```bash
|
||||
codex-mmo session list --help
|
||||
codex-mmo session list >sessions.json 2>diagnostics.log
|
||||
printf 'status=%s\n' "$?"
|
||||
```
|
||||
|
||||
Structured commands use human output only when stdout is a terminal; a pipe or redirection selects
|
||||
strict JSON automatically. Use `--json` to force JSON on a terminal. Progress, warnings, usage, and
|
||||
errors stay on stderr, and `--quiet` suppresses only interactive stage progress. A syntax or invalid
|
||||
option combination returns status 2; an operation or validation failure returns status 1. Add
|
||||
`--debug` only when a traceback is useful for diagnosis.
|
||||
|
||||
Global `--json`, `--quiet`, and `--debug` options may appear around a control command but must occur
|
||||
before the explicit `--` that begins Codex-owned arguments. Options are not abbreviated. For
|
||||
noninteractive root work, provide a positional prompt or pipe non-empty input; `codex-mmo exec`
|
||||
deliberately does not read an implicit prompt from a terminal. See [Command-line interface](CLI.md)
|
||||
for the complete output and exit-status contract.
|
||||
|
||||
## The Codex TUI is monochrome, malformed, or does not resize
|
||||
|
||||
Confirm that version 8.0.0 and the exact supported Codex 0.149.0 client are installed:
|
||||
|
||||
```bash
|
||||
codex-mmo version
|
||||
codex --version
|
||||
```
|
||||
|
||||
Start Codex MMO with both stdin and stdout attached to an interactive terminal rather than piping either through a log collector:
|
||||
|
||||
```bash
|
||||
cd /path/to/repository
|
||||
codex-mmo
|
||||
```
|
||||
|
||||
Stderr may be redirected to a diagnostic file without disabling the interactive handoff. Interactive runs preserve the caller's foreground TTY and do not inject `NO_COLOR`. Check whether your shell already exports a color-disabling value:
|
||||
|
||||
```bash
|
||||
printf 'TERM=%s COLORTERM=%s NO_COLOR=%s\n' "$TERM" "${COLORTERM-}" "${NO_COLOR-}"
|
||||
```
|
||||
|
||||
Unset `NO_COLOR` only when you do not want that standard behavior:
|
||||
|
||||
```bash
|
||||
unset NO_COLOR
|
||||
```
|
||||
|
||||
The 8.0.0 acceptance suite exercises the remote Codex TUI through a pseudo-terminal while stderr is redirected, checks raw ANSI passthrough, transfers foreground process-group ownership, changes the terminal from 80x24 to 132x42, and requires the client to receive the resulting resize event while its app-server host remains independent.
|
||||
|
||||
## Codex does not resume an MMO conversation
|
||||
|
||||
Use MMO's lifecycle command rather than forwarding Codex's subcommand through a new run:
|
||||
|
||||
```bash
|
||||
codex-mmo resume --last
|
||||
codex-mmo resume SESSION_ID
|
||||
codex-mmo resume ROOT_CODEX_THREAD_ID
|
||||
codex-mmo resume --last --all
|
||||
```
|
||||
|
||||
`codex-mmo run resume` is rejected because `run` creates a deliberately new isolated session. A valid resume reattaches to the original session ID, one immutable run, snapshot, generated `CODEX_HOME`, app-server host, current root-thread generation, capabilities, budgets, and working-directory boundary. A controller or lost host process may be replaced, but never the MMO session/run identity. An older root thread ID remains a valid locator for the session and resolves to its current generation.
|
||||
|
||||
Inspect the lifecycle with:
|
||||
|
||||
```bash
|
||||
codex-mmo session show SESSION_ID
|
||||
codex-mmo session runs SESSION_ID
|
||||
codex-mmo jobs list --run RUN_ID
|
||||
```
|
||||
|
||||
Only canonical generation-8 sessions with exact package/profile version 8.0.0 in `detached`,
|
||||
`paused`, or `suspended` state are eligible. Other records are rejected without a compatibility
|
||||
reader. If startup exits before
|
||||
MMO records a thread ID, the session fails truthfully; resume never searches rollout files or the
|
||||
user's general Codex history. It may query `thread/list` only inside the pinned session-isolated
|
||||
root home to reconcile a top-level successor already created by the attached stock TUI. Resume
|
||||
restarts a stopped managed gateway before validating generated homes and suspends the same run if
|
||||
the pinned cwd, snapshot, executable, required root-route credential, capability document, or
|
||||
required MCP dependency cannot be restored. Missing optional worker credentials remain explicit
|
||||
degraded routes. Dead current-run workers become `suspended` with retained partial evidence before
|
||||
the root resumes. Terminal `completed`, `failed`, `stopped`, and `cancelled` sessions cannot be resumed.
|
||||
A tainted session requires `--allow-tainted`; delegation and patch-integration restrictions remain.
|
||||
|
||||
If a TUI context refresh appears blank or reopening lands before the refresh, inspect
|
||||
`root_thread_generation`, `root_thread_lineage`, and `root_thread_transition` in `session show`.
|
||||
MMO adopts a fresh thread only when it is persistent, top-level, in the pinned cwd, observed through
|
||||
the root host while the fingerprinted TUI is attached, and the prior root has no active turn. Native
|
||||
child and fork threads are deliberately excluded. Do not start a replacement MMO session or copy
|
||||
rollout files; close the TUI so the session becomes detached, then resume by the MMO session ID or
|
||||
any thread ID already recorded in its lineage.
|
||||
|
||||
## Codex reports that model metadata was not found
|
||||
|
||||
A generated model such as `mmo-<snapshot>-zai_coding_responses__glm_5_3` must appear verbatim in the session's `models.json`, and the same file must be referenced by `model_catalog_json` in that agent home's `config.toml`. Version 8.0.0 generates these files for every external model process.
|
||||
|
||||
Reinstall the current development build after clearing incompatible active state, then start a **new** session:
|
||||
|
||||
```bash
|
||||
./install.sh
|
||||
codex-mmo profile compile adaptive-engineering
|
||||
codex-mmo
|
||||
```
|
||||
|
||||
Inspect the new session with:
|
||||
|
||||
```bash
|
||||
codex-mmo session list
|
||||
codex-mmo session show SESSION_ID
|
||||
```
|
||||
|
||||
The session record exposes each generated `CODEX_HOME`, `model_catalog_json`, and spawn- or control-capable role's `orchestration_skill` path. Verify that `AGENTS.md` and `skills/mmo-profile-orchestration/SKILL.md` match the immutable snapshot payload and that `config.toml` enables the exact `SKILL.md` path through `skills.config`. Verify that the selected `model` exactly matches one `slug` in `models.json`. Hybrid native/external homes must contain both the active Codex bundled rows and the generated external route rows. Do not work around a warning by accepting fallback metadata or copying guidance from another profile: either can change model behavior, context accounting, reasoning controls, compaction, tool shape, or delegation authority.
|
||||
|
||||
## A profile does not compile
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
codex-mmo profile validate PROFILE
|
||||
codex-mmo profile show PROFILE --resolved
|
||||
codex-mmo catalog model ROUTE__MODEL_KEY
|
||||
```
|
||||
|
||||
Common causes are a missing exact binding, unsupported modality, invalid reasoning level, unknown resource group, prohibited spawn edge, missing output contract, or a route that would lose required multimodal data.
|
||||
|
||||
If the preserved `active-profile` value names a retired bundled profile, select a current profile explicitly:
|
||||
|
||||
```bash
|
||||
codex-mmo profile list
|
||||
codex-mmo profile use adaptive-engineering
|
||||
```
|
||||
|
||||
There is no retired-ID alias, older-generation reader, or automatic profile migration.
|
||||
|
||||
If the error names an undefined Tool MCP server, inspect the profile's per-agent grants and the operator registry:
|
||||
|
||||
```bash
|
||||
codex-mmo tool-mcp list
|
||||
codex-mmo tool-mcp show SERVER
|
||||
```
|
||||
|
||||
Server definitions belong under `~/.config/codex-mmo/tool-mcp.d/*.toml`; a profile intentionally cannot supply one.
|
||||
|
||||
## A route or model is missing
|
||||
|
||||
Inspect the reviewed baseline and live account inventory:
|
||||
|
||||
```bash
|
||||
codex-mmo catalog inventory
|
||||
codex-mmo catalog models --inventory openrouter
|
||||
codex-mmo catalog models --inventory opencode-zen
|
||||
codex-mmo catalog models --inventory opencode-go
|
||||
codex-mmo catalog models --inventory zai-api
|
||||
codex-mmo catalog models --inventory openai-codex
|
||||
codex-mmo catalog verify --remote --codex
|
||||
```
|
||||
|
||||
Hosted inventories change. Refresh only after reviewing the report:
|
||||
|
||||
```bash
|
||||
codex-mmo catalog refresh --install-codex-overlay
|
||||
```
|
||||
|
||||
A discovered model ID is not automatically granted capabilities that the sources did not establish through that complete route.
|
||||
|
||||
## Authentication fails
|
||||
|
||||
For built-in Codex/ChatGPT access:
|
||||
|
||||
```bash
|
||||
codex-mmo auth status
|
||||
codex-mmo auth login
|
||||
```
|
||||
|
||||
For external routes, inspect `~/.config/codex-mmo/credentials.env` and the selected profile's route bindings. Do not place credentials inside profile packs.
|
||||
|
||||
OpenCode Zen and Go use exactly `OPENCODE_API_KEY`; obsolete OpenCode credential names are not recognized or migrated. OpenRouter inference requires `OPENROUTER_API_KEY`, although its catalog-discovery endpoint can currently be queried publicly. Bundled GLM-5.3 roles use exactly `ZAI_CODING_API_KEY`; `ZAI_API_KEY` is a separate General API credential and never a fallback.
|
||||
|
||||
For Tool MCP authentication, run:
|
||||
|
||||
```bash
|
||||
codex-mmo tool-mcp validate SERVER
|
||||
codex-mmo doctor --profile PROFILE
|
||||
```
|
||||
|
||||
The report names missing environment variables but never prints their values. Put the values in `~/.config/codex-mmo/credentials.env` or the launch environment. Tool MCP schema version 8 deliberately does not inherit Codex's `.credentials.json` OAuth sessions.
|
||||
|
||||
## A Tool MCP server is missing or fails to start
|
||||
|
||||
First separate Tool MCP from the internal Agent MCP server. `mmo_mesh` launches profile participants; operator-defined servers provide third-party tools.
|
||||
|
||||
```bash
|
||||
codex-mmo tool-mcp show SERVER
|
||||
codex-mmo tool-mcp validate SERVER
|
||||
codex-mmo profile show PROFILE --resolved
|
||||
codex-mmo doctor --profile PROFILE
|
||||
```
|
||||
|
||||
For STDIO, verify the resolved executable and configured working directory. For Streamable HTTP, verify the URL and environment-backed authentication. `validate` is intentionally non-launching, so a successful readiness result does not prove protocol compatibility or downstream authorization. Start a new session after changing a registry definition because existing sessions remain pinned to their compiled snapshot.
|
||||
|
||||
Native roles share their containing Codex process. The generated root config contains the union of servers required by reachable native roles, while each role layer disables servers and tools it was not granted. This preserves tool visibility but not credential isolation between in-process native roles; use an Agent-MCP participant when that distinction matters.
|
||||
|
||||
## Switchyard does not start
|
||||
|
||||
Check whether the selected profile actually requires it:
|
||||
|
||||
```bash
|
||||
codex-mmo gateway status PROFILE
|
||||
codex-mmo gateway logs PROFILE
|
||||
codex-mmo doctor --profile PROFILE --live
|
||||
```
|
||||
|
||||
Built-in-only profiles do not start Switchyard. For translated routes, verify the Switchyard binary, generated route configuration, endpoint reachability, and route credential.
|
||||
|
||||
## A spawn is rejected for capacity or scope
|
||||
|
||||
Inspect resource and scope admission:
|
||||
|
||||
```bash
|
||||
codex-mmo session show SESSION_ID
|
||||
codex-mmo jobs list --session SESSION_ID
|
||||
codex-mmo jobs status JOB_ID
|
||||
```
|
||||
|
||||
Admission rejects a batch immediately when route capacity, a single-slot local GPU group, the per-role active limit, the session active-agent limit, or a writable scope is unavailable; it does not maintain a capacity queue. A terminal worker releases these leases, so sequential follow-up work does not exhaust a lifetime quota. A `queued` job is only in the brief accepted-before-runner-start transition. Reduce concurrency, wait for the conflicting lease to finish, cancel obsolete work, or divide write ownership into disjoint scopes. If an accepted job remains `queued`, inspect its runner log and reconcile its recorded runner PID.
|
||||
|
||||
## A worker cannot spawn another agent
|
||||
|
||||
Nested delegation is allowed only when the profile grants the edge. Check the caller's `can_spawn` list, maximum depth, active child limit, ancestor-role policy, active resource capacity, and backend availability. Leaf and low-trust agents intentionally receive no delegation authority.
|
||||
|
||||
## A worker result is rejected
|
||||
|
||||
MCP workers may be required to satisfy a JSON output contract. Inspect:
|
||||
|
||||
```bash
|
||||
codex-mmo jobs result JOB_ID
|
||||
codex-mmo jobs result JOB_ID --cursor NEXT_CURSOR
|
||||
codex-mmo jobs status JOB_ID
|
||||
```
|
||||
|
||||
A failed strict result normally indicates invalid JSON, a missing/unexpected field, an invalid URI/date, command evidence absent from captured events, an artifact hash/path mismatch, or a role-specific literal-evidence mismatch. `agents_wait` returns compact state and progress revisions; terminal previews appear only when explicitly requested. Start `agent_result` or `jobs result` at cursor zero and follow each `next_cursor` until null; text pages are contiguous and lossless. When `agent_trace` reports a truncated valid event, use its `record_cursor` with `agent_trace_record` and follow `next_cursor` to reconstruct the exact filtered record. Malformed trace records expose only a bounded diagnostic because they cannot be structurally filtered. Do not open MMO's job result, event, stderr, or socket paths directly from an agent—use the result, inspect, and trace APIs. A result that validated may still be explicitly rejected by the root after review. Reading a result does not accept it, and accepting a writer result does not apply its patch until `agent_patch_integrate` succeeds.
|
||||
|
||||
Strict Agent-MCP roles use app-server `turn/start.outputSchema` when the selected model supports
|
||||
structured output, then validate the complete original contract independently. A malformed first
|
||||
response receives one shape-only repair turn on the same durable thread. There is no JSON-fixer
|
||||
agent and no repair is allowed to invent absent evidence.
|
||||
|
||||
## A long-running root or worker is quiet, paused, or suspended
|
||||
|
||||
Do not ask the model to estimate elapsed time. Inspect the host-owned state and filtered trace:
|
||||
|
||||
```bash
|
||||
codex-mmo jobs status JOB_ID
|
||||
codex-mmo jobs result JOB_ID
|
||||
codex-mmo session show SESSION_ID
|
||||
```
|
||||
|
||||
`stall_warning_seconds` is diagnostic only. It never interrupts a quiet model. Turn-mode roles have
|
||||
no profile task deadline; goal-mode roles use Codex's cumulative token accounting and remain
|
||||
recoverable when the current budget is exhausted. An authorized controller can inspect/trace,
|
||||
pause, continue with an optional larger total budget within the compiled ceiling, request
|
||||
finalization, detach, or fully stop. A worker client/transport failure reconnects to its exact
|
||||
host/thread; root recovery uses the current canonical thread generation. Unrecovered host loss
|
||||
produces `suspended`, not a missing result. A replacement turn-mode worker first settles any
|
||||
orphaned active turn; it starts one continuation only after an interrupted outcome and preserves a
|
||||
result that completed during the race. `stopped`, `failed`, and
|
||||
`cancelled` remain terminal but retain readable partial evidence, terminal history, and patches.
|
||||
|
||||
Do not try to fix a slow model by adding a model-authored “work for N seconds” instruction. The
|
||||
provider `stream_idle_timeout_ms`, app-server initialization/lifecycle request limits, Tool MCP
|
||||
call limits, goal token ceilings, and external smoke/evaluation wall limits are different
|
||||
boundaries. Direct provider idle values must be positive and should be sized for the route's real
|
||||
latency rather than a model instruction. App-server lifecycle waits cover slow MCP startup and
|
||||
thread operations but do not time model turns. Tool calls may retain their own safety bounds; the
|
||||
surrounding thread and evidence survive client detach, transport restart, or interruption.
|
||||
|
||||
For a noninteractive root, `exec --wall-timeout` is only a caller/harness boundary. It detaches
|
||||
without interrupting the app-server goal; use `codex-mmo resume SESSION_ID` to attach to the same
|
||||
work. `session detach` also leaves root and workers running. `session pause` checkpoints partial
|
||||
evidence, interrupts admitted turns, retires root and supervised-worker hosts, and releases their
|
||||
capacity. `session continue` freshly admits and reactivates the same root thread plus the exact
|
||||
paused worker set. Run `session compact` only from that cold-paused state; it temporarily hosts the
|
||||
root, compacts the same thread, and cold-pauses it again. During bootstrap use
|
||||
`session stop` or `session cancel` when no durable thread exists yet. `session stop --grace SECONDS`
|
||||
requests finalization and then retires all hosts; `session cancel` is immediate.
|
||||
|
||||
## A worker is waiting for input or approval
|
||||
|
||||
`approval_policy = "never"` declines escalation automatically inside the role's compiled
|
||||
sandbox. With `untrusted` or `on-request`, an authorized controller sees the raw pending method and
|
||||
request ID through `agent_inspect` and must send that method's exact response object with
|
||||
`agent_respond`. The request and thread remain durable while pending. Do not send a generic
|
||||
approval shape: command, file-change, permission, user-input, and MCP-elicitation methods have
|
||||
different generated app-server response schemas.
|
||||
|
||||
If a mutating control returns a transport error after delivery began, inspect the job before
|
||||
retrying. MMO records that revision as `delivery_unknown`, because the worker may have applied the
|
||||
request before its response was lost. A blind retry could steer, interrupt, finalize, or fork the
|
||||
thread twice. Controls are serialized per job, but serialization cannot turn an ambiguous network
|
||||
outcome into proof of non-application.
|
||||
|
||||
## A job changed files outside its scope
|
||||
|
||||
Writable MCP jobs run in isolated Git worktrees, so they do not edit the canonical checkout directly. An out-of-scope or unsafe patch boundary is a hard job failure that taints the session; inspect the job metadata and isolated artifact. Do not integrate it. Narrow the assignment or use a read-only reviewer, then launch a fresh bounded job if still justified.
|
||||
|
||||
## Cancellation appears incomplete
|
||||
|
||||
Cancel the root session to cascade through descendants:
|
||||
|
||||
```bash
|
||||
codex-mmo session cancel SESSION_ID
|
||||
```
|
||||
|
||||
For one branch:
|
||||
|
||||
```bash
|
||||
codex-mmo jobs cancel JOB_ID --reason "superseded"
|
||||
```
|
||||
|
||||
The supervisor terminates worker process groups and records terminal state. If a process remains, inspect the job stderr and verify that the process was launched by the current installed runtime rather than an unrelated manual command.
|
||||
|
||||
## Native agents and MCP agents disagree
|
||||
|
||||
Do not vote or average their conclusions. Resolve the conflict from primary evidence: source code, commands, logs, tests, specifications, or a reproducible external observation. Use a read-only reviewer when an independent check is useful.
|
||||
|
||||
## Release archives look incomplete
|
||||
|
||||
Verify checksums and member counts before installation:
|
||||
|
||||
```bash
|
||||
sha256sum -c codex-multimodel-orchestrator-8.0.0-SHA256SUMS.txt
|
||||
tar -tzf codex-multimodel-orchestrator-8.0.0-linux.tar.gz | head
|
||||
unzip -Z1 codex-multimodel-orchestrator-8.0.0-linux.zip | head
|
||||
```
|
||||
|
||||
Each archive must contain the top-level package directory, runtime modules, profiles, configuration, documentation, tests, installer, and `PACKAGE-MANIFEST.json`. The release builder rejects a truncated source tree and verifies member hashes before reporting success.
|
||||
|
||||
## Collecting diagnostics
|
||||
|
||||
Preserve the following when investigating a failure:
|
||||
|
||||
- The profile ID and snapshot hash.
|
||||
- `codex-mmo doctor` output.
|
||||
- Session and job IDs.
|
||||
- Job stderr and event logs.
|
||||
- Gateway logs for translated providers.
|
||||
- Route, serving-provider, HTTP status, policy, and retry information.
|
||||
- The exact repository commit and dirty state.
|
||||
- The installed Codex and Switchyard versions.
|
||||
|
||||
Remove credentials before sharing diagnostics.
|
||||
@@ -0,0 +1,4 @@
|
||||
# Cross-replica payment constraints
|
||||
|
||||
Requests have globally unique idempotency keys; six stateless replicas can receive retries; SQL supports unique constraints and transactions; the external payment cannot join the SQL transaction; the design must recover from crashes between durable state and the side effect. Replica-local memory is neither shared nor durable.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# Label normalization
|
||||
|
||||
`slug_label` trims surrounding whitespace, lowercases ASCII text, collapses every run of spaces or underscores to one hyphen, and removes leading or trailing hyphens.
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
max_batch_items = 128
|
||||
retry_limit = 4
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
import re
|
||||
|
||||
|
||||
def slug_label(value: str) -> str:
|
||||
return re.sub(r" +", "-", value.strip().lower())
|
||||
@@ -0,0 +1,15 @@
|
||||
import unittest
|
||||
|
||||
from labels import slug_label
|
||||
|
||||
|
||||
class LabelTests(unittest.TestCase):
|
||||
def test_mixed_separators(self):
|
||||
self.assertEqual(slug_label(" Release__ Candidate "), "release-candidate")
|
||||
|
||||
def test_outer_separators(self):
|
||||
self.assertEqual(slug_label("__Ready__"), "ready")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,23 @@
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
mode = sys.argv[1]
|
||||
text = Path(".mmo-eval-final.txt").read_text(encoding="utf-8").lower()
|
||||
holdout_path = Path("access-holdout.json")
|
||||
holdout = json.loads(holdout_path.read_text(encoding="utf-8")) if holdout_path.is_file() else {}
|
||||
if mode == "literal" and not (
|
||||
holdout.get("expected_value", "128").lower() in text
|
||||
and holdout.get("expected_path", "LIMITS.toml").lower() in text
|
||||
):
|
||||
raise SystemExit("literal result is not grounded in the hidden expected location")
|
||||
if mode == "routine":
|
||||
from labels import slug_label
|
||||
|
||||
if slug_label(" Release__ Candidate ") != "release-candidate":
|
||||
raise SystemExit("routine slug contract still fails")
|
||||
if mode == "decision":
|
||||
required = holdout.get("required_terms", ["unique", "durable", "crash", "external", "local"])
|
||||
missing = [term for term in required if term not in text]
|
||||
if missing:
|
||||
raise SystemExit("decision lacks constraints: " + ", ".join(missing))
|
||||
@@ -0,0 +1 @@
|
||||
{"required_terms":["unique","durable","crash","external","local"]}
|
||||
@@ -0,0 +1 @@
|
||||
{"expected_path":"LIMITS.toml","expected_value":"128"}
|
||||
@@ -0,0 +1,8 @@
|
||||
import unittest
|
||||
|
||||
from labels import slug_label
|
||||
|
||||
|
||||
class HoldoutLabelTests(unittest.TestCase):
|
||||
def test_long_mixed_separator_run(self):
|
||||
self.assertEqual(slug_label("___Alpha _ Beta___"), "alpha-beta")
|
||||
@@ -0,0 +1,175 @@
|
||||
schema_version = 8
|
||||
id = "access-efficient"
|
||||
profile = "access-efficient-escalation-lab"
|
||||
name = "Access-Efficient Escalation Lab"
|
||||
description = "Experimental matched routing trials for literal local work, economical patches, and deliberate scarce-tier escalation."
|
||||
fixture = "fixture"
|
||||
development_trials = 3
|
||||
release_trials = 5
|
||||
|
||||
[promotion]
|
||||
primary_metric = "scarce_tier_request_units"
|
||||
primary_baseline = "strongest_single_agent"
|
||||
direction = "lower"
|
||||
strongest_success_tolerance = 0.02
|
||||
minimum_relative_improvement = 0.50
|
||||
minimum_absolute_improvement = 1.0
|
||||
worker_minimum_success_contribution = 0.02
|
||||
worker_minimum_metric_contribution = 0.10
|
||||
no_regression_higher_metrics = []
|
||||
no_regression_lower_metrics = []
|
||||
require_complete_api_cost = true
|
||||
scarce_model_keys = [
|
||||
"codex_chatgpt_builtin__gpt_5_6_sol",
|
||||
"opencode_zen_anthropic_messages__claude_opus_5",
|
||||
"zai_coding_responses__glm_5_3",
|
||||
]
|
||||
|
||||
[[variants]]
|
||||
id = "configured-root"
|
||||
purpose = "Luna routing root alone."
|
||||
topology = "root_only"
|
||||
comparison_class = "configured_root_alone"
|
||||
|
||||
[[variants]]
|
||||
id = "strongest-task-single"
|
||||
purpose = "Sol maximum-capability single-agent control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "strongest_single_agent"
|
||||
|
||||
[[variants]]
|
||||
id = "codex-access-single"
|
||||
purpose = "ChatGPT Codex access control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "chatgpt_codex"
|
||||
|
||||
[[variants]]
|
||||
id = "go-access-single"
|
||||
purpose = "OpenCode Go economical access control."
|
||||
profile = "high-confidence-debugging"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_go"
|
||||
|
||||
[[variants]]
|
||||
id = "zen-access-single"
|
||||
purpose = "OpenCode Zen access control."
|
||||
profile = "secure-change"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_zen"
|
||||
|
||||
[[variants]]
|
||||
id = "zai-access-single"
|
||||
purpose = "Z.AI Coding Plan scarce-tier control."
|
||||
profile = "incident-hypothesis-triage"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "zai_coding_plan"
|
||||
|
||||
[[variants]]
|
||||
id = "openrouter-access-single"
|
||||
purpose = "Pinned OpenRouter API control."
|
||||
profile = "route-resilience-lab"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "openrouter_api"
|
||||
[variants.bindings]
|
||||
route_observer = "openrouter_openai_chat__deepseek_deepseek_v4_pro"
|
||||
|
||||
[[variants]]
|
||||
id = "root-plus-routine"
|
||||
purpose = "Luna root plus the economical isolated routine engineer."
|
||||
topology = "root_plus_worker"
|
||||
worker = "routine_engineer"
|
||||
comparison_class = "root_plus_highest_value"
|
||||
|
||||
[[variants]]
|
||||
id = "full-profile"
|
||||
purpose = "Contained literal scout, routine engineer, and trigger-gated GLM escalation."
|
||||
topology = "full"
|
||||
comparison_class = "full_profile"
|
||||
|
||||
[[variants]]
|
||||
id = "without-literal-scout"
|
||||
purpose = "Tests whether local literal reconnaissance earns its launch overhead."
|
||||
topology = "full_without_worker"
|
||||
worker = "literal_scout"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-routine-engineer"
|
||||
purpose = "Ablates the economical isolated patch lane."
|
||||
topology = "full_without_worker"
|
||||
worker = "routine_engineer"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-escalation"
|
||||
purpose = "Ablates scarce-tier GLM escalation."
|
||||
topology = "full_without_worker"
|
||||
worker = "flagship_escalation"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[tasks]]
|
||||
id = "literal-limit"
|
||||
description = "Trivial literal negative control where delegation should be avoided or strictly literal."
|
||||
sandbox = "read-only"
|
||||
difficulty = "easy"
|
||||
negative_control = true
|
||||
wall_timeout_seconds = 900
|
||||
validation_timeout_seconds = 60
|
||||
prompt = """
|
||||
Report max_batch_items from LIMITS.toml with the exact path. Do no semantic reasoning and do not escalate. If the local scout is used, submit only a structured literal extract request. End with EVAL_ACCESS_LITERAL.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["128", "EVAL_ACCESS_LITERAL"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python validate_access.py literal"]
|
||||
[tasks.orchestration_assertions]
|
||||
forbidden_agents = ["flagship_escalation"]
|
||||
max_jobs = 1
|
||||
|
||||
[[tasks]]
|
||||
id = "routine-slug"
|
||||
description = "Localized deterministic patch suitable for economical execution."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "medium"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 1800
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Implement LABEL_SPEC.md. This is a bounded objectively testable source change; use scarce-tier escalation only if concrete evidence exceeds routine capability. End with EVAL_ACCESS_ROUTINE.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_ACCESS_ROUTINE"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v && python validate_access.py routine"]
|
||||
[tasks.orchestration_assertions]
|
||||
required_agents = ["routine_engineer"]
|
||||
forbidden_agents = ["flagship_escalation"]
|
||||
max_jobs = 2
|
||||
max_contract_failures = 0
|
||||
|
||||
[[tasks]]
|
||||
id = "ambiguous-distributed-decision"
|
||||
description = "Consequential ambiguity with an explicit escalation trigger."
|
||||
sandbox = "read-only"
|
||||
difficulty = "hard"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 1800
|
||||
validation_timeout_seconds = 60
|
||||
prompt = """
|
||||
Read DISTRIBUTED_CONSTRAINTS.md and select an idempotency design. Record why the uncertainty and consequence justify or do not justify GLM escalation. Resolve claims with the supplied constraints, not model rank. End with EVAL_ACCESS_ESCALATION.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_ACCESS_ESCALATION"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python validate_access.py decision"]
|
||||
[tasks.orchestration_assertions]
|
||||
required_agents = ["flagship_escalation"]
|
||||
max_jobs = 1
|
||||
max_contract_failures = 0
|
||||
@@ -0,0 +1,5 @@
|
||||
# Adaptive change evaluation
|
||||
|
||||
This suite exercises the `adaptive-engineering` profile on a bounded but nontrivial correctness change. The root must keep ownership of diagnosis and integration, use a specialist only where it adds value, and obtain independent review before claiming success.
|
||||
|
||||
The native scout is intentionally not a worker assertion: native Codex participation is not represented in Agent MCP job telemetry. The evaluator does require the MCP implementation and review roles and measures whether their results were consumed.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Ledger invariants
|
||||
|
||||
`Ledger.apply(operation_id, account, delta)` applies an operation at most once globally.
|
||||
|
||||
- A successful operation permanently consumes its operation ID.
|
||||
- A rejected overdraft must not consume its operation ID.
|
||||
- A caller may fund the account and retry the same rejected operation.
|
||||
- Balances may never become negative.
|
||||
- Duplicate successful operations return the current balance without applying the delta again.
|
||||
|
||||
The current implementation violates the rejected-operation rule.
|
||||
@@ -0,0 +1,17 @@
|
||||
class Ledger:
|
||||
def __init__(self) -> None:
|
||||
self._balances: dict[str, int] = {}
|
||||
self._applied: set[str] = set()
|
||||
|
||||
def balance(self, account: str) -> int:
|
||||
return self._balances.get(account, 0)
|
||||
|
||||
def apply(self, operation_id: str, account: str, delta: int) -> int:
|
||||
if operation_id in self._applied:
|
||||
return self.balance(account)
|
||||
self._applied.add(operation_id)
|
||||
updated = self.balance(account) + delta
|
||||
if updated < 0:
|
||||
raise ValueError("insufficient funds")
|
||||
self._balances[account] = updated
|
||||
return updated
|
||||
@@ -0,0 +1,27 @@
|
||||
import unittest
|
||||
|
||||
from ledger import Ledger
|
||||
|
||||
|
||||
class LedgerTests(unittest.TestCase):
|
||||
def test_success_is_idempotent(self):
|
||||
ledger = Ledger()
|
||||
self.assertEqual(ledger.apply("deposit-1", "a", 10), 10)
|
||||
self.assertEqual(ledger.apply("deposit-1", "a", 10), 10)
|
||||
|
||||
def test_overdraft_does_not_change_balance(self):
|
||||
ledger = Ledger()
|
||||
with self.assertRaises(ValueError):
|
||||
ledger.apply("withdraw-1", "a", -4)
|
||||
self.assertEqual(ledger.balance("a"), 0)
|
||||
|
||||
def test_rejected_operation_can_be_retried(self):
|
||||
ledger = Ledger()
|
||||
with self.assertRaises(ValueError):
|
||||
ledger.apply("withdraw-1", "a", -4)
|
||||
ledger.apply("deposit-1", "a", 10)
|
||||
self.assertEqual(ledger.apply("withdraw-1", "a", -4), 6)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,16 @@
|
||||
import unittest
|
||||
|
||||
from ledger import Ledger
|
||||
|
||||
|
||||
class HoldoutLedgerTests(unittest.TestCase):
|
||||
def test_rejected_id_remains_globally_available(self):
|
||||
ledger = Ledger()
|
||||
with self.assertRaises(ValueError):
|
||||
ledger.apply("shared", "a", -1)
|
||||
self.assertEqual(ledger.apply("shared", "b", 3), 3)
|
||||
|
||||
def test_successful_id_is_global_across_accounts(self):
|
||||
ledger = Ledger()
|
||||
ledger.apply("shared", "a", 3)
|
||||
self.assertEqual(ledger.apply("shared", "b", 9), 0)
|
||||
@@ -0,0 +1,13 @@
|
||||
import unittest
|
||||
|
||||
from ledger import Ledger
|
||||
|
||||
|
||||
class HoldoutLedgerTests(unittest.TestCase):
|
||||
def test_multiple_rejections_do_not_consume_id(self):
|
||||
ledger = Ledger()
|
||||
for _ in range(2):
|
||||
with self.assertRaises(ValueError):
|
||||
ledger.apply("withdraw", "a", -2)
|
||||
ledger.apply("fund", "a", 4)
|
||||
self.assertEqual(ledger.apply("withdraw", "a", -2), 2)
|
||||
@@ -0,0 +1,149 @@
|
||||
schema_version = 8
|
||||
id = "adaptive-change"
|
||||
profile = "adaptive-engineering"
|
||||
name = "Adaptive Engineering Change"
|
||||
description = "Matched trials for selective delegation on decomposable and tightly coupled engineering work."
|
||||
fixture = "fixture"
|
||||
development_trials = 3
|
||||
release_trials = 5
|
||||
|
||||
[promotion]
|
||||
primary_metric = "success_rate"
|
||||
direction = "higher"
|
||||
strongest_success_tolerance = 0.02
|
||||
minimum_relative_improvement = 0.10
|
||||
minimum_absolute_improvement = 0.05
|
||||
worker_minimum_success_contribution = 0.02
|
||||
worker_minimum_metric_contribution = 0.10
|
||||
no_regression_higher_metrics = []
|
||||
no_regression_lower_metrics = []
|
||||
require_complete_api_cost = true
|
||||
|
||||
[[variants]]
|
||||
id = "configured-root"
|
||||
purpose = "Configured Sol root with delegation mechanically disabled."
|
||||
topology = "root_only"
|
||||
comparison_class = "configured_root_alone"
|
||||
|
||||
[[variants]]
|
||||
id = "strongest-task-single"
|
||||
purpose = "Task-specific flagship Sol single-agent control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "strongest_single_agent"
|
||||
|
||||
[[variants]]
|
||||
id = "codex-access-single"
|
||||
purpose = "ChatGPT Codex access-service control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "chatgpt_codex"
|
||||
|
||||
[[variants]]
|
||||
id = "go-access-single"
|
||||
purpose = "OpenCode Go economical DeepSeek control."
|
||||
profile = "high-confidence-debugging"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_go"
|
||||
|
||||
[[variants]]
|
||||
id = "zen-access-single"
|
||||
purpose = "OpenCode Zen Claude control."
|
||||
profile = "secure-change"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_zen"
|
||||
|
||||
[[variants]]
|
||||
id = "zai-access-single"
|
||||
purpose = "Z.AI Coding Plan GLM control."
|
||||
profile = "incident-hypothesis-triage"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "zai_coding_plan"
|
||||
|
||||
[[variants]]
|
||||
id = "openrouter-access-single"
|
||||
purpose = "Pinned OpenRouter DeepSeek route control."
|
||||
profile = "route-resilience-lab"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "openrouter_api"
|
||||
[variants.bindings]
|
||||
route_observer = "openrouter_openai_chat__deepseek_deepseek_v4_pro"
|
||||
|
||||
[[variants]]
|
||||
id = "root-plus-implementation"
|
||||
purpose = "Configured root plus the objectively testable implementation branch."
|
||||
topology = "root_plus_worker"
|
||||
worker = "implementation_specialist"
|
||||
comparison_class = "root_plus_highest_value"
|
||||
|
||||
[[variants]]
|
||||
id = "full-profile"
|
||||
purpose = "Complete selective adaptive topology."
|
||||
topology = "full"
|
||||
comparison_class = "full_profile"
|
||||
|
||||
[[variants]]
|
||||
id = "without-scout"
|
||||
purpose = "Ablates native reconnaissance."
|
||||
topology = "full_without_worker"
|
||||
worker = "repo_scout"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-implementation"
|
||||
purpose = "Ablates the economical isolated implementation branch."
|
||||
topology = "full_without_worker"
|
||||
worker = "implementation_specialist"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-review"
|
||||
purpose = "Ablates independent cross-maker review."
|
||||
topology = "full_without_worker"
|
||||
worker = "adversarial_reviewer"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[tasks]]
|
||||
id = "decomposable-ledger"
|
||||
description = "Repair an idempotency defect with separable reconnaissance, implementation, and review work."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "hard"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 2400
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Repair the ledger defect described in SPEC.md. Own diagnosis and canonical integration. Delegate only work with a concrete expected contribution, continue non-overlapping critical-path work while it runs, explicitly accept or reject results, and end with EVAL_ADAPTIVE_VERIFIED.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_ADAPTIVE_VERIFIED"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v"]
|
||||
[tasks.orchestration_assertions]
|
||||
required_agents = ["implementation_specialist", "adversarial_reviewer"]
|
||||
max_jobs = 3
|
||||
max_contract_failures = 0
|
||||
max_observed_mcp_wait_ratio = 0.8
|
||||
|
||||
[[tasks]]
|
||||
id = "tight-small-change"
|
||||
description = "Negative control where worker launch overhead should not be required."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "easy"
|
||||
negative_control = true
|
||||
wall_timeout_seconds = 1200
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Make the smallest correct ledger repair. This task is intentionally compact: remain single-agent unless independent work has a concrete positive expected value. Run the tests and end with EVAL_ADAPTIVE_SMALL.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_ADAPTIVE_SMALL"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v"]
|
||||
[tasks.orchestration_assertions]
|
||||
max_jobs = 1
|
||||
max_contract_failures = 0
|
||||
@@ -0,0 +1,4 @@
|
||||
# Policy precedence
|
||||
|
||||
Signed exceptions override the current handbook. The current handbook overrides archived drafts.
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Current handbook
|
||||
|
||||
## Retries
|
||||
|
||||
At most four delivery attempts are permitted.
|
||||
|
||||
## Retention
|
||||
|
||||
Operational delivery records are retained for 30 days.
|
||||
|
||||
## Rollout
|
||||
|
||||
Canary rollout is mandatory.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# Archived draft
|
||||
|
||||
This obsolete draft proposed seven retries and 90-day retention.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# Signed exception: emergency rollout
|
||||
|
||||
For incident INC-42 only, canary rollout may be skipped after two-person approval. Retry and retention rules are unchanged.
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
text = Path(".mmo-eval-final.txt").read_text(encoding="utf-8").lower()
|
||||
holdout_path = Path("research-holdout.json")
|
||||
if not holdout_path.is_file():
|
||||
raise SystemExit("post-run research holdout is required")
|
||||
holdout = json.loads(holdout_path.read_text(encoding="utf-8"))
|
||||
required = [*holdout["required_values"], *holdout["required_sources"]]
|
||||
missing = [term for term in required if term not in text]
|
||||
if missing:
|
||||
raise SystemExit("research synthesis misses authoritative facts: " + ", ".join(missing))
|
||||
if "seven" in text and not any(term in text for term in ("obsolete", "archived", "reject")):
|
||||
raise SystemExit("archived retry value was not rejected")
|
||||
print(
|
||||
json.dumps(
|
||||
{"metrics": {"source_coverage": 1.0, "contradiction_resolution": 1.0}},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1 @@
|
||||
{"required_values":["four","30","inc-42"],"required_sources":["10-handbook.md","30-signed-exception.md"]}
|
||||
@@ -0,0 +1 @@
|
||||
{"required_values":["four","30","inc-42"],"required_sources":["10-handbook.md","30-signed-exception.md"]}
|
||||
@@ -0,0 +1,140 @@
|
||||
schema_version = 8
|
||||
id = "bounded-research"
|
||||
profile = "bounded-research-organization-lab"
|
||||
name = "Bounded Research Organization Lab"
|
||||
description = "Experimental flat-versus-hierarchical long-corpus research with bounded nesting and handoff accounting."
|
||||
fixture = "fixture"
|
||||
development_trials = 3
|
||||
release_trials = 5
|
||||
|
||||
[promotion]
|
||||
primary_metric = "source_coverage"
|
||||
direction = "higher"
|
||||
strongest_success_tolerance = 0.02
|
||||
minimum_relative_improvement = 0.10
|
||||
minimum_absolute_improvement = 0.05
|
||||
worker_minimum_success_contribution = 0.02
|
||||
worker_minimum_metric_contribution = 0.10
|
||||
no_regression_higher_metrics = ["contradiction_resolution"]
|
||||
no_regression_lower_metrics = []
|
||||
require_complete_api_cost = true
|
||||
|
||||
[[variants]]
|
||||
id = "configured-root"
|
||||
purpose = "Sol repository root alone."
|
||||
topology = "root_only"
|
||||
comparison_class = "configured_root_alone"
|
||||
|
||||
[[variants]]
|
||||
id = "strongest-task-single"
|
||||
purpose = "Sol long-corpus single-agent control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "strongest_single_agent"
|
||||
|
||||
[[variants]]
|
||||
id = "codex-access-single"
|
||||
purpose = "ChatGPT Codex access control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "chatgpt_codex"
|
||||
|
||||
[[variants]]
|
||||
id = "go-access-single"
|
||||
purpose = "OpenCode Go Kimi/MiniMax access control."
|
||||
profile = "research-backed-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_go"
|
||||
|
||||
[[variants]]
|
||||
id = "zen-access-single"
|
||||
purpose = "OpenCode Zen access control."
|
||||
profile = "secure-change"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_zen"
|
||||
|
||||
[[variants]]
|
||||
id = "zai-access-single"
|
||||
purpose = "Z.AI Coding Plan access control."
|
||||
profile = "incident-hypothesis-triage"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "zai_coding_plan"
|
||||
|
||||
[[variants]]
|
||||
id = "openrouter-access-single"
|
||||
purpose = "Pinned OpenRouter access control."
|
||||
profile = "route-resilience-lab"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "openrouter_api"
|
||||
[variants.bindings]
|
||||
route_observer = "openrouter_openai_chat__deepseek_deepseek_v4_pro"
|
||||
|
||||
[[variants]]
|
||||
id = "flat-fanout"
|
||||
purpose = "Root plus a direct MiniMax source scout, with hierarchy disabled."
|
||||
topology = "root_plus_worker"
|
||||
worker = "source_scout"
|
||||
comparison_class = "root_plus_highest_value"
|
||||
|
||||
[[variants]]
|
||||
id = "full-hierarchy"
|
||||
purpose = "Kimi research lead may organize at most two MiniMax scouts."
|
||||
topology = "full"
|
||||
comparison_class = "full_profile"
|
||||
|
||||
[[variants]]
|
||||
id = "without-research-lead"
|
||||
purpose = "Ablates the hierarchical Kimi organizer, yielding direct flat scouting only."
|
||||
topology = "full_without_worker"
|
||||
worker = "research_lead"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-source-scout"
|
||||
purpose = "Ablates MiniMax source extraction while retaining the Kimi lead."
|
||||
topology = "full_without_worker"
|
||||
worker = "source_scout"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[tasks]]
|
||||
id = "distributed-policy-corpus"
|
||||
description = "Resolve cross-document conflicts and trace the final policy to authoritative sections."
|
||||
sandbox = "read-only"
|
||||
difficulty = "hard"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 2400
|
||||
validation_timeout_seconds = 60
|
||||
prompt = """
|
||||
Read every document under corpus/. Determine the authoritative retry ceiling, retention period, and rollout exception, cite exact paths and headings, and resolve contradictions using the stated precedence rule. Compare bounded hierarchy with direct scouting without duplicate work. End with EVAL_BOUNDED_RESEARCH.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_BOUNDED_RESEARCH"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python validate_research.py"]
|
||||
[tasks.orchestration_assertions]
|
||||
required_agents = ["research_lead", "source_scout"]
|
||||
max_jobs = 3
|
||||
max_contract_failures = 0
|
||||
|
||||
[[tasks]]
|
||||
id = "small-corpus-negative"
|
||||
description = "Negative control where hierarchical handoffs should not be assumed valuable."
|
||||
sandbox = "read-only"
|
||||
difficulty = "easy"
|
||||
negative_control = true
|
||||
wall_timeout_seconds = 1200
|
||||
validation_timeout_seconds = 60
|
||||
prompt = """
|
||||
Answer the three literal policy questions from corpus/ with exact citations. The corpus is intentionally small enough that nesting may cost more than it adds; launch only with a concrete expected contribution. End with EVAL_BOUNDED_SMALL.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_BOUNDED_SMALL"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python validate_research.py"]
|
||||
[tasks.orchestration_assertions]
|
||||
max_jobs = 2
|
||||
@@ -0,0 +1,4 @@
|
||||
# Deep merge contract
|
||||
|
||||
`merge_settings(base, overlay)` returns a fresh mapping. Nested mappings merge recursively; overlay scalars replace base values; no input or nested output may be mutated or aliased.
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
|
||||
def merge_settings(base: Mapping[str, Any], overlay: Mapping[str, Any]) -> dict[str, Any]:
|
||||
result = dict(base)
|
||||
for key, value in overlay.items():
|
||||
if isinstance(value, Mapping) and isinstance(result.get(key), Mapping):
|
||||
result[key].update(value)
|
||||
else:
|
||||
result[key] = value
|
||||
return result
|
||||
@@ -0,0 +1,23 @@
|
||||
import unittest
|
||||
|
||||
from settings import merge_settings
|
||||
|
||||
|
||||
class MergeSettingsTests(unittest.TestCase):
|
||||
def test_recursive_merge(self):
|
||||
self.assertEqual(
|
||||
merge_settings({"s": {"host": "x", "port": 80}}, {"s": {"port": 443}}),
|
||||
{"s": {"host": "x", "port": 443}},
|
||||
)
|
||||
|
||||
def test_inputs_and_outputs_do_not_alias(self):
|
||||
base = {"s": {"host": "x"}, "flags": {"safe": True}}
|
||||
overlay = {"s": {"port": 443}}
|
||||
merged = merge_settings(base, overlay)
|
||||
merged["s"]["host"] = "changed"
|
||||
merged["flags"]["safe"] = False
|
||||
self.assertEqual(base, {"s": {"host": "x"}, "flags": {"safe": True}})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,13 @@
|
||||
import unittest
|
||||
|
||||
from settings import merge_settings
|
||||
|
||||
|
||||
class HoldoutMergeTests(unittest.TestCase):
|
||||
def test_three_level_merge_and_no_alias(self):
|
||||
base = {"a": {"b": {"left": 1}}, "untouched": {"x": 1}}
|
||||
overlay = {"a": {"b": {"right": 2}}}
|
||||
merged = merge_settings(base, overlay)
|
||||
self.assertEqual(merged["a"]["b"], {"left": 1, "right": 2})
|
||||
merged["untouched"]["x"] = 9
|
||||
self.assertEqual(base["untouched"]["x"], 1)
|
||||
@@ -0,0 +1,8 @@
|
||||
import unittest
|
||||
|
||||
from settings import merge_settings
|
||||
|
||||
|
||||
class HoldoutMergeTests(unittest.TestCase):
|
||||
def test_overlay_scalar_replaces_mapping(self):
|
||||
self.assertEqual(merge_settings({"a": {"x": 1}}, {"a": 3}), {"a": 3})
|
||||
@@ -0,0 +1,156 @@
|
||||
schema_version = 8
|
||||
id = "codex-harness"
|
||||
profile = "codex-harness-team"
|
||||
name = "Codex Harness Team"
|
||||
description = "Tests native-first context isolation, homogeneous Sol peers, and a fresh supervised critic."
|
||||
fixture = "fixture"
|
||||
development_trials = 3
|
||||
release_trials = 5
|
||||
|
||||
[promotion]
|
||||
primary_metric = "success_rate"
|
||||
direction = "higher"
|
||||
strongest_success_tolerance = 0.02
|
||||
minimum_relative_improvement = 0.10
|
||||
minimum_absolute_improvement = 0.05
|
||||
worker_minimum_success_contribution = 0.02
|
||||
worker_minimum_metric_contribution = 0.10
|
||||
no_regression_higher_metrics = []
|
||||
no_regression_lower_metrics = []
|
||||
require_complete_api_cost = true
|
||||
|
||||
[[variants]]
|
||||
id = "configured-root"
|
||||
purpose = "Sol root alone."
|
||||
topology = "root_only"
|
||||
comparison_class = "configured_root_alone"
|
||||
|
||||
[[variants]]
|
||||
id = "strongest-task-single"
|
||||
purpose = "Independent strongest single-agent Sol control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "strongest_single_agent"
|
||||
|
||||
[[variants]]
|
||||
id = "codex-access-single"
|
||||
purpose = "ChatGPT Codex service control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "chatgpt_codex"
|
||||
|
||||
[[variants]]
|
||||
id = "go-access-single"
|
||||
purpose = "OpenCode Go DeepSeek service control."
|
||||
profile = "high-confidence-debugging"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_go"
|
||||
|
||||
[[variants]]
|
||||
id = "zen-access-single"
|
||||
purpose = "OpenCode Zen Claude service control."
|
||||
profile = "secure-change"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_zen"
|
||||
|
||||
[[variants]]
|
||||
id = "zai-access-single"
|
||||
purpose = "Z.AI Coding Plan GLM service control."
|
||||
profile = "incident-hypothesis-triage"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "zai_coding_plan"
|
||||
|
||||
[[variants]]
|
||||
id = "openrouter-access-single"
|
||||
purpose = "Pinned OpenRouter service control."
|
||||
profile = "route-resilience-lab"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "openrouter_api"
|
||||
[variants.bindings]
|
||||
route_observer = "openrouter_openai_chat__deepseek_deepseek_v4_pro"
|
||||
|
||||
[[variants]]
|
||||
id = "pure-native"
|
||||
purpose = "Root plus the highest-value native invariant designer."
|
||||
topology = "root_plus_worker"
|
||||
worker = "invariant_designer"
|
||||
comparison_class = "root_plus_highest_value"
|
||||
|
||||
[[variants]]
|
||||
id = "homogeneous-sol"
|
||||
purpose = "Full isolated-role control using Sol for every participant."
|
||||
topology = "full"
|
||||
comparison_class = "control"
|
||||
[variants.bindings]
|
||||
repo_scout = "codex_chatgpt_builtin__gpt_5_6_sol"
|
||||
invariant_designer = "codex_chatgpt_builtin__gpt_5_6_sol"
|
||||
|
||||
[[variants]]
|
||||
id = "full-profile"
|
||||
purpose = "Native-first Luna/Terra team followed by a fresh Sol critic."
|
||||
topology = "full"
|
||||
comparison_class = "full_profile"
|
||||
|
||||
[[variants]]
|
||||
id = "without-scout"
|
||||
purpose = "Ablates fast repository reconnaissance."
|
||||
topology = "full_without_worker"
|
||||
worker = "repo_scout"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-invariants"
|
||||
purpose = "Ablates isolated invariant and test design."
|
||||
topology = "full_without_worker"
|
||||
worker = "invariant_designer"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-critic"
|
||||
purpose = "Ablates fresh-context adversarial review."
|
||||
topology = "full_without_worker"
|
||||
worker = "fresh_critic"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[tasks]]
|
||||
id = "deep-merge-contract"
|
||||
description = "Correct a mutation-prone recursive merge under independently derivable invariants."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "hard"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 2400
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Implement the merge contract in SPEC.md. The root owns the change. Launch isolated read-only contexts only when they can derive tests, invariants, or critique concurrently; inspect and adjudicate their evidence. End with EVAL_HARNESS_VERIFIED.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_HARNESS_VERIFIED"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v"]
|
||||
[tasks.orchestration_assertions]
|
||||
required_agents = ["fresh_critic"]
|
||||
max_jobs = 1
|
||||
max_contract_failures = 0
|
||||
|
||||
[[tasks]]
|
||||
id = "small-merge-control"
|
||||
description = "Negative control for context-launch overhead on a compact implementation."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "easy"
|
||||
negative_control = true
|
||||
wall_timeout_seconds = 1200
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Repair merge_settings with the smallest coherent change. Avoid launching peers unless they have a specific expected contribution. Run the suite and end with EVAL_HARNESS_SMALL.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_HARNESS_SMALL"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v"]
|
||||
[tasks.orchestration_assertions]
|
||||
max_jobs = 1
|
||||
@@ -0,0 +1,4 @@
|
||||
# Stable unique contract
|
||||
|
||||
`stable_unique(values, key)` returns the first value for each distinct key, preserves encounter order, accepts unhashable values when the supplied key result is hashable, never mutates input, and must scale linearly for 20,000 values.
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
import json
|
||||
import time
|
||||
|
||||
from dedupe import stable_unique
|
||||
|
||||
values = [index % 5000 for index in range(20_000)]
|
||||
started = time.perf_counter()
|
||||
result = stable_unique(values, lambda value: value)
|
||||
elapsed = time.perf_counter() - started
|
||||
if result != list(range(5000)):
|
||||
raise SystemExit("stable_unique produced an incorrect result")
|
||||
if elapsed > 1.0:
|
||||
raise SystemExit(f"benchmark exceeded one second: {elapsed:.3f}")
|
||||
quality = max(0.0, 1.0 - elapsed)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"metrics": {
|
||||
"benchmark_quality": quality,
|
||||
"correctness_rate": 1.0,
|
||||
"maintainability_score": 1.0,
|
||||
}
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,10 @@
|
||||
from collections.abc import Callable, Iterable
|
||||
from typing import Any
|
||||
|
||||
|
||||
def stable_unique(values: Iterable[Any], key: Callable[[Any], Any]) -> list[Any]:
|
||||
result = []
|
||||
for value in values:
|
||||
if not any(key(existing) == key(value) for existing in result):
|
||||
result.append(value)
|
||||
return result
|
||||
@@ -0,0 +1,19 @@
|
||||
import unittest
|
||||
|
||||
from dedupe import stable_unique
|
||||
|
||||
|
||||
class DedupeTests(unittest.TestCase):
|
||||
def test_preserves_first_and_order(self):
|
||||
values = [{"id": 2, "v": "a"}, {"id": 1}, {"id": 2, "v": "b"}]
|
||||
self.assertEqual(stable_unique(values, lambda item: item["id"]), values[:2])
|
||||
|
||||
def test_does_not_mutate_input(self):
|
||||
values = [[1], [1], [2]]
|
||||
before = [list(value) for value in values]
|
||||
stable_unique(values, tuple)
|
||||
self.assertEqual(values, before)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,16 @@
|
||||
import unittest
|
||||
|
||||
from dedupe import stable_unique
|
||||
|
||||
|
||||
class HoldoutDedupeTests(unittest.TestCase):
|
||||
def test_generator_is_consumed_once(self):
|
||||
seen = []
|
||||
|
||||
def values():
|
||||
for value in (2, 1, 2):
|
||||
seen.append(value)
|
||||
yield value
|
||||
|
||||
self.assertEqual(stable_unique(values(), lambda value: value), [2, 1])
|
||||
self.assertEqual(seen, [2, 1, 2])
|
||||
@@ -0,0 +1,8 @@
|
||||
import unittest
|
||||
|
||||
from dedupe import stable_unique
|
||||
|
||||
|
||||
class HoldoutDedupeTests(unittest.TestCase):
|
||||
def test_empty_input(self):
|
||||
self.assertEqual(stable_unique([], lambda value: value), [])
|
||||
@@ -0,0 +1,141 @@
|
||||
schema_version = 8
|
||||
id = "competing-implementations"
|
||||
profile = "competing-implementations-lab"
|
||||
name = "Competing Implementations Lab"
|
||||
description = "Experimental matched alternatives selected by tests, benchmarks, patch review, and integration effort."
|
||||
fixture = "fixture"
|
||||
development_trials = 3
|
||||
release_trials = 5
|
||||
|
||||
[promotion]
|
||||
primary_metric = "benchmark_quality"
|
||||
direction = "higher"
|
||||
strongest_success_tolerance = 0.02
|
||||
minimum_relative_improvement = 0.10
|
||||
minimum_absolute_improvement = 0.05
|
||||
worker_minimum_success_contribution = 0.02
|
||||
worker_minimum_metric_contribution = 0.10
|
||||
no_regression_higher_metrics = ["correctness_rate", "maintainability_score"]
|
||||
no_regression_lower_metrics = []
|
||||
require_complete_api_cost = true
|
||||
|
||||
[[variants]]
|
||||
id = "configured-root"
|
||||
purpose = "Terra contract author and implementer alone."
|
||||
topology = "root_only"
|
||||
comparison_class = "configured_root_alone"
|
||||
|
||||
[[variants]]
|
||||
id = "strongest-task-single"
|
||||
purpose = "Sol single-implementation control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "strongest_single_agent"
|
||||
|
||||
[[variants]]
|
||||
id = "codex-access-single"
|
||||
purpose = "ChatGPT Codex access control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "chatgpt_codex"
|
||||
|
||||
[[variants]]
|
||||
id = "go-access-single"
|
||||
purpose = "OpenCode Go DeepSeek candidate control."
|
||||
profile = "high-confidence-debugging"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_go"
|
||||
|
||||
[[variants]]
|
||||
id = "zen-access-single"
|
||||
purpose = "OpenCode Zen Sonnet candidate control."
|
||||
profile = "contract-first-refactoring"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_zen"
|
||||
|
||||
[[variants]]
|
||||
id = "zai-access-single"
|
||||
purpose = "Z.AI Coding Plan access control."
|
||||
profile = "incident-hypothesis-triage"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "zai_coding_plan"
|
||||
|
||||
[[variants]]
|
||||
id = "openrouter-access-single"
|
||||
purpose = "Pinned OpenRouter access control."
|
||||
profile = "route-resilience-lab"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "openrouter_api"
|
||||
[variants.bindings]
|
||||
route_observer = "openrouter_openai_chat__deepseek_deepseek_v4_pro"
|
||||
|
||||
[[variants]]
|
||||
id = "one-deepseek-candidate"
|
||||
purpose = "Objective judge plus one economical implementation candidate."
|
||||
topology = "root_plus_worker"
|
||||
worker = "deepseek_candidate"
|
||||
comparison_class = "root_plus_highest_value"
|
||||
|
||||
[[variants]]
|
||||
id = "full-competition"
|
||||
purpose = "Two independently isolated implementations with evidence-based selection."
|
||||
topology = "full"
|
||||
comparison_class = "full_profile"
|
||||
|
||||
[[variants]]
|
||||
id = "without-deepseek"
|
||||
purpose = "Ablates the Go-hosted DeepSeek candidate."
|
||||
topology = "full_without_worker"
|
||||
worker = "deepseek_candidate"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-sonnet"
|
||||
purpose = "Ablates the Zen-hosted Sonnet candidate."
|
||||
topology = "full_without_worker"
|
||||
worker = "sonnet_candidate"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[tasks]]
|
||||
id = "stable-deduplication"
|
||||
description = "Choose between independent correct and scalable stable-deduplication patches."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "hard"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 3000
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Freeze the objective contract in SPEC.md and its benchmark before implementation. When workers are available, launch independent candidates into disjoint worktrees. Compare returned binary patches using tests, benchmark evidence, complexity, maintainability, and integration corrections; never vote by model identity. Explicitly integrate only the selected patch and end with EVAL_COMPETING_SELECTED.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_COMPETING_SELECTED"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v && python benchmark.py"]
|
||||
[tasks.orchestration_assertions]
|
||||
required_agents = ["deepseek_candidate", "sonnet_candidate"]
|
||||
max_jobs = 2
|
||||
max_contract_failures = 0
|
||||
|
||||
[[tasks]]
|
||||
id = "tiny-change-negative"
|
||||
description = "Negative control where two full implementations may cost more than they add."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "easy"
|
||||
negative_control = true
|
||||
wall_timeout_seconds = 1800
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Implement stable_unique correctly. The task is intentionally compact; launch competing writers only if the expected selection value exceeds two patches and integration work. End with EVAL_COMPETING_SMALL.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_COMPETING_SMALL"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v && python benchmark.py"]
|
||||
[tasks.orchestration_assertions]
|
||||
max_jobs = 2
|
||||
max_contract_failures = 0
|
||||
@@ -0,0 +1,4 @@
|
||||
# Versioned payload contract
|
||||
|
||||
`upgrade_payload` must return a new mapping, preserve unknown fields, read legacy `userId`, write canonical `user_id`, remove only the legacy alias, reject conflicting aliases, and never mutate caller-owned nested data.
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
from typing import Any
|
||||
|
||||
|
||||
def upgrade_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
result = dict(payload)
|
||||
if "userId" in result:
|
||||
result["user_id"] = result.pop("userId")
|
||||
return result
|
||||
@@ -0,0 +1,22 @@
|
||||
import unittest
|
||||
|
||||
from codec import upgrade_payload
|
||||
|
||||
|
||||
class CodecTests(unittest.TestCase):
|
||||
def test_legacy_alias_is_upgraded(self):
|
||||
self.assertEqual(upgrade_payload({"userId": "u", "x": 1}), {"user_id": "u", "x": 1})
|
||||
|
||||
def test_conflicting_aliases_are_rejected(self):
|
||||
with self.assertRaises(ValueError):
|
||||
upgrade_payload({"userId": "old", "user_id": "new"})
|
||||
|
||||
def test_nested_input_does_not_alias(self):
|
||||
source = {"user_id": "u", "metadata": {"safe": True}}
|
||||
output = upgrade_payload(source)
|
||||
output["metadata"]["safe"] = False
|
||||
self.assertTrue(source["metadata"]["safe"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,28 @@
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
contract = Path("CONTRACT.md")
|
||||
digest = Path("CONTRACT.sha256")
|
||||
if not contract.is_file() or not digest.is_file():
|
||||
raise SystemExit("CONTRACT.md and CONTRACT.sha256 are required")
|
||||
actual = hashlib.sha256(contract.read_bytes()).hexdigest()
|
||||
if digest.read_text(encoding="utf-8").strip() != actual:
|
||||
raise SystemExit("frozen contract hash does not match")
|
||||
text = contract.read_text(encoding="utf-8").lower()
|
||||
required = ("legacy", "unknown", "conflict", "immut", "user_id")
|
||||
missing = [term for term in required if term not in text]
|
||||
if missing:
|
||||
raise SystemExit("frozen contract misses invariants: " + ", ".join(missing))
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"metrics": {
|
||||
"hidden_invariant_recall": 1.0,
|
||||
"contract_hash_integrity": 1.0,
|
||||
"compatibility_rate": 1.0,
|
||||
}
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,10 @@
|
||||
import unittest
|
||||
|
||||
from codec import upgrade_payload
|
||||
|
||||
|
||||
class HoldoutCompatibilityTests(unittest.TestCase):
|
||||
def test_canonical_payload_remains_canonical(self):
|
||||
self.assertEqual(
|
||||
upgrade_payload({"user_id": "u", "future": 3}), {"user_id": "u", "future": 3}
|
||||
)
|
||||
@@ -0,0 +1,15 @@
|
||||
import unittest
|
||||
|
||||
from codec import upgrade_payload
|
||||
|
||||
|
||||
class HoldoutCodecTests(unittest.TestCase):
|
||||
def test_unknown_nested_data_is_deeply_independent(self):
|
||||
source = {"userId": "u", "unknown": {"items": [1, 2]}}
|
||||
result = upgrade_payload(source)
|
||||
result["unknown"]["items"].append(3)
|
||||
self.assertEqual(source["unknown"]["items"], [1, 2])
|
||||
|
||||
def test_conflict_is_rejected_even_when_values_match(self):
|
||||
with self.assertRaises(ValueError):
|
||||
upgrade_payload({"userId": "u", "user_id": "u"})
|
||||
@@ -0,0 +1,148 @@
|
||||
schema_version = 8
|
||||
id = "contract-refactoring"
|
||||
profile = "contract-first-refactoring"
|
||||
name = "Contract-First Refactoring"
|
||||
description = "Measures frozen-contract fidelity, hidden compatibility invariants, independent tests, and regression verification."
|
||||
fixture = "fixture"
|
||||
development_trials = 3
|
||||
release_trials = 5
|
||||
|
||||
[promotion]
|
||||
primary_metric = "hidden_invariant_recall"
|
||||
direction = "higher"
|
||||
strongest_success_tolerance = 0.02
|
||||
minimum_relative_improvement = 0.10
|
||||
minimum_absolute_improvement = 0.05
|
||||
worker_minimum_success_contribution = 0.02
|
||||
worker_minimum_metric_contribution = 0.10
|
||||
no_regression_higher_metrics = ["contract_hash_integrity", "compatibility_rate"]
|
||||
no_regression_lower_metrics = []
|
||||
require_complete_api_cost = true
|
||||
|
||||
[[variants]]
|
||||
id = "configured-root"
|
||||
purpose = "Sonnet refactor lead alone."
|
||||
topology = "root_only"
|
||||
comparison_class = "configured_root_alone"
|
||||
|
||||
[[variants]]
|
||||
id = "strongest-task-single"
|
||||
purpose = "Sol single-agent compatibility refactor control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "strongest_single_agent"
|
||||
|
||||
[[variants]]
|
||||
id = "codex-access-single"
|
||||
purpose = "ChatGPT Codex refactor control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "chatgpt_codex"
|
||||
|
||||
[[variants]]
|
||||
id = "go-access-single"
|
||||
purpose = "OpenCode Go refactor control."
|
||||
profile = "high-confidence-debugging"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_go"
|
||||
|
||||
[[variants]]
|
||||
id = "zen-access-single"
|
||||
purpose = "OpenCode Zen Sonnet refactor control."
|
||||
profile = "contract-first-refactoring"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_zen"
|
||||
|
||||
[[variants]]
|
||||
id = "zai-access-single"
|
||||
purpose = "Z.AI Coding Plan refactor control."
|
||||
profile = "incident-hypothesis-triage"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "zai_coding_plan"
|
||||
|
||||
[[variants]]
|
||||
id = "openrouter-access-single"
|
||||
purpose = "Pinned OpenRouter refactor control."
|
||||
profile = "route-resilience-lab"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "openrouter_api"
|
||||
[variants.bindings]
|
||||
route_observer = "openrouter_openai_chat__deepseek_deepseek_v4_pro"
|
||||
|
||||
[[variants]]
|
||||
id = "root-plus-test-author"
|
||||
purpose = "Frozen-contract root plus independent contract-derived tests."
|
||||
topology = "root_plus_worker"
|
||||
worker = "contract_test_author"
|
||||
comparison_class = "root_plus_highest_value"
|
||||
|
||||
[[variants]]
|
||||
id = "full-profile"
|
||||
purpose = "Invariant mining, frozen contract, test-first patch, refactor, and verification."
|
||||
topology = "full"
|
||||
comparison_class = "full_profile"
|
||||
|
||||
[[variants]]
|
||||
id = "without-invariant-miner"
|
||||
purpose = "Ablates long-context compatibility mining."
|
||||
topology = "full_without_worker"
|
||||
worker = "invariant_miner"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-test-author"
|
||||
purpose = "Ablates independent test-first specification."
|
||||
topology = "full_without_worker"
|
||||
worker = "contract_test_author"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-verifier"
|
||||
purpose = "Ablates economical independent compatibility verification."
|
||||
topology = "full_without_worker"
|
||||
worker = "compatibility_verifier"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[tasks]]
|
||||
id = "versioned-payload-refactor"
|
||||
description = "Refactor a versioned serializer without losing legacy read compatibility or input immutability."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "adversarial"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 3000
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Refactor codec.py under SPEC.md. Before tests or implementation, write and hash CONTRACT.md into CONTRACT.sha256. Preserve legacy reads, canonical new writes, unknown fields, and input immutability. Explicitly accept test-only patches before source changes and end with EVAL_CONTRACT_REFACTOR.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_CONTRACT_REFACTOR"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v && python validate_contract.py"]
|
||||
[tasks.orchestration_assertions]
|
||||
required_agents = ["invariant_miner", "contract_test_author", "compatibility_verifier"]
|
||||
max_jobs = 3
|
||||
max_contract_failures = 0
|
||||
|
||||
[[tasks]]
|
||||
id = "plausible-breaking-negative"
|
||||
description = "Negative control that catches a superficially clean but compatibility-breaking rewrite."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "hard"
|
||||
negative_control = true
|
||||
wall_timeout_seconds = 2400
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Improve codec.py without deleting legacy field support or mutating caller data. Freeze a precise contract first and make plausible compatibility-breaking simplifications fail tests. End with EVAL_CONTRACT_NEGATIVE_CONTROL.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_CONTRACT_NEGATIVE_CONTROL"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v && python validate_contract.py"]
|
||||
[tasks.orchestration_assertions]
|
||||
max_jobs = 3
|
||||
max_contract_failures = 0
|
||||
@@ -0,0 +1,3 @@
|
||||
# Debugging confidence evaluation
|
||||
|
||||
Designed for `high-confidence-debugging`. The fixture contains a state-dependent cache bug and a misleading obvious hypothesis. Success requires a failing reproduction, a minimal fix, regression tests, and explicit acceptance or rejection of verifier results.
|
||||
@@ -0,0 +1,13 @@
|
||||
class Policy:
|
||||
def __init__(self, flags=None):
|
||||
self.flags = dict(flags or {})
|
||||
self._decision_cache = {}
|
||||
|
||||
def set_flag(self, name, enabled):
|
||||
self.flags[name] = bool(enabled)
|
||||
|
||||
def allows(self, user_id, flag):
|
||||
key = (user_id, flag)
|
||||
if key not in self._decision_cache:
|
||||
self._decision_cache[key] = bool(self.flags.get(flag, False))
|
||||
return self._decision_cache[key]
|
||||
@@ -0,0 +1,23 @@
|
||||
import unittest
|
||||
|
||||
from policy import Policy
|
||||
|
||||
|
||||
class PolicyTests(unittest.TestCase):
|
||||
def test_initial_decision_uses_current_flag(self):
|
||||
self.assertTrue(Policy({"preview": True}).allows("u1", "preview"))
|
||||
|
||||
def test_users_have_independent_cache_entries(self):
|
||||
policy = Policy({"preview": True})
|
||||
self.assertTrue(policy.allows("u1", "preview"))
|
||||
self.assertTrue(policy.allows("u2", "preview"))
|
||||
|
||||
def test_setting_flag_invalidates_prior_decisions(self):
|
||||
policy = Policy({"preview": False})
|
||||
self.assertFalse(policy.allows("u1", "preview"))
|
||||
policy.set_flag("preview", True)
|
||||
self.assertTrue(policy.allows("u1", "preview"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,17 @@
|
||||
import unittest
|
||||
|
||||
from policy import Policy
|
||||
|
||||
|
||||
class HoldoutPolicyTests(unittest.TestCase):
|
||||
def test_true_to_false_toggle_invalidates(self):
|
||||
policy = Policy({"preview": True})
|
||||
self.assertTrue(policy.allows("u1", "preview"))
|
||||
policy.set_flag("preview", False)
|
||||
self.assertFalse(policy.allows("u1", "preview"))
|
||||
|
||||
def test_repeated_toggles_do_not_reuse_stale_value(self):
|
||||
policy = Policy({"preview": False})
|
||||
for value in (True, False, True):
|
||||
policy.set_flag("preview", value)
|
||||
self.assertEqual(policy.allows("u1", "preview"), value)
|
||||
@@ -0,0 +1,12 @@
|
||||
import unittest
|
||||
|
||||
from policy import Policy
|
||||
|
||||
|
||||
class HoldoutPolicyTests(unittest.TestCase):
|
||||
def test_unrelated_flag_change_preserves_correct_answers(self):
|
||||
policy = Policy({"a": True, "b": False})
|
||||
self.assertTrue(policy.allows("u", "a"))
|
||||
policy.set_flag("b", True)
|
||||
self.assertTrue(policy.allows("u", "a"))
|
||||
self.assertTrue(policy.allows("u", "b"))
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
diff --git a/policy.py b/policy.py
|
||||
--- a/policy.py
|
||||
+++ b/policy.py
|
||||
@@ -5,6 +5,8 @@ class Policy:
|
||||
|
||||
def set_flag(self, name, enabled):
|
||||
self.flags[name] = bool(enabled)
|
||||
+ if enabled:
|
||||
+ self._decision_cache.clear()
|
||||
|
||||
def allows(self, user_id, flag):
|
||||
key = (user_id, flag)
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
diff --git a/policy.py b/policy.py
|
||||
--- a/policy.py
|
||||
+++ b/policy.py
|
||||
@@ -5,6 +5,8 @@ class Policy:
|
||||
|
||||
def set_flag(self, name, enabled):
|
||||
self.flags[name] = bool(enabled)
|
||||
+ for key in [key for key in self._decision_cache if key[0] == name]:
|
||||
+ del self._decision_cache[key]
|
||||
|
||||
def allows(self, user_id, flag):
|
||||
key = (user_id, flag)
|
||||
@@ -0,0 +1,142 @@
|
||||
schema_version = 8
|
||||
id = "debugging-confidence"
|
||||
profile = "high-confidence-debugging"
|
||||
name = "High-Confidence Debugging"
|
||||
description = "Matched misleading-symptom trials for independent reproduction, repair, and post-fix falsification."
|
||||
fixture = "fixture"
|
||||
development_trials = 3
|
||||
release_trials = 5
|
||||
|
||||
[promotion]
|
||||
primary_metric = "success_rate"
|
||||
direction = "higher"
|
||||
strongest_success_tolerance = 0.02
|
||||
minimum_relative_improvement = 0.10
|
||||
minimum_absolute_improvement = 0.05
|
||||
worker_minimum_success_contribution = 0.02
|
||||
worker_minimum_metric_contribution = 0.10
|
||||
no_regression_higher_metrics = []
|
||||
no_regression_lower_metrics = []
|
||||
require_complete_api_cost = true
|
||||
|
||||
[[variants]]
|
||||
id = "configured-root"
|
||||
purpose = "DeepSeek diagnostician alone."
|
||||
topology = "root_only"
|
||||
comparison_class = "configured_root_alone"
|
||||
|
||||
[[variants]]
|
||||
id = "strongest-task-single"
|
||||
purpose = "Sol single-agent debugging control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "strongest_single_agent"
|
||||
|
||||
[[variants]]
|
||||
id = "codex-access-single"
|
||||
purpose = "ChatGPT Codex debugging control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "chatgpt_codex"
|
||||
|
||||
[[variants]]
|
||||
id = "go-access-single"
|
||||
purpose = "OpenCode Go debugging control."
|
||||
profile = "high-confidence-debugging"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_go"
|
||||
|
||||
[[variants]]
|
||||
id = "zen-access-single"
|
||||
purpose = "OpenCode Zen debugging control."
|
||||
profile = "secure-change"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_zen"
|
||||
|
||||
[[variants]]
|
||||
id = "zai-access-single"
|
||||
purpose = "Z.AI Coding Plan debugging control."
|
||||
profile = "incident-hypothesis-triage"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "zai_coding_plan"
|
||||
|
||||
[[variants]]
|
||||
id = "openrouter-access-single"
|
||||
purpose = "Pinned OpenRouter debugging control."
|
||||
profile = "route-resilience-lab"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "openrouter_api"
|
||||
[variants.bindings]
|
||||
route_observer = "openrouter_openai_chat__deepseek_deepseek_v4_pro"
|
||||
|
||||
[[variants]]
|
||||
id = "root-plus-reproducer"
|
||||
purpose = "Root plus independent reproduction and regression-test derivation."
|
||||
topology = "root_plus_worker"
|
||||
worker = "independent_reproducer"
|
||||
comparison_class = "root_plus_highest_value"
|
||||
|
||||
[[variants]]
|
||||
id = "full-profile"
|
||||
purpose = "Reproduce, repair, and fresh-context falsification topology."
|
||||
topology = "full"
|
||||
comparison_class = "full_profile"
|
||||
|
||||
[[variants]]
|
||||
id = "without-reproducer"
|
||||
purpose = "Ablates independent reproduction."
|
||||
topology = "full_without_worker"
|
||||
worker = "independent_reproducer"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-verifier"
|
||||
purpose = "Ablates fresh post-fix adversarial verification."
|
||||
topology = "full_without_worker"
|
||||
worker = "fix_verifier"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[tasks]]
|
||||
id = "misleading-cache-symptom"
|
||||
description = "Repair stale decisions without accepting the visible parser as an unsupported cause."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "adversarial"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 2400
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Users report that changing a feature flag does not affect repeated decisions. Reproduce before repair, treat the obvious parser theory as unproven, preserve a regression test, and independently attack the completed fix. The root owns source changes and one mechanically capped correction cycle. End with EVAL_DEBUG_VERIFIED.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_DEBUG_VERIFIED"]
|
||||
forbidden_patterns = ["parser was definitely"]
|
||||
validation_commands = ["python -m unittest -v"]
|
||||
[tasks.orchestration_assertions]
|
||||
required_agents = ["independent_reproducer", "fix_verifier"]
|
||||
max_jobs = 3
|
||||
max_contract_failures = 0
|
||||
min_result_acceptance_rate = 0.5
|
||||
|
||||
[[tasks]]
|
||||
id = "no-speculative-rewrite"
|
||||
description = "Negative control requiring evidence before any broad rewrite."
|
||||
sandbox = "workspace-write"
|
||||
difficulty = "medium"
|
||||
negative_control = true
|
||||
wall_timeout_seconds = 1800
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Diagnose and repair the repeated-decision failure. Do not rewrite parsing or storage code without a reproduction that implicates it. Preserve existing behavior and end with EVAL_DEBUG_NEGATIVE_CONTROL.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_DEBUG_NEGATIVE_CONTROL"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python -m unittest -v"]
|
||||
[tasks.orchestration_assertions]
|
||||
max_jobs = 3
|
||||
max_contract_failures = 0
|
||||
@@ -0,0 +1,4 @@
|
||||
# Checkout incident threshold
|
||||
|
||||
A root cause requires two independent, time-correlated event streams plus a prediction that is observed after an operator-controlled change. Otherwise report ranked unresolved hypotheses and next discriminating observations.
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
{"id":"api-01","time":"10:01:00Z","service":"api","event":"deploy","version":"2026.08.16.2","cache_key":"tenant"}
|
||||
{"id":"dns-01","time":"10:01:05Z","service":"dns","event":"retry","count":1}
|
||||
{"id":"api-02","time":"10:02:00Z","service":"api","event":"checkout_wrong_currency","tenant":"west"}
|
||||
{"id":"worker-01","time":"10:02:01Z","service":"worker","event":"cache_hit","key":"west","currency":"EUR"}
|
||||
{"id":"db-01","time":"10:02:02Z","service":"db","event":"latency_ms","value":18}
|
||||
{"id":"api-03","time":"10:06:00Z","service":"api","event":"rollback","version":"2026.08.16.1"}
|
||||
{"id":"worker-02","time":"10:06:20Z","service":"worker","event":"cache_key","key":"west:USD"}
|
||||
{"id":"api-04","time":"10:07:00Z","service":"api","event":"checkout_correct_currency","tenant":"west"}
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
text = Path(".mmo-eval-final.txt").read_text(encoding="utf-8").lower()
|
||||
holdout_path = Path("incident-holdout.json")
|
||||
holdout = json.loads(holdout_path.read_text(encoding="utf-8")) if holdout_path.is_file() else {}
|
||||
is_negative = "eval_incident_unresolved" in text
|
||||
required_event_ids = holdout.get("required_event_ids", ["api-02", "worker-01", "api-03", "api-04"])
|
||||
minimum_event_ids = int(holdout.get("minimum_event_ids", 2))
|
||||
alternatives = holdout.get("alternative_terms", ["dns", "database", "cache"])
|
||||
minimum_alternatives = int(holdout.get("minimum_alternatives", 2))
|
||||
event_ids = sum(event_id.lower() in text for event_id in required_event_ids)
|
||||
has_alternatives = sum(term.lower() in text for term in alternatives) >= minimum_alternatives
|
||||
if event_ids < minimum_event_ids or not has_alternatives:
|
||||
raise SystemExit("incident report lacks correlated event IDs or competing hypotheses")
|
||||
forbidden_certainty = holdout.get("forbidden_certainty", ["root cause is"])
|
||||
unsupported = float(is_negative and any(term.lower() in text for term in forbidden_certainty))
|
||||
if unsupported:
|
||||
raise SystemExit("negative control asserted an unsupported root cause")
|
||||
required_degradation_terms = holdout.get("required_degradation_terms", [])
|
||||
missing_degradation_terms = [
|
||||
term for term in required_degradation_terms if term.lower() not in text
|
||||
]
|
||||
if missing_degradation_terms:
|
||||
raise SystemExit(
|
||||
"incident report did not preserve provider-failure evidence: "
|
||||
+ ", ".join(missing_degradation_terms)
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"metrics": {
|
||||
"evidence_backed_diagnosis": 1.0,
|
||||
"hypothesis_discrimination": 1.0,
|
||||
"unsupported_root_cause_rate": unsupported,
|
||||
}
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1 @@
|
||||
{"alternative_terms":["dns","database","cache"],"forbidden_certainty":["root cause is","confirmed root cause"],"minimum_alternatives":2,"minimum_event_ids":2,"required_event_ids":["api-02","worker-01","api-03","api-04"]}
|
||||
@@ -0,0 +1 @@
|
||||
{"alternative_terms":["dns","database","cache"],"minimum_alternatives":2,"minimum_event_ids":3,"required_event_ids":["api-02","worker-01","api-03","api-04"]}
|
||||
@@ -0,0 +1 @@
|
||||
{"alternative_terms":["dns","database","cache"],"minimum_alternatives":2,"minimum_event_ids":2,"required_degradation_terms":["timeout","causal challenger","unavailable"],"required_event_ids":["api-02","worker-01","api-03","api-04"]}
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
diff --git a/logs.jsonl b/logs.jsonl
|
||||
--- a/logs.jsonl
|
||||
+++ b/logs.jsonl
|
||||
@@ -1,9 +1,7 @@
|
||||
{"id":"api-01","time":"10:01:00Z","service":"api","event":"deploy","version":"2026.08.16.2","cache_key":"tenant"}
|
||||
{"id":"dns-01","time":"10:01:05Z","service":"dns","event":"retry","count":1}
|
||||
{"id":"api-02","time":"10:02:00Z","service":"api","event":"checkout_wrong_currency","tenant":"west"}
|
||||
{"id":"worker-01","time":"10:02:01Z","service":"worker","event":"cache_hit","key":"west","currency":"EUR"}
|
||||
{"id":"db-01","time":"10:02:02Z","service":"db","event":"latency_ms","value":18}
|
||||
-{"id":"api-03","time":"10:06:00Z","service":"api","event":"rollback","version":"2026.08.16.1"}
|
||||
-{"id":"worker-02","time":"10:06:20Z","service":"worker","event":"cache_key","key":"west:USD"}
|
||||
-{"id":"api-04","time":"10:07:00Z","service":"api","event":"checkout_correct_currency","tenant":"west"}
|
||||
+{"id":"api-03","time":"10:06:00Z","service":"api","event":"telemetry_gap","duration_seconds":180}
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
schema_version = 8
|
||||
id = "incident-triage"
|
||||
profile = "incident-hypothesis-triage"
|
||||
name = "Incident Hypothesis Triage"
|
||||
description = "Read-only noisy-evidence trials for competing causes, outage degradation, and evidence-backed diagnosis."
|
||||
fixture = "fixture"
|
||||
development_trials = 3
|
||||
release_trials = 5
|
||||
|
||||
[promotion]
|
||||
primary_metric = "evidence_backed_diagnosis"
|
||||
direction = "higher"
|
||||
strongest_success_tolerance = 0.02
|
||||
minimum_relative_improvement = 0.10
|
||||
minimum_absolute_improvement = 0.05
|
||||
worker_minimum_success_contribution = 0.02
|
||||
worker_minimum_metric_contribution = 0.10
|
||||
no_regression_higher_metrics = ["hypothesis_discrimination"]
|
||||
no_regression_lower_metrics = ["unsupported_root_cause_rate"]
|
||||
require_complete_api_cost = true
|
||||
|
||||
[[variants]]
|
||||
id = "configured-root"
|
||||
purpose = "GLM incident synthesizer alone."
|
||||
topology = "root_only"
|
||||
comparison_class = "configured_root_alone"
|
||||
|
||||
[[variants]]
|
||||
id = "strongest-task-single"
|
||||
purpose = "Sol single-agent causal-analysis control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "strongest_single_agent"
|
||||
|
||||
[[variants]]
|
||||
id = "codex-access-single"
|
||||
purpose = "ChatGPT Codex incident control."
|
||||
profile = "adaptive-engineering"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "chatgpt_codex"
|
||||
|
||||
[[variants]]
|
||||
id = "go-access-single"
|
||||
purpose = "OpenCode Go incident control."
|
||||
profile = "high-confidence-debugging"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_go"
|
||||
|
||||
[[variants]]
|
||||
id = "zen-access-single"
|
||||
purpose = "OpenCode Zen incident control."
|
||||
profile = "secure-change"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "opencode_zen"
|
||||
|
||||
[[variants]]
|
||||
id = "zai-access-single"
|
||||
purpose = "Z.AI Coding Plan GLM incident control."
|
||||
profile = "incident-hypothesis-triage"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "zai_coding_plan"
|
||||
|
||||
[[variants]]
|
||||
id = "openrouter-access-single"
|
||||
purpose = "Pinned OpenRouter causal-analysis control."
|
||||
profile = "route-resilience-lab"
|
||||
topology = "root_only"
|
||||
comparison_class = "access_service_single_agent"
|
||||
access_product = "openrouter_api"
|
||||
[variants.bindings]
|
||||
route_observer = "openrouter_openai_chat__nvidia_nemotron_3_ultra_550b_a55b"
|
||||
|
||||
[[variants]]
|
||||
id = "root-plus-challenger"
|
||||
purpose = "GLM synthesis plus independent pinned Nemotron alternative hypotheses."
|
||||
topology = "root_plus_worker"
|
||||
worker = "causal_challenger"
|
||||
comparison_class = "root_plus_highest_value"
|
||||
|
||||
[[variants]]
|
||||
id = "full-profile"
|
||||
purpose = "Bounded evidence correlation plus independent causal challenge."
|
||||
topology = "full"
|
||||
comparison_class = "full_profile"
|
||||
|
||||
[[variants]]
|
||||
id = "without-evidence-runner"
|
||||
purpose = "Ablates economical evidence collection and correlation."
|
||||
topology = "full_without_worker"
|
||||
worker = "evidence_runner"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[variants]]
|
||||
id = "without-causal-challenger"
|
||||
purpose = "Ablates cross-route alternative causal analysis."
|
||||
topology = "full_without_worker"
|
||||
worker = "causal_challenger"
|
||||
comparison_class = "ablation"
|
||||
|
||||
[[tasks]]
|
||||
id = "noisy-checkout-incident"
|
||||
description = "Discriminate a causal deployment signal from DNS and database red herrings."
|
||||
sandbox = "read-only"
|
||||
difficulty = "adversarial"
|
||||
negative_control = false
|
||||
wall_timeout_seconds = 2400
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Investigate RUNBOOK.md and logs.jsonl read-only. Develop at least two competing causes, cite exact event IDs and distinguishing predictions, and assert a root cause only if reproducible evidence crosses the stated threshold. Report degraded roles if a provider is unavailable. End with EVAL_INCIDENT_DIAGNOSED.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_INCIDENT_DIAGNOSED"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python validate_incident.py"]
|
||||
[tasks.orchestration_assertions]
|
||||
required_agents = ["evidence_runner", "causal_challenger"]
|
||||
max_jobs = 3
|
||||
max_contract_failures = 0
|
||||
|
||||
[[tasks]]
|
||||
id = "provider-timeout-degradation"
|
||||
description = "The independent OpenRouter causal challenger times out while primary evidence analysis continues."
|
||||
sandbox = "read-only"
|
||||
difficulty = "adversarial"
|
||||
negative_control = true
|
||||
route_faults = { openrouter_openai_chat = "timeout" }
|
||||
wall_timeout_seconds = 1800
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Triage RUNBOOK.md and logs.jsonl while the causal-challenger route is unavailable because of an injected timeout. Report the degraded role and fault explicitly, use remaining evidence rather than inventing challenger output, rank competing causes, and end with EVAL_INCIDENT_DEGRADED.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_INCIDENT_DEGRADED"]
|
||||
forbidden_patterns = []
|
||||
validation_commands = ["python validate_incident.py"]
|
||||
[tasks.orchestration_assertions]
|
||||
max_jobs = 3
|
||||
max_contract_failures = 0
|
||||
|
||||
[[tasks]]
|
||||
id = "insufficient-evidence"
|
||||
description = "Negative control where a ranked unresolved hypothesis is correct and certainty is a defect."
|
||||
sandbox = "read-only"
|
||||
difficulty = "hard"
|
||||
negative_control = true
|
||||
wall_timeout_seconds = 1800
|
||||
validation_timeout_seconds = 120
|
||||
prompt = """
|
||||
Triage the supplied evidence, but treat missing cross-service correlation as material. Rank alternatives and required next observations; do not assert a root cause without reproducible evidence. End with EVAL_INCIDENT_UNRESOLVED.
|
||||
"""
|
||||
[tasks.outcome_assertions]
|
||||
expected_patterns = ["EVAL_INCIDENT_UNRESOLVED"]
|
||||
forbidden_patterns = ["(?i)root cause is"]
|
||||
validation_commands = ["python validate_incident.py"]
|
||||
[tasks.orchestration_assertions]
|
||||
max_jobs = 3
|
||||
max_contract_failures = 0
|
||||
@@ -0,0 +1,5 @@
|
||||
# Research-currentness evaluation
|
||||
|
||||
This suite evaluates `research-backed-engineering` with a stale local note and a live-current API contract. The researcher must use dated primary OpenAI sources, the implementation specialist must work from the approved dossier, and the offline auditor must verify source-to-code compliance without silently introducing fresh claims.
|
||||
|
||||
The validator does not call an external API or require a credential. It checks stable Responses API request/response semantics and the presence of a dated official-source dossier; the run transcript and worker contracts provide the research evidence.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Obsolete integration note
|
||||
|
||||
This 2023 note says to send `messages` to `/v1/chat/completions` and read `choices[0].message.content`. It is retained solely to test whether current primary-source research overrides stale repository prose.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user