codex-mmo/docs/ARCHITECTURE.md

# Architecture

## Application purpose

Codex MMO is a single-user, host-local CLI and process supervisor for running immutable,
provider-neutral multi-agent Codex profiles. Its normal path is: resolve a profile, compile an
immutable snapshot, create or resume one durable root session, and coordinate native Codex agents
and isolated Agent-MCP workers while retaining evidence and enforcing the profile's lineage,
resource, workspace, and lifecycle policy.

The application owns orchestration state, generated Codex homes, local worker processes, immutable
profile snapshots, and optional local Switchyard processes. Codex app-server owns model-thread and
turn execution; providers own inference; operator-defined Tool MCP servers own their external tools.
The deployment target is one workstation and a bounded number of local processes—not a distributed,
multi-tenant scheduler. Correct recovery, explicit ownership, inspectable state, and useful root
productivity matter more than horizontal scale or a general workflow language.

## Design objective

Codex MMO provides a small generic execution kernel for many model compositions. The runtime owns process and state mechanics; profiles own intelligence policy. No scheduler branch depends on a particular model or provider name.

```text
static profile pack
      |
      v
profile resolver + capability validation
      |
      v
content-addressed immutable snapshot
      |
      +-- compiled per-role AGENTS.md and profile orchestration SKILL.md
      +-- generated Codex homes and native role files
      +-- generated Switchyard routes when required
      +-- resolved spawn/control graphs, contracts, resources, and trust
      |
      v
immutable root app-server host + persistent thread
      |
      +-- detachable stock Codex TUI/noninteractive client
      +-- native Codex participant threads
      |
      +-- Agent MCP supervisor
             |
             +-- isolated persistent Unix app-server workers
```

## Layer 1: runtime kernel

The kernel implements only generic mechanisms:

- Session and job lifecycle.
- Content-addressed snapshot lookup.
- Agent lineage and authenticated caller identity.
- Directed spawn- and control-edge admission.
- Depth, active-child, per-role, session, writer, and resource limits.
- Workspace and attachment containment.
- Write-scope conflict detection.
- Asynchronous app-server hosting, same-thread recovery, live control, result retrieval, graceful stop, and immediate cancellation.
- Codex goal-token lifecycle, turn-mode execution without a task clock, warning-only stalls, and bounded explicit terminal serialization.
- Output-contract extraction and validation.
- Durable audit, headerless app-server protocol event trace, stderr, partial/final result, patch, and metadata storage.
- Credential filtering and generated Codex homes.
- Content-addressed gateway lifecycle shared by equivalent route sets.

The kernel does not decide that a task is “Qwen work,” “GLM work,” or “DeepSeek work.” A profile maps task categories and role instructions to models.

### Code ownership

The implementation is organized around that execution path:

- `mmo_profiles.py` resolves composition policy and profile lifecycle; `mmo_catalog_data.py` owns
  catalog records and overlays, while `mmo_catalog.py` owns discovery and inventory operations.
- `mmo_snapshot.py` compiles immutable resolved artifacts. `mmo_codex_home.py` projects those
  artifacts into generated Codex homes, native-role files, and process environments.
- `mmo_runtime.py` is the orchestration center: admission, root/session commands, lineage, resource
  and write-scope policy, result disposition, and stop/cancel decisions live there.
- `mmo_state.py` owns durable state paths, strict record readers, canonical session/job publication,
  session/run mirroring, and persisted session capabilities. Callers do not cache bearer tokens or
  write `session.json`/`metadata.json` directly. `mmo_workspace.py` owns Git worktree and patch
  mechanics.
- `mmo_app_server.py` owns the exact Codex 0.149.0 Unix WebSocket transport, protocol gate,
  persistent-thread requests, server-request validation, bounded local control framing, control
  delivery, and partial-evidence extraction. `root_runner.py` and `worker_runner.py` own their
  respective session/job lifecycle, recovery, authoritative history, goal state, action semantics,
  and terminal publication around that shared transport. Each runner has one concrete lifetime owner;
  they deliberately do not share a base class.
- `mmo_gateway.py` owns Switchyard lifecycle, route availability, and interpretation of provider/router
  telemetry. Generic worker execution records the resulting observation but does not parse
  OpenRouter- or Switchyard-specific event shapes.
- `mmo_diagnostics.py` composes profile validation, doctor, MCP handshake, and live smoke workflows.
  `mmoctl.py` parses commands and renders results; it does not own those workflows.

These are concrete ownership boundaries, not replaceable service interfaces. Direct module calls are
intentional because MMO is one local application, not a plugin container or distributed service.

## Layer 2: composition profiles

A profile defines:

- One root agent.
- Any number of participant roles.
- Model binding and reasoning effort per role.
- Native, MCP, or hybrid execution per participant.
- Directed `can_spawn` edges and per-target action-specific `controls` grants.
- Task-kind admission.
- Trust and verification policy.
- Read-only or workspace-write permission.
- Required input modalities.
- Output contract and enforcement level.
- Per-role concurrency, child, task-size, goal-token/stall/finalization, reasoning-effort, and resource limits.
- Session-wide depth, spawn, writer, and active-agent limits.
- Coordination and contradiction policy.
- Role-specific instructions and smoke tests.
- Per-role grants to operator-defined third-party Tool MCP servers.

Profile packs are static and auditable. They cannot include executable code.

## Layer 3: model and route catalog

The global catalog keeps model maker/upstream identity separate from executable route identity. Routes record API operator, access product, wire protocol, billing mode, endpoint, credential names, driver, transport behavior, and resource pool. Models record capabilities, costs where documented, inventory membership, context/output limits, reasoning levels, modalities, tools, and availability.

Externally sourced built-in inventories are catalog-ready records under `config/inventory-snapshots/`. OpenRouter, OpenCode Zen/Go, Z.AI, and Codex use the same schema, provenance envelope, and canonical model-record digest regardless of inventory size. `scripts/generate_catalog.py` validates and aggregates those snapshots offline, derives aggregate metadata from them, and runtime inventory verification compares their exact normalized records with the catalog. Local deployment entries remain explicit curated configuration.

Profiles refer to exact catalog keys rather than embedding endpoint configuration. Every model key begins with its exact route key plus `__`, so a binding identifies its access path without consulting a hidden default. Operators can add or override generation-8 catalog entries under:

```text
~/.config/codex-mmo/catalog.d/*.toml
```

Fragments are applied in lexical order. Profile-local `catalog.toml` fragments are applied after global overlays. Fragments from any other MMO generation and old model keys are rejected without migration or lookup fallback.

## Layer 4: Tool MCP registry

Machine-specific third-party MCP commands and endpoints live under
`~/.config/codex-mmo/tool-mcp.d/*.toml`, outside installable profile packs. The registry defines a
finite tool and approval-policy ceiling. Profiles reference stable server IDs and may narrow that
ceiling per role but cannot change transport or operator policy. Referenced normalized definitions
are part of snapshot identity; credential values are not.

At generated-config time, standalone Agent-MCP workers receive only relevant definitions. A process
with native roles receives the union needed by that native tree, with an explicit enable/disable
matrix per role because Codex custom-agent tables merge recursively. `mmo_mesh` remains a reserved
runtime-owned Agent MCP identity.

## Layer 5: credentials

Credentials are never stored in profile packs or snapshots.

- Built-in ChatGPT/Codex access links or explicitly copies file-backed `auth.json` from the configured base `CODEX_HOME`; keyring-only state cannot cross Codex 0.149's home-scoped keyring namespace.
- Switchyard receives the credentials declared by the snapshot’s selected routes; conventional unrelated credential-like names are filtered from its inherited environment.
- MCP worker shells receive the same name-based filter, with only a direct selected route's declared credential/header variables admitted explicitly. Unconventionally named secrets cannot be identified mechanically.
- A direct `codex_custom` route is the explicit exception: its required credential must be supplied to the Codex process that calls it.
- Environment variables referenced by selected Tool MCP definitions are supplied explicitly. Native
  roles share these values with their parent process; Agent-MCP workers retain process isolation.

## Snapshot compilation

`codex-mmo profile compile PROFILE` performs:

1. Static profile-pack tree validation.
2. Global catalog load and deterministic overlay merge.
3. Operator Tool MCP registry resolution and grant validation.
4. Provider, resource, and model validation.
5. Runtime model rebinding.
6. Agent capability, modality, trust, backend, and contract validation.
7. Spawn-graph and coordination validation.
8. Logical hash calculation excluding profile installation location.
9. Deterministic per-role `AGENTS.md` and profile orchestration `SKILL.md` compilation.
10. Snapshot identity calculation over resolved semantics and exact generated guidance bytes.
11. Snapshot materialization under the state root.
12. Deterministic Switchyard route generation if required.
13. Read-only permission application.

Snapshots live under:

```text
~/.local/state/codex-mmo/snapshots/SHA256/
```

Every session records the profile ID, profile version, logical hash, physical snapshot hash, and resolved Codex executable. Descendants and resumed runs use the same snapshot and pinned executable even if PATH, the active profile, or future-session settings change. Within the current resolved-profile schema, snapshot loading authenticates historical generated guidance from the exact stored bytes covered by that snapshot's content address; it does not regenerate prose with a newer compiler. A snapshot carrying a retired resolved-profile schema is rejected rather than interpreted under new admission semantics. The remaining manifest semantics and non-guidance payloads are still derived from the stored resolved profile and verified byte-for-byte, so this is immutable verification rather than a migration or compatibility rewrite.

## Persistent sessions and execution runs

A root session is a durable logical container for one immutable profile snapshot, pinned working-directory boundary, generated Codex home, protected control identity, one app-server host, and one current top-level Codex thread in an ordered lineage. Plain `codex-mmo` records `session_kind = "interactive"`; `codex-mmo exec` records `"noninteractive"`. Both use the same canonical execution host. The kind describes the first client, not a backend choice.

Each session contains exactly one immutable run record. Reattach, pause/continue, controller replacement, and transport recovery preserve its run ID, capabilities, goal token budgets, workers, evidence, and exact thread identities. Detach retains scheduler capacity while its hosts remain active; cold pause and suspension release scheduler admission capacity, and continuation performs fresh admission before starting replacement controllers for those same threads. Gateway leases are deliberately separate: a preserved live root or worker app-server still consumes its route-set gateway even when its logical session/job no longer reserves scheduler capacity. The stock TUI may deliberately start a fresh top-level context; that advances an explicit root-thread generation inside the same immutable MMO session/run instead of silently creating another MMO session. Every predecessor, successor, timestamp, reason, and in-progress transition is durable, while native child/fork threads remain ineligible. Resume accepts any predecessor ID as a locator but always attaches the current canonical generation.

Resume first requires the session's exact MMO package-generation marker, then verifies the pinned snapshot and executable, restores its content-addressed gateway before refreshing generated homes, and reconciles dead session workers to evidence-retaining suspension before returning control. It never compiles the current profile, searches rollout files or unrelated Codex history, creates a second session/run, rotates the MMO identity, or adopts another runtime generation. Controller crash recovery may call the pinned app-server's `thread/list` only inside the session-isolated root home to recover a top-level successor already created by the attached TUI. A terminal session cannot be resumed.

`session detach` disconnects the client while the root host and descendants continue. `session pause` first persists a cold-pause checkpoint and partial evidence, then pauses/interrupts admitted turns and retires fingerprint-matched root and supervised-worker hosts. `session continue` freshly admits and reactivates the same root generation and exact paused worker set; it may increase a goal's total token budget only within the compiled ceiling. `session compact` temporarily hosts the paused root, invokes app-server compaction, and cold-pauses it again. `session stop --grace` requests evidence-only finalization, waits a bounded operator grace, then retires every host. `session cancel` is immediate termination with evidence retention.

Every MMO-owned runtime record uses schema 8 and exact package version 8.0.0; the redundant lifecycle-version field no longer exists. Job app-server and controller sockets use immutable identity-derived names in a private short runtime directory, so workspace and state-root length cannot disable control. Retired fields, other generations, and additional run inventories are rejected at load time, including inspection. Historical evidence belongs in an external manual archive and can enter a new session only through an explicit bounded handoff.

## Root sessions

A root session creates a session-specific generated Codex home containing:

- Root model/route configuration.
- The snapshot's exact root `AGENTS.md`.
- A registered `skills/mmo-profile-orchestration/SKILL.md` when the role may spawn or control another agent.
- Native custom-agent files reachable from the root.
- An Agent MCP server definition when the root has MCP-reachable children.
- Root-granted Tool MCP definitions plus disabled definitions needed by reachable native roles.
- Links to built-in Codex authentication when required.
- A copy/link of the model catalog metadata needed by Codex.

Every root runner starts or reconnects to one isolated `codex app-server --listen unix://…` host and persists its current thread generation, ordered lineage, Unix socket, event stream, terminal history, goal state, pending requests, and process identity. Interactive use starts the stock Codex TUI with `--remote unix://… resume THREAD_ID`; the TUI never owns the durable host. A `thread/started` notification can advance the canonical generation only while the fingerprinted attached TUI is alive, the prior canonical thread matches the host, the prior root has no active turn, and the candidate is a persistent top-level thread in the pinned working directory. The transition is staged durably before publication; a replacement controller reconciles an interrupted transition from the isolated app-server thread index. A fresh generation receives the ongoing interactive goal before its first accepted turn is activated.

Noninteractive execution is another client of the same lifecycle. An external `--wall-timeout` detaches that client while the app-server goal continues; it never interrupts or deletes work. Pending user input, MCP elicitation, and non-auto-denied approval requests remain durably inspectable and cause the noninteractive caller to detach rather than fabricate a response.

A retryable failed turn, including a provider usage limit or transient transport failure, is durable thread history. While the fingerprinted TUI remains attached, the root controller retains partial evidence and keeps the app-server connection alive so the user can submit a later turn after recovery. Headless execution suspends and retires the host; explicitly detaching after an attached retryable failure does the same. Resume starts a replacement controller against the same thread rather than replaying or deleting the failed turn.

A root is published as completed only when its terminal history contains a readable agent result. If Codex reports completion but terminal history cannot supply that result, MMO reports failure and retains the available event evidence as partial output; it never converts an empty or unreadable result into success.

`root_execution_host` has one valid value: `app_server`. Client detach, TUI exit, terminal suspension, or controller replacement does not rewrite it. This keeps process ownership and observability truthful across every interface.

## Interactive terminal ownership

The stock remote TUI is launched without an intermediate terminal emulator or output proxy. The wrapper creates a dedicated client process group, assigns it as the terminal foreground owner, and waits with job-control awareness. Resize, interrupt, suspend, and input reach the TUI directly; the independently hosted app-server remains under MMO lifecycle control. On completion or suspension, the wrapper restores foreground ownership and saved terminal attributes before returning control to the shell, then records a client detach if the session remains active.

Root/worker runners and their app-server hosts use isolated process groups so full stop/cancel can retire exact descendants without targeting an unrelated shell group. Noninteractive callers are disposable clients; host lifetime is independent from them.

## Codex startup model catalogs

A generated provider route ID is also the model slug Codex sees. For every process containing a non-built-in model, the compiler writes `models.json` inside that process's isolated `CODEX_HOME` and points `model_catalog_json` at it. Rows are derived from the resolved model/provider binding and include exact slug, context window, reasoning presets, patch-tool transport, modalities, parallel-tool capability, and compaction headroom.

A custom startup catalog replaces Codex's normal startup catalog. If one native-agent process mixes built-in Codex models with external participants, the runtime therefore queries the active binary with `codex debug models --bundled`, caches the result by binary fingerprint, and merges the generated external route rows into those exact rows. Discovery occurs before the global runtime admission lock so a slow binary probe cannot serialize unrelated sessions.

## Native participants

Native participants are generated as Codex custom-agent TOML files. They inherit the current Codex workspace and sandbox environment. Their role file pins the model, provider, reasoning effort, description, instructions, and optional MCP server access.

Tool MCP visibility is role-scoped, but native roles share one operating-system process environment.
The compiler warns when a native role's Tool MCP uses environment-backed authentication.

Native root-to-participant delegation is not visible to the external supervisor at every tool boundary. Therefore native limits are partly advisory. The default hybrid policy does not permit unobserved recursive native trees: a native participant that can delegate receives Agent MCP access for its children.

## Agent MCP participants

Agent MCP is a local STDIO server with up to twenty-four generic tools. Each generated server exposes exactly the subset authorized for that caller. A controller without spawn authority receives only its action-granted observation/control and result-read tools, not lineage cancellation, spawn, disposition, or patch-integration authority.

The server implements the MCP `2025-06-18` initialization lifecycle over newline-delimited UTF-8 STDIO and validates the JSON-RPC 2.0 request envelope. Calls other than `ping` are unavailable until a valid `initialize` request and `notifications/initialized` notification complete. If a client requests an unsupported protocol revision, the server returns its supported revision rather than echoing the unknown value.

- `agent_spawn`
- `agents_spawn`
- `agent_status`
- `agents_wait`
- `agent_result`
- `agent_result_accept`
- `agent_result_reject`
- `agent_patch_integrate`
- `agent_cancel`
- `agent_list`
- `agent_inspect`
- `agent_trace`
- `agent_trace_record`
- `agent_steer`
- `agent_interrupt`
- `agent_pause`
- `agent_continue`
- `agent_detach`
- `agent_stop`
- `agent_finalize`
- `agent_compact`
- `agent_respond`
- `agent_set_effort`
- `agent_fork`

Each spawn starts a detached runner that privately hosts `codex app-server --listen unix://…`. Controller connections are disposable; the non-ephemeral Codex thread plus authoritative event/terminal history is the durable worker identity. A worker has:

- A minimal generated `CODEX_HOME`.
- A pinned model/provider/reasoning role.
- A bounded prompt and compiled `goal` or `turn` lifecycle.
- A sandbox no stronger than the role maximum.
- Optional validated attachments.
- Optional explicit write scopes.
- A role output contract plus command/artifact correlation when declared.
- App-server `turn/start.outputSchema` constrained generation when a strict contract can be projected onto Codex's supported schema subset and the resolved model advertises structured-output support. The full original schema remains the independent enforcement authority. One same-thread shape-only repair turn may correct malformed JSON; no separate fixer role may invent evidence.
- An isolated Git worktree and patch capture for writable workers.
- Durable headerless app-server protocol events, terminal thread history, stderr, partial/final results, usage, and exact thread identity.
- A distinct inherited MCP caller capability whose plaintext is not persisted.
- Only Tool MCP definitions selected for that worker and its reachable native roles.

Each runner exposes an owner-only Unix control socket. The app-server boundary owns the shared strict
JSON/newline framing and size limit; root and worker runners retain their different action semantics.
Agents do not receive a worker's raw app-server socket: the authenticated Agent-MCP kernel resolves
an opaque `agent_run_ref`, checks the action-specific `controls` grant, exact session/run identity,
role, and target backend, then relays the request with compare-and-swap revision semantics. The same
interface covers root, native, and MCP runs. Inspect and paginated trace filter private reasoning while
retaining completed messages and empirical tool evidence. An oversized event becomes a checksummed
bounded summary with a private `record_cursor`; `agent_trace_record` then returns the exact filtered
record in contiguous checksummed pages. Malformed records retain only a bounded diagnostic because
their raw content cannot be structurally filtered. Mutations support steering, current-turn
interrupt, goal pause, same-thread continue/token extension, client detach, full stop, evidence-only
finalization, compaction, exact pending-request response, allowed effort changes, and fork. Mutations
serialize revision assignment through delivery. A reply lost after delivery begins is
`delivery_unknown`; callers inspect before deciding whether a retry is safe.

The root and MCP participants receive Agent MCP access only for declared child/action roles. One 0600
session capability document is the only runtime source of the root/native identities required by
independently hosted detach/recovery across controller processes; public state contains only hashes
and no process-global token cache can outlive or mask revocation of that file. It is stable for the
immutable run and destroyed when the session becomes terminal. Job capabilities are inherited only
by their detached runners. Native role capabilities live only in protected generated role
configuration. The supervisor authenticates the exact session/run pair and durable lineage; a caller
cannot claim another role, parent, or run.

## Goal, turn, and finalization model

Models are never asked to measure time. A `turn` role has no profile task clock. A `goal` role uses Codex's own cumulative token accounting, an initial `goal_token_budget`, and a profile ceiling. An authorized `continue` may raise the total budget within that ceiling. Once the objective and every required integration are complete, the model calls `update_goal(status="complete")` in its terminal turn; a final assistant message alone deliberately does not end an active goal. Goal completion is intent, not a replacement for `turn/completed`: the host remains attached until that terminal turn and its final message are durably recorded. Stall intervals publish operator warnings only; slow providers, long tool calls, pending input, and silence do not erase work or trigger model-blind termination.

Goal budget exhaustion never fabricates a terminal answer. The host first consumes the authoritative completion of any already-admitted turn. It publishes that turn only when its current-turn identity, successful status, and strict result contract all validate; the result records the limit and completes with a warning. Otherwise the worker suspends recoverably with partial evidence. Explicit finalization or full stop may request one evidence-only terminal serialization bounded by `finalization_grace_seconds`; strict contracts may receive at most one same-thread shape-only repair. Failure to serialize never deletes the authoritative events, complete terminal history, bounded readable partial, or isolated patch.

If a client transport exits, the runner reconnects to the exact Unix host/thread. If the host itself is lost, recovery starts one replacement host only for that same persisted thread. A turn committed before transport loss is read from authoritative history rather than duplicated. If hosting still fails, the run becomes `suspended`; trace, terminal history, pending requests, partial result, patch, and thread identity remain available for explicit continuation.

## Lineage and admission

For each MCP spawn, the runtime validates:

1. The root session is active and uses the requested snapshot.
2. The caller identity and capability token are valid.
3. The requested child is present in the caller’s `can_spawn` list.
4. The child supports MCP.
5. The task kind is allowed.
6. Task length and role goal/turn lifecycle are within compiled bounds; callers cannot supply a per-spawn timeout or task wall clock.
7. The requested sandbox does not exceed role permissions.
8. Required write scopes and attachments are valid and contained.
9. Maximum depth and ancestor-role policy are satisfied.
10. The child route is available for the current credential/endpoint overlay.
11. Active-agent target, caller child limit, role concurrency, global resource capacity, and writer capacity are available. Terminal workers release these leases for later sequential delegation.
12. No active writer in any session has an overlapping absolute canonical scope; nested session roots cannot bypass a lease.

Rejected work is recorded in the session audit log where appropriate.

## Resource groups

A resource group represents shared capacity rather than a hard-coded model limit. Examples include a subscription, provider rate pool, one local GPU, or a private endpoint.

```toml
[resources.local_gpu_0]
lock_key = "gpu:0"
max_active = 1
```

Agents consume `resource_units`; providers or models supply a default resource group. Capacity is enforced across sessions for MCP jobs and root sessions.

## Write isolation, disposition, and integration

MCP workspace writers require a Git repository and execute in isolated worktrees pinned to a captured base. `write_scope_required = true` requires explicit canonical scopes; setting it to `false` makes omission mean the whole delegated tree (`.`), never an unscoped write. Overlapping canonical leases are rejected across sessions, including nested workspace roots.

The runner captures text and binary-safe patch artifacts, base fingerprints, changed paths, file modes, hashes, and declared artifacts. A path outside the authorized scope, unsafe member, symlink/special artifact, or patch-boundary error fails the job and taints the session. The canonical checkout remains unchanged.

Successful results use a monotonic lifecycle. `agent_result` marks a terminal result read and returns either one complete strict JSON value or contiguous text pages. Callers begin at cursor zero and follow `next_cursor` until null. `agents_wait` returns compact lifecycle state and deterministic per-job progress revisions; supplying the exact observed revision map wakes on the first durable change, while bounded result previews are opt-in. Model-visible MCP responses remove supervisor result/event/stderr/socket locations, and generated guidance requires the lifecycle APIs instead of direct state-file reads. After transport recovery, generated guidance requires discovery and inspection of retained runs before replacement admission. The authorized ancestor must explicitly accept or reject a successfully completed result with a reason. Suspended jobs expose partial evidence without pretending it is final and may be continued. Typed provider-limit, transport, malformed-tool, and terminal-turn failures retain the complete upstream error plus partial evidence on that same lineage. If a dead turn-mode host left an active turn in persisted history, its replacement interrupts and settles that exact turn before starting at most one continuation; a terminal result that wins the recovery race remains authoritative. Stopped, failed, and cancelled jobs remain readable evidence but cannot be dispositioned as successful results. Only an accepted writable result may be passed to `agent_patch_integrate`, which rechecks patch identity, base fingerprints, scope, lease, and `git apply --check` before changing the canonical workspace. Reading is not acceptance, and acceptance is not integration.

There is no progress-checkpoint tool or model-authored liveness deadline. Durable app-server events are the progress record. An authorized controller inspects state or reads a filtered trace when a real dependency requires it; silence alone is not failure.

If publishing the integrated lifecycle state fails after `git apply`, the runtime attempts to reverse-apply the exact patch before returning the persistence error. If reverse-apply also fails, it durably taints the session when state storage remains writable and always reports that manual workspace recovery is required. If even the taint marker cannot be persisted, the error says so explicitly; the caller must stop using the session rather than assuming durable containment.

Native participants do not pass through this worktree/result supervisor. All featured bundled native participants are read-only; third-party profiles with native writers receive warnings and must rely on Codex sandboxing and manual review.

## Gateway lifecycle

Snapshots that use one or more `switchyard` providers receive a deterministic `routes.toml`. The gateway identity hashes only the selected clients, targets, and route semantics, so profiles with equivalent transport/model route sets can share one process without sharing their immutable profile snapshots. At runtime:

- The gateway bind setting is restricted to an unbracketed IPv4 or IPv6 loopback literal. Switchyard 0.2.0 accepts an IP address, not a hostname, and the generated ingress has no client-authentication layer.
- A port is selected deterministically from the configured range using the gateway hash.
- Credentials required by the snapshot are loaded from `credentials.env` or the process environment.
- Switchyard starts in a separate process group with declared provider secrets and the filtered non-credential environment.
- The gateway state records the exact executable version. Reuse requires that version to still match the configured binary; changing it recycles the gateway and its dependent root app-server so version-scoped transport behavior cannot drift underneath a durable thread.
- Health and route advertisement can be checked by `doctor --live`.
- Sessions whose snapshots have the same gateway hash reuse the gateway, even when their profile or snapshot hashes differ.
- Different gateway hashes use independent instances and may run concurrently.
- Per-session injected route faults are immutable supervisor admission overlays. They do not alter route discovery, restart the shared process, or affect another session using that gateway.
- `gateway stop-idle` uses side-effect-free validated state reads and retains an instance for every
  live root controller, root app-server, worker runner, or worker app-server, including detached and
  recoverable suspended hosts. It starts the configured idle interval at the latest canonical
  consumer release or gateway use timestamp; filesystem modification time is not a lifecycle clock.
- Route telemetry is interpreted at this integration boundary from Codex events and the exact
  Switchyard routing log. A worker records only the normalized observation.

A profile containing only built-in Codex or direct custom providers starts no gateway.

## Durable state

Default XDG paths:

```text
~/.config/codex-mmo/
  active-profile
  credentials.env
  settings.toml
  catalog.d/
  profiles.d/
  tool-mcp.d/

~/.local/state/codex-mmo/
  snapshots/
  sessions/
  jobs/
  gateways/
  evaluations/
  backups/
```

State uses atomic file replacement and a process-shared file lock for critical admissions. Append-only JSONL records hold an advisory lock across all short-write retries so concurrent records cannot interleave. Job, root, test, and gateway processes use process groups so cancellation and normal lifecycle completion retire descendants. State enumeration validates and returns records without reconciling processes or changing lifecycle fields; runtime-owned inspection/admission paths explicitly reconcile when that behavior is required.

Within a logical session, `session.json` is the canonical projection of the session and its one
immutable run. Exactly one `runs/<run-id>/run.json` mirrors run fields through the same locked
persistence operation; additional run inventory is invalid. Runtime admission/control code decides
session and admission transitions; the state module validates and publishes the exact records.
The root host updates root-owned lifecycle fields, and each isolated worker owns execution
transitions only for its own `jobs/<job-id>/metadata.json` plus append-only trace/audit artifacts.
Configuration and immutable snapshots are never used as mutable runtime state.

## Architectural invariants

- `mmo_runtime.py` owns admission and orchestration policy; providers and runners do not choose
  delegation, routing, trust, or resource policy.
- `mmo_state.py` owns the only canonical session/job publication functions and side-effect-free
  validated record readers. `mmo_runtime.py` owns stale-host reconciliation and all session lifecycle
  policy. Session/run mirror ordering remains explicit at the transition site because terminal
  retirement must mirror the active run before clearing its ID.
- One `WorkerRunner` owns each worker process's stop signal and app-server client. Root and worker
  action implementations remain separate because their lifecycle and native-thread responsibilities
  differ.
- Local control transport owns framing only. Authorization and compare-and-swap admission happen in
  the runtime before a runner executes an action.
- Provider-specific telemetry interpretation stays in the gateway/provider integration boundary;
  the scheduling kernel remains model- and provider-name agnostic.
- Immutable profile snapshots and generated homes are inputs to a session, never mutable runtime
  state or an alternate session registry.

## Architectural non-goals

Codex MMO is deliberately not a distributed scheduler, multi-tenant service, arbitrary workflow
language, general plugin runtime, provider catalog replacement, or persistence platform. Profile
packs are data, not executable extensions. Provider and Tool MCP seams exist because multiple real
integrations are supported; internal runtime components use direct calls rather than speculative
factories, dependency containers, event buses, or remote boundaries.

## Failure model

- A worker host that exits without a terminal update is marked `suspended`; partial evidence is materialized and the persisted thread may be continued.
- A root TUI/client disconnect detaches from the still-live app-server host; descendants and the root goal continue. A controller that exits with a recorded thread becomes recoverable, while a host that exits before any thread identity is recorded fails admission truthfully.
- Worker transport recovery resumes only its exact recorded thread. Root recovery resumes the current canonical generation or records a newer eligible top-level generation already created by the attached TUI; it never scans rollout files, searches outside the session-isolated app-server index, or substitutes a new MMO session/run. A replaced turn-mode worker interrupts and observes an orphaned in-progress turn before launching one continuation, while a concurrently completed turn remains deliverable.
- Resume restores a stopped content-addressed gateway before rewriting generated homes and reconciles dead current-run workers to suspended partial evidence before returning control.
- Graceful stop preserves results when agents finish in time and otherwise retains partials before bounded termination. Immediate cancellation remains distinct and is never described as graceful finalization.
- Provider startup failures preserve logs and a failed gateway record.
- Contract, command-correlation, artifact-correlation, and literal-evidence failures produce a failed job in strict mode and a warning in advisory mode.
- Unavailable optional worker routes produce typed admission failures without creating jobs; an unavailable root route blocks session creation.
- A completed isolated writer cannot affect the canonical workspace until explicit accepted integration succeeds.
- Output, completed messages, bounded empirical tool observations, and isolated patches are retained when available even without a valid final result.
- Profile changes cannot affect active snapshots.
- Live model inventory changes are reported, not silently interpreted.

## Performance model

The design favors bounded process isolation and durable correctness over an in-process agent framework. Native agents are available when lower launch overhead matters. MCP is used when enforcement and isolation have greater value. Profiles can combine both rather than forcing every task through the more expensive path.