codex-mmo/docs/TROUBLESHOOTING.md

Troubleshooting

Start with deterministic checks

From the source or extracted release directory:

make test
make validate

From an installed environment:

codex-mmo validate --all-profiles
codex-mmo tool-mcp validate
codex-mmo doctor --profile PROFILE

Add --live --probe only after local validation succeeds.

A command is rejected or produces unexpected output

Use command-local help and preserve stderr separately from requested data:

codex-mmo session list --help
codex-mmo session list >sessions.json 2>diagnostics.log
printf 'status=%s\n' "$?"

Structured commands use human output only when stdout is a terminal; a pipe or redirection selects strict JSON automatically. Use --json to force JSON on a terminal. Progress, warnings, usage, and errors stay on stderr, and --quiet suppresses only interactive stage progress. A syntax or invalid option combination returns status 2; an operation or validation failure returns status 1. Add --debug only when a traceback is useful for diagnosis.

Global --json, --quiet, and --debug options may appear around a control command but must occur before the explicit -- that begins Codex-owned arguments. Options are not abbreviated. For noninteractive root work, provide a positional prompt or pipe non-empty input; codex-mmo exec deliberately does not read an implicit prompt from a terminal. See Command-line interface for the complete output and exit-status contract.

The Codex TUI is monochrome, malformed, or does not resize

Confirm that version 8.0.0 and the exact supported Codex 0.149.0 client are installed:

codex-mmo version
codex --version

Start Codex MMO with both stdin and stdout attached to an interactive terminal rather than piping either through a log collector:

cd /path/to/repository
codex-mmo

Stderr may be redirected to a diagnostic file without disabling the interactive handoff. Interactive runs preserve the caller's foreground TTY and do not inject NO_COLOR. Check whether your shell already exports a color-disabling value:

printf 'TERM=%s COLORTERM=%s NO_COLOR=%s\n' "$TERM" "${COLORTERM-}" "${NO_COLOR-}"

Unset NO_COLOR only when you do not want that standard behavior:

unset NO_COLOR

The 8.0.0 acceptance suite exercises the remote Codex TUI through a pseudo-terminal while stderr is redirected, checks raw ANSI passthrough, transfers foreground process-group ownership, changes the terminal from 80x24 to 132x42, and requires the client to receive the resulting resize event while its app-server host remains independent.

Codex does not resume an MMO conversation

Use MMO's lifecycle command rather than forwarding Codex's subcommand through a new run:

codex-mmo resume --last
codex-mmo resume SESSION_ID
codex-mmo resume ROOT_CODEX_THREAD_ID
codex-mmo resume --last --all

codex-mmo run resume is rejected because run creates a deliberately new isolated session. A valid resume reattaches to the original session ID, one immutable run, snapshot, generated CODEX_HOME, app-server host, current root-thread generation, capabilities, budgets, and working-directory boundary. A controller or lost host process may be replaced, but never the MMO session/run identity. An older root thread ID remains a valid locator for the session and resolves to its current generation.

Inspect the lifecycle with:

codex-mmo session show SESSION_ID
codex-mmo session runs SESSION_ID
codex-mmo jobs list --run RUN_ID

Only canonical generation-8 sessions with exact package/profile version 8.0.0 in detached, paused, or suspended state are eligible. Other records are rejected without a compatibility reader. If startup exits before MMO records a thread ID, the session fails truthfully; resume never searches rollout files or the user's general Codex history. It may query thread/list only inside the pinned session-isolated root home to reconcile a top-level successor already created by the attached stock TUI. Resume restarts a stopped managed gateway before validating generated homes and suspends the same run if the pinned cwd, snapshot, executable, required root-route credential, capability document, or required MCP dependency cannot be restored. Missing optional worker credentials remain explicit degraded routes. Dead current-run workers become suspended with retained partial evidence before the root resumes. Terminal completed, failed, stopped, and cancelled sessions cannot be resumed. A tainted session requires --allow-tainted; delegation and patch-integration restrictions remain.

If a TUI context refresh appears blank or reopening lands before the refresh, inspect root_thread_generation, root_thread_lineage, and root_thread_transition in session show. MMO adopts a fresh thread only when it is persistent, top-level, in the pinned cwd, observed through the root host while the fingerprinted TUI is attached, and the prior root has no active turn. Native child and fork threads are deliberately excluded. Do not start a replacement MMO session or copy rollout files; close the TUI so the session becomes detached, then resume by the MMO session ID or any thread ID already recorded in its lineage.

Codex reports that model metadata was not found

A generated model such as mmo-<snapshot>-zai_coding_responses__glm_5_3 must appear verbatim in the session's models.json, and the same file must be referenced by model_catalog_json in that agent home's config.toml. Version 8.0.0 generates these files for every external model process.

Reinstall the current development build after clearing incompatible active state, then start a new session:

./install.sh
codex-mmo profile compile adaptive-engineering
codex-mmo

Inspect the new session with:

codex-mmo session list
codex-mmo session show SESSION_ID

The session record exposes each generated CODEX_HOME, model_catalog_json, and spawn- or control-capable role's orchestration_skill path. Verify that AGENTS.md and skills/mmo-profile-orchestration/SKILL.md match the immutable snapshot payload and that config.toml enables the exact SKILL.md path through skills.config. Verify that the selected model exactly matches one slug in models.json. Hybrid native/external homes must contain both the active Codex bundled rows and the generated external route rows. Do not work around a warning by accepting fallback metadata or copying guidance from another profile: either can change model behavior, context accounting, reasoning controls, compaction, tool shape, or delegation authority.

A profile does not compile

Run:

codex-mmo profile validate PROFILE
codex-mmo profile show PROFILE --resolved
codex-mmo catalog model ROUTE__MODEL_KEY

Common causes are a missing exact binding, unsupported modality, invalid reasoning level, unknown resource group, prohibited spawn edge, missing output contract, or a route that would lose required multimodal data.

If the preserved active-profile value names a retired bundled profile, select a current profile explicitly:

codex-mmo profile list
codex-mmo profile use adaptive-engineering

There is no retired-ID alias, older-generation reader, or automatic profile migration.

If the error names an undefined Tool MCP server, inspect the profile's per-agent grants and the operator registry:

codex-mmo tool-mcp list
codex-mmo tool-mcp show SERVER

Server definitions belong under ~/.config/codex-mmo/tool-mcp.d/*.toml; a profile intentionally cannot supply one.

A route or model is missing

Inspect the reviewed baseline and live account inventory:

codex-mmo catalog inventory
codex-mmo catalog models --inventory openrouter
codex-mmo catalog models --inventory opencode-zen
codex-mmo catalog models --inventory opencode-go
codex-mmo catalog models --inventory zai-api
codex-mmo catalog models --inventory openai-codex
codex-mmo catalog verify --remote --codex

Hosted inventories change. Refresh only after reviewing the report:

codex-mmo catalog refresh --install-codex-overlay

A discovered model ID is not automatically granted capabilities that the sources did not establish through that complete route.

Authentication fails

For built-in Codex/ChatGPT access:

codex-mmo auth status
codex-mmo auth login

For external routes, inspect ~/.config/codex-mmo/credentials.env and the selected profile's route bindings. Do not place credentials inside profile packs.

OpenCode Zen and Go use exactly OPENCODE_API_KEY; obsolete OpenCode credential names are not recognized or migrated. OpenRouter inference requires OPENROUTER_API_KEY, although its catalog-discovery endpoint can currently be queried publicly. Bundled GLM-5.3 roles use exactly ZAI_CODING_API_KEY; ZAI_API_KEY is a separate General API credential and never a fallback.

For Tool MCP authentication, run:

codex-mmo tool-mcp validate SERVER
codex-mmo doctor --profile PROFILE

The report names missing environment variables but never prints their values. Put the values in ~/.config/codex-mmo/credentials.env or the launch environment. Tool MCP schema version 8 deliberately does not inherit Codex's .credentials.json OAuth sessions.

A Tool MCP server is missing or fails to start

First separate Tool MCP from the internal Agent MCP server. mmo_mesh launches profile participants; operator-defined servers provide third-party tools.

codex-mmo tool-mcp show SERVER
codex-mmo tool-mcp validate SERVER
codex-mmo profile show PROFILE --resolved
codex-mmo doctor --profile PROFILE

For STDIO, verify the resolved executable and configured working directory. For Streamable HTTP, verify the URL and environment-backed authentication. validate is intentionally non-launching, so a successful readiness result does not prove protocol compatibility or downstream authorization. Start a new session after changing a registry definition because existing sessions remain pinned to their compiled snapshot.

Native roles share their containing Codex process. The generated root config contains the union of servers required by reachable native roles, while each role layer disables servers and tools it was not granted. This preserves tool visibility but not credential isolation between in-process native roles; use an Agent-MCP participant when that distinction matters.

Switchyard does not start

Check whether the selected profile actually requires it:

codex-mmo gateway status PROFILE
codex-mmo gateway logs PROFILE
codex-mmo doctor --profile PROFILE --live

Built-in-only profiles do not start Switchyard. For translated routes, verify the Switchyard binary, generated route configuration, endpoint reachability, and route credential.

A spawn is rejected for capacity or scope

Inspect resource and scope admission:

codex-mmo session show SESSION_ID
codex-mmo jobs list --session SESSION_ID
codex-mmo jobs status JOB_ID

Admission rejects a batch immediately when route capacity, a single-slot local GPU group, the per-role active limit, the session active-agent limit, or a writable scope is unavailable; it does not maintain a capacity queue. A terminal worker releases these leases, so sequential follow-up work does not exhaust a lifetime quota. A queued job is only in the brief accepted-before-runner-start transition. Reduce concurrency, wait for the conflicting lease to finish, cancel obsolete work, or divide write ownership into disjoint scopes. If an accepted job remains queued, inspect its runner log and reconcile its recorded runner PID.

A worker cannot spawn another agent

Nested delegation is allowed only when the profile grants the edge. Check the caller's can_spawn list, maximum depth, active child limit, ancestor-role policy, active resource capacity, and backend availability. Leaf and low-trust agents intentionally receive no delegation authority.

A worker result is rejected

MCP workers may be required to satisfy a JSON output contract. Inspect:

codex-mmo jobs result JOB_ID
codex-mmo jobs result JOB_ID --cursor NEXT_CURSOR
codex-mmo jobs status JOB_ID

A failed strict result normally indicates invalid JSON, a missing/unexpected field, an invalid URI/date, command evidence absent from captured events, an artifact hash/path mismatch, or a role-specific literal-evidence mismatch. agents_wait returns compact state and progress revisions; terminal previews appear only when explicitly requested. Start agent_result or jobs result at cursor zero and follow each next_cursor until null; text pages are contiguous and lossless. When agent_trace reports a truncated valid event, use its record_cursor with agent_trace_record and follow next_cursor to reconstruct the exact filtered record. Malformed trace records expose only a bounded diagnostic because they cannot be structurally filtered. Do not open MMO's job result, event, stderr, or socket paths directly from an agent—use the result, inspect, and trace APIs. A result that validated may still be explicitly rejected by the root after review. Reading a result does not accept it, and accepting a writer result does not apply its patch until agent_patch_integrate succeeds.

Strict Agent-MCP roles use app-server turn/start.outputSchema when the selected model supports structured output, then validate the complete original contract independently. A malformed first response receives one shape-only repair turn on the same durable thread. There is no JSON-fixer agent and no repair is allowed to invent absent evidence.

A long-running root or worker is quiet, paused, or suspended

Do not ask the model to estimate elapsed time. Inspect the host-owned state and filtered trace:

codex-mmo jobs status JOB_ID
codex-mmo jobs result JOB_ID
codex-mmo session show SESSION_ID

stall_warning_seconds is diagnostic only. It never interrupts a quiet model. Turn-mode roles have no profile task deadline; goal-mode roles use Codex's cumulative token accounting and remain recoverable when the current budget is exhausted. An authorized controller can inspect/trace, pause, continue with an optional larger total budget within the compiled ceiling, request finalization, detach, or fully stop. A worker client/transport failure reconnects to its exact host/thread; root recovery uses the current canonical thread generation. Unrecovered host loss produces suspended, not a missing result. A replacement turn-mode worker first settles any orphaned active turn; it starts one continuation only after an interrupted outcome and preserves a result that completed during the race. stopped, failed, and cancelled remain terminal but retain readable partial evidence, terminal history, and patches.

Do not try to fix a slow model by adding a model-authored “work for N seconds” instruction. The provider stream_idle_timeout_ms, app-server initialization/lifecycle request limits, Tool MCP call limits, goal token ceilings, and external smoke/evaluation wall limits are different boundaries. Direct provider idle values must be positive and should be sized for the route's real latency rather than a model instruction. App-server lifecycle waits cover slow MCP startup and thread operations but do not time model turns. Tool calls may retain their own safety bounds; the surrounding thread and evidence survive client detach, transport restart, or interruption.

For a noninteractive root, exec --wall-timeout is only a caller/harness boundary. It detaches without interrupting the app-server goal; use codex-mmo resume SESSION_ID to attach to the same work. session detach also leaves root and workers running. session pause checkpoints partial evidence, interrupts admitted turns, retires root and supervised-worker hosts, and releases their capacity. session continue freshly admits and reactivates the same root thread plus the exact paused worker set. Run session compact only from that cold-paused state; it temporarily hosts the root, compacts the same thread, and cold-pauses it again. During bootstrap use session stop or session cancel when no durable thread exists yet. session stop --grace SECONDS requests finalization and then retires all hosts; session cancel is immediate.

A worker is waiting for input or approval

approval_policy = "never" declines escalation automatically inside the role's compiled sandbox. With untrusted or on-request, an authorized controller sees the raw pending method and request ID through agent_inspect and must send that method's exact response object with agent_respond. The request and thread remain durable while pending. Do not send a generic approval shape: command, file-change, permission, user-input, and MCP-elicitation methods have different generated app-server response schemas.

If a mutating control returns a transport error after delivery began, inspect the job before retrying. MMO records that revision as delivery_unknown, because the worker may have applied the request before its response was lost. A blind retry could steer, interrupt, finalize, or fork the thread twice. Controls are serialized per job, but serialization cannot turn an ambiguous network outcome into proof of non-application.

A job changed files outside its scope

Writable MCP jobs run in isolated Git worktrees, so they do not edit the canonical checkout directly. An out-of-scope or unsafe patch boundary is a hard job failure that taints the session; inspect the job metadata and isolated artifact. Do not integrate it. Narrow the assignment or use a read-only reviewer, then launch a fresh bounded job if still justified.

Cancellation appears incomplete

Cancel the root session to cascade through descendants:

codex-mmo session cancel SESSION_ID

For one branch:

codex-mmo jobs cancel JOB_ID --reason "superseded"

The supervisor terminates worker process groups and records terminal state. If a process remains, inspect the job stderr and verify that the process was launched by the current installed runtime rather than an unrelated manual command.

Native agents and MCP agents disagree

Do not vote or average their conclusions. Resolve the conflict from primary evidence: source code, commands, logs, tests, specifications, or a reproducible external observation. Use a read-only reviewer when an independent check is useful.

Release archives look incomplete

Verify checksums and member counts before installation:

sha256sum -c codex-multimodel-orchestrator-8.0.0-SHA256SUMS.txt
tar -tzf codex-multimodel-orchestrator-8.0.0-linux.tar.gz | head
unzip -Z1 codex-multimodel-orchestrator-8.0.0-linux.zip | head

Each archive must contain the top-level package directory, runtime modules, profiles, configuration, documentation, tests, installer, and PACKAGE-MANIFEST.json. The release builder rejects a truncated source tree and verifies member hashes before reporting success.

Collecting diagnostics

Preserve the following when investigating a failure:

• The profile ID and snapshot hash.
• codex-mmo doctor output.
• Session and job IDs.
• Job stderr and event logs.
• Gateway logs for translated providers.
• Route, serving-provider, HTTP status, policy, and retry information.
• The exact repository commit and dirty state.
• The installed Codex and Switchyard versions.

Remove credentials before sharing diagnostics.