# Troubleshooting ## Start with deterministic checks From the source or extracted release directory: ```bash make test make validate ``` From an installed environment: ```bash codex-mmo validate --all-profiles codex-mmo tool-mcp validate codex-mmo doctor --profile PROFILE ``` Add `--live --probe` only after local validation succeeds. ## A command is rejected or produces unexpected output Use command-local help and preserve stderr separately from requested data: ```bash codex-mmo session list --help codex-mmo session list >sessions.json 2>diagnostics.log printf 'status=%s\n' "$?" ``` Structured commands use human output only when stdout is a terminal; a pipe or redirection selects strict JSON automatically. Use `--json` to force JSON on a terminal. Progress, warnings, usage, and errors stay on stderr, and `--quiet` suppresses only interactive stage progress. A syntax or invalid option combination returns status 2; an operation or validation failure returns status 1. Add `--debug` only when a traceback is useful for diagnosis. Global `--json`, `--quiet`, and `--debug` options may appear around a control command but must occur before the explicit `--` that begins Codex-owned arguments. Options are not abbreviated. For noninteractive root work, provide a positional prompt or pipe non-empty input; `codex-mmo exec` deliberately does not read an implicit prompt from a terminal. See [Command-line interface](CLI.md) for the complete output and exit-status contract. ## The Codex TUI is monochrome, malformed, or does not resize Confirm that version 8.0.0 and the exact supported Codex 0.149.0 client are installed: ```bash codex-mmo version codex --version ``` Start Codex MMO with both stdin and stdout attached to an interactive terminal rather than piping either through a log collector: ```bash cd /path/to/repository codex-mmo ``` Stderr may be redirected to a diagnostic file without disabling the interactive handoff. Interactive runs preserve the caller's foreground TTY and do not inject `NO_COLOR`. Check whether your shell already exports a color-disabling value: ```bash printf 'TERM=%s COLORTERM=%s NO_COLOR=%s\n' "$TERM" "${COLORTERM-}" "${NO_COLOR-}" ``` Unset `NO_COLOR` only when you do not want that standard behavior: ```bash unset NO_COLOR ``` The 8.0.0 acceptance suite exercises the remote Codex TUI through a pseudo-terminal while stderr is redirected, checks raw ANSI passthrough, transfers foreground process-group ownership, changes the terminal from 80x24 to 132x42, and requires the client to receive the resulting resize event while its app-server host remains independent. ## Codex does not resume an MMO conversation Use MMO's lifecycle command rather than forwarding Codex's subcommand through a new run: ```bash codex-mmo resume --last codex-mmo resume SESSION_ID codex-mmo resume ROOT_CODEX_THREAD_ID codex-mmo resume --last --all ``` `codex-mmo run resume` is rejected because `run` creates a deliberately new isolated session. A valid resume reattaches to the original session ID, one immutable run, snapshot, generated `CODEX_HOME`, app-server host, current root-thread generation, capabilities, budgets, and working-directory boundary. A controller or lost host process may be replaced, but never the MMO session/run identity. An older root thread ID remains a valid locator for the session and resolves to its current generation. Inspect the lifecycle with: ```bash codex-mmo session show SESSION_ID codex-mmo session runs SESSION_ID codex-mmo jobs list --run RUN_ID ``` Only canonical generation-8 sessions with exact package/profile version 8.0.0 in `detached`, `paused`, or `suspended` state are eligible. Other records are rejected without a compatibility reader. If startup exits before MMO records a thread ID, the session fails truthfully; resume never searches rollout files or the user's general Codex history. It may query `thread/list` only inside the pinned session-isolated root home to reconcile a top-level successor already created by the attached stock TUI. Resume restarts a stopped managed gateway before validating generated homes and suspends the same run if the pinned cwd, snapshot, executable, required root-route credential, capability document, or required MCP dependency cannot be restored. Missing optional worker credentials remain explicit degraded routes. Dead current-run workers become `suspended` with retained partial evidence before the root resumes. Terminal `completed`, `failed`, `stopped`, and `cancelled` sessions cannot be resumed. A tainted session requires `--allow-tainted`; delegation and patch-integration restrictions remain. If a TUI context refresh appears blank or reopening lands before the refresh, inspect `root_thread_generation`, `root_thread_lineage`, and `root_thread_transition` in `session show`. MMO adopts a fresh thread only when it is persistent, top-level, in the pinned cwd, observed through the root host while the fingerprinted TUI is attached, and the prior root has no active turn. Native child and fork threads are deliberately excluded. Do not start a replacement MMO session or copy rollout files; close the TUI so the session becomes detached, then resume by the MMO session ID or any thread ID already recorded in its lineage. ## Codex reports that model metadata was not found A generated model such as `mmo--zai_coding_responses__glm_5_3` must appear verbatim in the session's `models.json`, and the same file must be referenced by `model_catalog_json` in that agent home's `config.toml`. Version 8.0.0 generates these files for every external model process. Reinstall the current development build after clearing incompatible active state, then start a **new** session: ```bash ./install.sh codex-mmo profile compile adaptive-engineering codex-mmo ``` Inspect the new session with: ```bash codex-mmo session list codex-mmo session show SESSION_ID ``` The session record exposes each generated `CODEX_HOME`, `model_catalog_json`, and spawn- or control-capable role's `orchestration_skill` path. Verify that `AGENTS.md` and `skills/mmo-profile-orchestration/SKILL.md` match the immutable snapshot payload and that `config.toml` enables the exact `SKILL.md` path through `skills.config`. Verify that the selected `model` exactly matches one `slug` in `models.json`. Hybrid native/external homes must contain both the active Codex bundled rows and the generated external route rows. Do not work around a warning by accepting fallback metadata or copying guidance from another profile: either can change model behavior, context accounting, reasoning controls, compaction, tool shape, or delegation authority. ## A profile does not compile Run: ```bash codex-mmo profile validate PROFILE codex-mmo profile show PROFILE --resolved codex-mmo catalog model ROUTE__MODEL_KEY ``` Common causes are a missing exact binding, unsupported modality, invalid reasoning level, unknown resource group, prohibited spawn edge, missing output contract, or a route that would lose required multimodal data. If the preserved `active-profile` value names a retired bundled profile, select a current profile explicitly: ```bash codex-mmo profile list codex-mmo profile use adaptive-engineering ``` There is no retired-ID alias, older-generation reader, or automatic profile migration. If the error names an undefined Tool MCP server, inspect the profile's per-agent grants and the operator registry: ```bash codex-mmo tool-mcp list codex-mmo tool-mcp show SERVER ``` Server definitions belong under `~/.config/codex-mmo/tool-mcp.d/*.toml`; a profile intentionally cannot supply one. ## A route or model is missing Inspect the reviewed baseline and live account inventory: ```bash codex-mmo catalog inventory codex-mmo catalog models --inventory openrouter codex-mmo catalog models --inventory opencode-zen codex-mmo catalog models --inventory opencode-go codex-mmo catalog models --inventory zai-api codex-mmo catalog models --inventory openai-codex codex-mmo catalog verify --remote --codex ``` Hosted inventories change. Refresh only after reviewing the report: ```bash codex-mmo catalog refresh --install-codex-overlay ``` A discovered model ID is not automatically granted capabilities that the sources did not establish through that complete route. ## Authentication fails For built-in Codex/ChatGPT access: ```bash codex-mmo auth status codex-mmo auth login ``` For external routes, inspect `~/.config/codex-mmo/credentials.env` and the selected profile's route bindings. Do not place credentials inside profile packs. OpenCode Zen and Go use exactly `OPENCODE_API_KEY`; obsolete OpenCode credential names are not recognized or migrated. OpenRouter inference requires `OPENROUTER_API_KEY`, although its catalog-discovery endpoint can currently be queried publicly. Bundled GLM-5.3 roles use exactly `ZAI_CODING_API_KEY`; `ZAI_API_KEY` is a separate General API credential and never a fallback. For Tool MCP authentication, run: ```bash codex-mmo tool-mcp validate SERVER codex-mmo doctor --profile PROFILE ``` The report names missing environment variables but never prints their values. Put the values in `~/.config/codex-mmo/credentials.env` or the launch environment. Tool MCP schema version 8 deliberately does not inherit Codex's `.credentials.json` OAuth sessions. ## A Tool MCP server is missing or fails to start First separate Tool MCP from the internal Agent MCP server. `mmo_mesh` launches profile participants; operator-defined servers provide third-party tools. ```bash codex-mmo tool-mcp show SERVER codex-mmo tool-mcp validate SERVER codex-mmo profile show PROFILE --resolved codex-mmo doctor --profile PROFILE ``` For STDIO, verify the resolved executable and configured working directory. For Streamable HTTP, verify the URL and environment-backed authentication. `validate` is intentionally non-launching, so a successful readiness result does not prove protocol compatibility or downstream authorization. Start a new session after changing a registry definition because existing sessions remain pinned to their compiled snapshot. Native roles share their containing Codex process. The generated root config contains the union of servers required by reachable native roles, while each role layer disables servers and tools it was not granted. This preserves tool visibility but not credential isolation between in-process native roles; use an Agent-MCP participant when that distinction matters. ## Switchyard does not start Check whether the selected profile actually requires it: ```bash codex-mmo gateway status PROFILE codex-mmo gateway logs PROFILE codex-mmo doctor --profile PROFILE --live ``` Built-in-only profiles do not start Switchyard. For translated routes, verify the Switchyard binary, generated route configuration, endpoint reachability, and route credential. ## A spawn is rejected for capacity or scope Inspect resource and scope admission: ```bash codex-mmo session show SESSION_ID codex-mmo jobs list --session SESSION_ID codex-mmo jobs status JOB_ID ``` Admission rejects a batch immediately when route capacity, a single-slot local GPU group, the per-role active limit, the session active-agent limit, or a writable scope is unavailable; it does not maintain a capacity queue. A terminal worker releases these leases, so sequential follow-up work does not exhaust a lifetime quota. A `queued` job is only in the brief accepted-before-runner-start transition. Reduce concurrency, wait for the conflicting lease to finish, cancel obsolete work, or divide write ownership into disjoint scopes. If an accepted job remains `queued`, inspect its runner log and reconcile its recorded runner PID. ## A worker cannot spawn another agent Nested delegation is allowed only when the profile grants the edge. Check the caller's `can_spawn` list, maximum depth, active child limit, ancestor-role policy, active resource capacity, and backend availability. Leaf and low-trust agents intentionally receive no delegation authority. ## A worker result is rejected MCP workers may be required to satisfy a JSON output contract. Inspect: ```bash codex-mmo jobs result JOB_ID codex-mmo jobs result JOB_ID --cursor NEXT_CURSOR codex-mmo jobs status JOB_ID ``` A failed strict result normally indicates invalid JSON, a missing/unexpected field, an invalid URI/date, command evidence absent from captured events, an artifact hash/path mismatch, or a role-specific literal-evidence mismatch. `agents_wait` returns compact state and progress revisions; terminal previews appear only when explicitly requested. Start `agent_result` or `jobs result` at cursor zero and follow each `next_cursor` until null; text pages are contiguous and lossless. When `agent_trace` reports a truncated valid event, use its `record_cursor` with `agent_trace_record` and follow `next_cursor` to reconstruct the exact filtered record. Malformed trace records expose only a bounded diagnostic because they cannot be structurally filtered. Do not open MMO's job result, event, stderr, or socket paths directly from an agent—use the result, inspect, and trace APIs. A result that validated may still be explicitly rejected by the root after review. Reading a result does not accept it, and accepting a writer result does not apply its patch until `agent_patch_integrate` succeeds. Strict Agent-MCP roles use app-server `turn/start.outputSchema` when the selected model supports structured output, then validate the complete original contract independently. A malformed first response receives one shape-only repair turn on the same durable thread. There is no JSON-fixer agent and no repair is allowed to invent absent evidence. ## A long-running root or worker is quiet, paused, or suspended Do not ask the model to estimate elapsed time. Inspect the host-owned state and filtered trace: ```bash codex-mmo jobs status JOB_ID codex-mmo jobs result JOB_ID codex-mmo session show SESSION_ID ``` `stall_warning_seconds` is diagnostic only. It never interrupts a quiet model. Turn-mode roles have no profile task deadline; goal-mode roles use Codex's cumulative token accounting and remain recoverable when the current budget is exhausted. An authorized controller can inspect/trace, pause, continue with an optional larger total budget within the compiled ceiling, request finalization, detach, or fully stop. A worker client/transport failure reconnects to its exact host/thread; root recovery uses the current canonical thread generation. Unrecovered host loss produces `suspended`, not a missing result. A replacement turn-mode worker first settles any orphaned active turn; it starts one continuation only after an interrupted outcome and preserves a result that completed during the race. `stopped`, `failed`, and `cancelled` remain terminal but retain readable partial evidence, terminal history, and patches. Do not try to fix a slow model by adding a model-authored “work for N seconds” instruction. The provider `stream_idle_timeout_ms`, app-server initialization/lifecycle request limits, Tool MCP call limits, goal token ceilings, and external smoke/evaluation wall limits are different boundaries. Direct provider idle values must be positive and should be sized for the route's real latency rather than a model instruction. App-server lifecycle waits cover slow MCP startup and thread operations but do not time model turns. Tool calls may retain their own safety bounds; the surrounding thread and evidence survive client detach, transport restart, or interruption. For a noninteractive root, `exec --wall-timeout` is only a caller/harness boundary. It detaches without interrupting the app-server goal; use `codex-mmo resume SESSION_ID` to attach to the same work. `session detach` also leaves root and workers running. `session pause` checkpoints partial evidence, interrupts admitted turns, retires root and supervised-worker hosts, and releases their capacity. `session continue` freshly admits and reactivates the same root thread plus the exact paused worker set. Run `session compact` only from that cold-paused state; it temporarily hosts the root, compacts the same thread, and cold-pauses it again. During bootstrap use `session stop` or `session cancel` when no durable thread exists yet. `session stop --grace SECONDS` requests finalization and then retires all hosts; `session cancel` is immediate. ## A worker is waiting for input or approval `approval_policy = "never"` declines escalation automatically inside the role's compiled sandbox. With `untrusted` or `on-request`, an authorized controller sees the raw pending method and request ID through `agent_inspect` and must send that method's exact response object with `agent_respond`. The request and thread remain durable while pending. Do not send a generic approval shape: command, file-change, permission, user-input, and MCP-elicitation methods have different generated app-server response schemas. If a mutating control returns a transport error after delivery began, inspect the job before retrying. MMO records that revision as `delivery_unknown`, because the worker may have applied the request before its response was lost. A blind retry could steer, interrupt, finalize, or fork the thread twice. Controls are serialized per job, but serialization cannot turn an ambiguous network outcome into proof of non-application. ## A job changed files outside its scope Writable MCP jobs run in isolated Git worktrees, so they do not edit the canonical checkout directly. An out-of-scope or unsafe patch boundary is a hard job failure that taints the session; inspect the job metadata and isolated artifact. Do not integrate it. Narrow the assignment or use a read-only reviewer, then launch a fresh bounded job if still justified. ## Cancellation appears incomplete Cancel the root session to cascade through descendants: ```bash codex-mmo session cancel SESSION_ID ``` For one branch: ```bash codex-mmo jobs cancel JOB_ID --reason "superseded" ``` The supervisor terminates worker process groups and records terminal state. If a process remains, inspect the job stderr and verify that the process was launched by the current installed runtime rather than an unrelated manual command. ## Native agents and MCP agents disagree Do not vote or average their conclusions. Resolve the conflict from primary evidence: source code, commands, logs, tests, specifications, or a reproducible external observation. Use a read-only reviewer when an independent check is useful. ## Release archives look incomplete Verify checksums and member counts before installation: ```bash sha256sum -c codex-multimodel-orchestrator-8.0.0-SHA256SUMS.txt tar -tzf codex-multimodel-orchestrator-8.0.0-linux.tar.gz | head unzip -Z1 codex-multimodel-orchestrator-8.0.0-linux.zip | head ``` Each archive must contain the top-level package directory, runtime modules, profiles, configuration, documentation, tests, installer, and `PACKAGE-MANIFEST.json`. The release builder rejects a truncated source tree and verifies member hashes before reporting success. ## Collecting diagnostics Preserve the following when investigating a failure: - The profile ID and snapshot hash. - `codex-mmo doctor` output. - Session and job IDs. - Job stderr and event logs. - Gateway logs for translated providers. - Route, serving-provider, HTTP status, policy, and retry information. - The exact repository commit and dirty state. - The installed Codex and Switchyard versions. Remove credentials before sharing diagnostics.