codex-mmo/docs/TROUBLESHOOTING.md

# Troubleshooting

## Start with deterministic checks

From the source or extracted release directory:

```bash
make test
make validate
```

From an installed environment:

```bash
codex-mmo validate --all-profiles
codex-mmo tool-mcp validate
codex-mmo doctor --profile PROFILE
```

Add `--live --probe` only after local validation succeeds.

## A command is rejected or produces unexpected output

Use command-local help and preserve stderr separately from requested data:

```bash
codex-mmo session list --help
codex-mmo session list >sessions.json 2>diagnostics.log
printf 'status=%s\n' "$?"
```

Structured commands use human output only when stdout is a terminal; a pipe or redirection selects
strict JSON automatically. Use `--json` to force JSON on a terminal. Progress, warnings, usage, and
errors stay on stderr, and `--quiet` suppresses only interactive stage progress. A syntax or invalid
option combination returns status 2; an operation or validation failure returns status 1. Add
`--debug` only when a traceback is useful for diagnosis.

Global `--json`, `--quiet`, and `--debug` options may appear around a control command but must occur
before the explicit `--` that begins Codex-owned arguments. Options are not abbreviated. For
noninteractive root work, provide a positional prompt or pipe non-empty input; `codex-mmo exec`
deliberately does not read an implicit prompt from a terminal. See [Command-line interface](CLI.md)
for the complete output and exit-status contract.

## The Codex TUI is monochrome, malformed, or does not resize

Confirm that version 8.0.0 and the exact supported Codex 0.149.0 client are installed:

```bash
codex-mmo version
codex --version
```

Start Codex MMO with both stdin and stdout attached to an interactive terminal rather than piping either through a log collector:

```bash
cd /path/to/repository
codex-mmo
```

Stderr may be redirected to a diagnostic file without disabling the interactive handoff. Interactive runs preserve the caller's foreground TTY and do not inject `NO_COLOR`. Check whether your shell already exports a color-disabling value:

```bash
printf 'TERM=%s COLORTERM=%s NO_COLOR=%s\n' "$TERM" "${COLORTERM-}" "${NO_COLOR-}"
```

Unset `NO_COLOR` only when you do not want that standard behavior:

```bash
unset NO_COLOR
```

The 8.0.0 acceptance suite exercises the remote Codex TUI through a pseudo-terminal while stderr is redirected, checks raw ANSI passthrough, transfers foreground process-group ownership, changes the terminal from 80x24 to 132x42, and requires the client to receive the resulting resize event while its app-server host remains independent.

## Codex does not resume an MMO conversation

Use MMO's lifecycle command rather than forwarding Codex's subcommand through a new run:

```bash
codex-mmo resume --last
codex-mmo resume SESSION_ID
codex-mmo resume ROOT_CODEX_THREAD_ID
codex-mmo resume --last --all
```

`codex-mmo run resume` is rejected because `run` creates a deliberately new isolated session. A valid resume reattaches to the original session ID, one immutable run, snapshot, generated `CODEX_HOME`, app-server host, current root-thread generation, capabilities, budgets, and working-directory boundary. A controller or lost host process may be replaced, but never the MMO session/run identity. An older root thread ID remains a valid locator for the session and resolves to its current generation.

Inspect the lifecycle with:

```bash
codex-mmo session show SESSION_ID
codex-mmo session runs SESSION_ID
codex-mmo jobs list --run RUN_ID
```

Only canonical generation-8 sessions with exact package/profile version 8.0.0 in `detached`,
`paused`, or `suspended` state are eligible. Other records are rejected without a compatibility
reader. If startup exits before
MMO records a thread ID, the session fails truthfully; resume never searches rollout files or the
user's general Codex history. It may query `thread/list` only inside the pinned session-isolated
root home to reconcile a top-level successor already created by the attached stock TUI. Resume
restarts a stopped managed gateway before validating generated homes and suspends the same run if
the pinned cwd, snapshot, executable, required root-route credential, capability document, or
required MCP dependency cannot be restored. Missing optional worker credentials remain explicit
degraded routes. Dead current-run workers become `suspended` with retained partial evidence before
the root resumes. Terminal `completed`, `failed`, `stopped`, and `cancelled` sessions cannot be resumed.
A tainted session requires `--allow-tainted`; delegation and patch-integration restrictions remain.

If a TUI context refresh appears blank or reopening lands before the refresh, inspect
`root_thread_generation`, `root_thread_lineage`, and `root_thread_transition` in `session show`.
MMO adopts a fresh thread only when it is persistent, top-level, in the pinned cwd, observed through
the root host while the fingerprinted TUI is attached, and the prior root has no active turn. Native
child and fork threads are deliberately excluded. Do not start a replacement MMO session or copy
rollout files; close the TUI so the session becomes detached, then resume by the MMO session ID or
any thread ID already recorded in its lineage.

## Codex reports that model metadata was not found

A generated model such as `mmo-<snapshot>-zai_coding_responses__glm_5_3` must appear verbatim in the session's `models.json`, and the same file must be referenced by `model_catalog_json` in that agent home's `config.toml`. Version 8.0.0 generates these files for every external model process.

Reinstall the current development build after clearing incompatible active state, then start a **new** session:

```bash
./install.sh
codex-mmo profile compile adaptive-engineering
codex-mmo
```

Inspect the new session with:

```bash
codex-mmo session list
codex-mmo session show SESSION_ID
```

The session record exposes each generated `CODEX_HOME`, `model_catalog_json`, and spawn- or control-capable role's `orchestration_skill` path. Verify that `AGENTS.md` and `skills/mmo-profile-orchestration/SKILL.md` match the immutable snapshot payload and that `config.toml` enables the exact `SKILL.md` path through `skills.config`. Verify that the selected `model` exactly matches one `slug` in `models.json`. Hybrid native/external homes must contain both the active Codex bundled rows and the generated external route rows. Do not work around a warning by accepting fallback metadata or copying guidance from another profile: either can change model behavior, context accounting, reasoning controls, compaction, tool shape, or delegation authority.

## A profile does not compile

Run:

```bash
codex-mmo profile validate PROFILE
codex-mmo profile show PROFILE --resolved
codex-mmo catalog model ROUTE__MODEL_KEY
```

Common causes are a missing exact binding, unsupported modality, invalid reasoning level, unknown resource group, prohibited spawn edge, missing output contract, or a route that would lose required multimodal data.

If the preserved `active-profile` value names a retired bundled profile, select a current profile explicitly:

```bash
codex-mmo profile list
codex-mmo profile use adaptive-engineering
```

There is no retired-ID alias, older-generation reader, or automatic profile migration.

If the error names an undefined Tool MCP server, inspect the profile's per-agent grants and the operator registry:

```bash
codex-mmo tool-mcp list
codex-mmo tool-mcp show SERVER
```

Server definitions belong under `~/.config/codex-mmo/tool-mcp.d/*.toml`; a profile intentionally cannot supply one.

## A route or model is missing

Inspect the reviewed baseline and live account inventory:

```bash
codex-mmo catalog inventory
codex-mmo catalog models --inventory openrouter
codex-mmo catalog models --inventory opencode-zen
codex-mmo catalog models --inventory opencode-go
codex-mmo catalog models --inventory zai-api
codex-mmo catalog models --inventory openai-codex
codex-mmo catalog verify --remote --codex
```

Hosted inventories change. Refresh only after reviewing the report:

```bash
codex-mmo catalog refresh --install-codex-overlay
```

A discovered model ID is not automatically granted capabilities that the sources did not establish through that complete route.

## Authentication fails

For built-in Codex/ChatGPT access:

```bash
codex-mmo auth status
codex-mmo auth login
```

For external routes, inspect `~/.config/codex-mmo/credentials.env` and the selected profile's route bindings. Do not place credentials inside profile packs.

OpenCode Zen and Go use exactly `OPENCODE_API_KEY`; obsolete OpenCode credential names are not recognized or migrated. OpenRouter inference requires `OPENROUTER_API_KEY`, although its catalog-discovery endpoint can currently be queried publicly. Bundled GLM-5.3 roles use exactly `ZAI_CODING_API_KEY`; `ZAI_API_KEY` is a separate General API credential and never a fallback.

For Tool MCP authentication, run:

```bash
codex-mmo tool-mcp validate SERVER
codex-mmo doctor --profile PROFILE
```

The report names missing environment variables but never prints their values. Put the values in `~/.config/codex-mmo/credentials.env` or the launch environment. Tool MCP schema version 8 deliberately does not inherit Codex's `.credentials.json` OAuth sessions.

## A Tool MCP server is missing or fails to start

First separate Tool MCP from the internal Agent MCP server. `mmo_mesh` launches profile participants; operator-defined servers provide third-party tools.

```bash
codex-mmo tool-mcp show SERVER
codex-mmo tool-mcp validate SERVER
codex-mmo profile show PROFILE --resolved
codex-mmo doctor --profile PROFILE
```

For STDIO, verify the resolved executable and configured working directory. For Streamable HTTP, verify the URL and environment-backed authentication. `validate` is intentionally non-launching, so a successful readiness result does not prove protocol compatibility or downstream authorization. Start a new session after changing a registry definition because existing sessions remain pinned to their compiled snapshot.

Native roles share their containing Codex process. The generated root config contains the union of servers required by reachable native roles, while each role layer disables servers and tools it was not granted. This preserves tool visibility but not credential isolation between in-process native roles; use an Agent-MCP participant when that distinction matters.

## Switchyard does not start

Check whether the selected profile actually requires it:

```bash
codex-mmo gateway status PROFILE
codex-mmo gateway logs PROFILE
codex-mmo doctor --profile PROFILE --live
```

Built-in-only profiles do not start Switchyard. For translated routes, verify the Switchyard binary, generated route configuration, endpoint reachability, and route credential.

## A spawn is rejected for capacity or scope

Inspect resource and scope admission:

```bash
codex-mmo session show SESSION_ID
codex-mmo jobs list --session SESSION_ID
codex-mmo jobs status JOB_ID
```

Admission rejects a batch immediately when route capacity, a single-slot local GPU group, the per-role active limit, the session active-agent limit, or a writable scope is unavailable; it does not maintain a capacity queue. A terminal worker releases these leases, so sequential follow-up work does not exhaust a lifetime quota. A `queued` job is only in the brief accepted-before-runner-start transition. Reduce concurrency, wait for the conflicting lease to finish, cancel obsolete work, or divide write ownership into disjoint scopes. If an accepted job remains `queued`, inspect its runner log and reconcile its recorded runner PID.

## A worker cannot spawn another agent

Nested delegation is allowed only when the profile grants the edge. Check the caller's `can_spawn` list, maximum depth, active child limit, ancestor-role policy, active resource capacity, and backend availability. Leaf and low-trust agents intentionally receive no delegation authority.

## A worker result is rejected

MCP workers may be required to satisfy a JSON output contract. Inspect:

```bash
codex-mmo jobs result JOB_ID
codex-mmo jobs result JOB_ID --cursor NEXT_CURSOR
codex-mmo jobs status JOB_ID
```

A failed strict result normally indicates invalid JSON, a missing/unexpected field, an invalid URI/date, command evidence absent from captured events, an artifact hash/path mismatch, or a role-specific literal-evidence mismatch. `agents_wait` returns compact state and progress revisions; terminal previews appear only when explicitly requested. Start `agent_result` or `jobs result` at cursor zero and follow each `next_cursor` until null; text pages are contiguous and lossless. When `agent_trace` reports a truncated valid event, use its `record_cursor` with `agent_trace_record` and follow `next_cursor` to reconstruct the exact filtered record. Malformed trace records expose only a bounded diagnostic because they cannot be structurally filtered. Do not open MMO's job result, event, stderr, or socket paths directly from an agent—use the result, inspect, and trace APIs. A result that validated may still be explicitly rejected by the root after review. Reading a result does not accept it, and accepting a writer result does not apply its patch until `agent_patch_integrate` succeeds.

Strict Agent-MCP roles use app-server `turn/start.outputSchema` when the selected model supports
structured output, then validate the complete original contract independently. A malformed first
response receives one shape-only repair turn on the same durable thread. There is no JSON-fixer
agent and no repair is allowed to invent absent evidence.

## A long-running root or worker is quiet, paused, or suspended

Do not ask the model to estimate elapsed time. Inspect the host-owned state and filtered trace:

```bash
codex-mmo jobs status JOB_ID
codex-mmo jobs result JOB_ID
codex-mmo session show SESSION_ID
```

`stall_warning_seconds` is diagnostic only. It never interrupts a quiet model. Turn-mode roles have
no profile task deadline; goal-mode roles use Codex's cumulative token accounting and remain
recoverable when the current budget is exhausted. An authorized controller can inspect/trace,
pause, continue with an optional larger total budget within the compiled ceiling, request
finalization, detach, or fully stop. A worker client/transport failure reconnects to its exact
host/thread; root recovery uses the current canonical thread generation. Unrecovered host loss
produces `suspended`, not a missing result. A replacement turn-mode worker first settles any
orphaned active turn; it starts one continuation only after an interrupted outcome and preserves a
result that completed during the race. `stopped`, `failed`, and
`cancelled` remain terminal but retain readable partial evidence, terminal history, and patches.

Do not try to fix a slow model by adding a model-authored “work for N seconds” instruction. The
provider `stream_idle_timeout_ms`, app-server initialization/lifecycle request limits, Tool MCP
call limits, goal token ceilings, and external smoke/evaluation wall limits are different
boundaries. Direct provider idle values must be positive and should be sized for the route's real
latency rather than a model instruction. App-server lifecycle waits cover slow MCP startup and
thread operations but do not time model turns. Tool calls may retain their own safety bounds; the
surrounding thread and evidence survive client detach, transport restart, or interruption.

For a noninteractive root, `exec --wall-timeout` is only a caller/harness boundary. It detaches
without interrupting the app-server goal; use `codex-mmo resume SESSION_ID` to attach to the same
work. `session detach` also leaves root and workers running. `session pause` checkpoints partial
evidence, interrupts admitted turns, retires root and supervised-worker hosts, and releases their
capacity. `session continue` freshly admits and reactivates the same root thread plus the exact
paused worker set. Run `session compact` only from that cold-paused state; it temporarily hosts the
root, compacts the same thread, and cold-pauses it again. During bootstrap use
`session stop` or `session cancel` when no durable thread exists yet. `session stop --grace SECONDS`
requests finalization and then retires all hosts; `session cancel` is immediate.

## A worker is waiting for input or approval

`approval_policy = "never"` declines escalation automatically inside the role's compiled
sandbox. With `untrusted` or `on-request`, an authorized controller sees the raw pending method and
request ID through `agent_inspect` and must send that method's exact response object with
`agent_respond`. The request and thread remain durable while pending. Do not send a generic
approval shape: command, file-change, permission, user-input, and MCP-elicitation methods have
different generated app-server response schemas.

If a mutating control returns a transport error after delivery began, inspect the job before
retrying. MMO records that revision as `delivery_unknown`, because the worker may have applied the
request before its response was lost. A blind retry could steer, interrupt, finalize, or fork the
thread twice. Controls are serialized per job, but serialization cannot turn an ambiguous network
outcome into proof of non-application.

## A job changed files outside its scope

Writable MCP jobs run in isolated Git worktrees, so they do not edit the canonical checkout directly. An out-of-scope or unsafe patch boundary is a hard job failure that taints the session; inspect the job metadata and isolated artifact. Do not integrate it. Narrow the assignment or use a read-only reviewer, then launch a fresh bounded job if still justified.

## Cancellation appears incomplete

Cancel the root session to cascade through descendants:

```bash
codex-mmo session cancel SESSION_ID
```

For one branch:

```bash
codex-mmo jobs cancel JOB_ID --reason "superseded"
```

The supervisor terminates worker process groups and records terminal state. If a process remains, inspect the job stderr and verify that the process was launched by the current installed runtime rather than an unrelated manual command.

## Native agents and MCP agents disagree

Do not vote or average their conclusions. Resolve the conflict from primary evidence: source code, commands, logs, tests, specifications, or a reproducible external observation. Use a read-only reviewer when an independent check is useful.

## Release archives look incomplete

Verify checksums and member counts before installation:

```bash
sha256sum -c codex-multimodel-orchestrator-8.0.0-SHA256SUMS.txt
tar -tzf codex-multimodel-orchestrator-8.0.0-linux.tar.gz | head
unzip -Z1 codex-multimodel-orchestrator-8.0.0-linux.zip | head
```

Each archive must contain the top-level package directory, runtime modules, profiles, configuration, documentation, tests, installer, and `PACKAGE-MANIFEST.json`. The release builder rejects a truncated source tree and verifies member hashes before reporting success.

## Collecting diagnostics

Preserve the following when investigating a failure:

- The profile ID and snapshot hash.
- `codex-mmo doctor` output.
- Session and job IDs.
- Job stderr and event logs.
- Gateway logs for translated providers.
- Route, serving-provider, HTTP status, policy, and retry information.
- The exact repository commit and dirty state.
- The installed Codex and Switchyard versions.

Remove credentials before sharing diagnostics.