# Validation

Codex MMO has two validation levels.

## Release acceptance

Run from the package source or an extracted release:

```bash
PYTHONDONTWRITEBYTECODE=1 python3 scripts/validate_package.py
```

Write a machine-readable report:

```bash
python3 scripts/validate_package.py \
  --report codex-multimodel-orchestrator-8.0.0-VALIDATION.json
```

The release validator checks:

1. Required source, documentation, profile, configuration, and evaluation files.
2. Exact source-tree agreement with the checked-in `PACKAGE-MANIFEST.json`.
3. Python AST and bytecode compilation without writing into the source tree.
4. Shell syntax for launchers and wrapper templates.
5. TOML and JSON parsing across the package.
6. No symlinks, special files, bytecode caches, unresolved release placeholders, embedded private keys, or common credential formats.
7. Static non-executable profile-pack constraints.
8. Exact twelve-profile inventory—eight featured profiles and four labs—with no retired-ID aliases and coverage of MCP, native, and hybrid orchestration.
9. Resolution and immutable compilation of every profile.
10. Closed Tool MCP registry definitions, per-agent subset grants, and secret-free snapshot material.
11. Low-trust mechanical boundaries.
12. Complete, hash-checked catalog snapshots: OpenRouter, OpenCode Zen/Go, Z.AI API, Z.AI Coding Plan, and Codex.
13. Deterministic regeneration of the catalog and upstream inventory.
14. Evaluation-suite validity, including safe image fixtures and closed worker assertions.
15. The full offline integration suite unless `--skip-tests` is specified.

## Development quality gates

Run `make quality` to check Ruff formatting and lint, mypy, Pyright, ShellCheck, and shfmt. These gates are complementary to the behavioral and release-acceptance checks above.

## Offline integration suite

```bash
make test
```

The suite uses deterministic local stand-ins rather than external APIs. It covers:

- Common inventory-snapshot integrity, catalog baselines, OpenCode Zen/OpenRouter discovery, and runtime Codex discovery.
- Authenticated Z.AI discovery without credential leakage.
- All profile resolution and content-addressed snapshots.
- MCP/native/hybrid configuration generation.
- Tool MCP registry layering, profile grant bounds, native-role enable/disable overlays, credential forwarding, and CLI readiness diagnostics.
- Native-agent capability tokens and caller authentication.
- Persistent interactive and noninteractive app-server resume with immutable snapshot/home/run
  reuse, ordered top-level root-thread succession, predecessor-ID resolution, stable run-scoped
  capabilities, stale worker/launcher reconciliation, reusable active-only admission capacity,
  and token budgets that are never replenished by reattachment.
- Asynchronous app-server worker completion, strict `outputSchema` projection plus full-contract
  validation, same-thread shape repair, orphaned-turn settlement before exactly one replacement
  continuation, durable partial evidence, and structured result contracts.
- Codex goal-token budgets, clock-free turn execution, warning-only stalls, external harness
  detachment, finalization grace, slow-provider accounting, durable pending requests, transport
  recovery, suspended continuation, live controls, graceful stop, and immediate cancellation.
- Low-trust task, permission, and backend enforcement.
- Resource limits and conflicting write-scope rejection.
- Cancellation reasons and descendant lifecycle handling.
- Git audit of new and pre-existing dirty files.
- Native and MCP smoke paths.
- Atomic installation, upgrade backup, credential preservation, paths containing spaces, and uninstallation.
- Evaluation image transport, closed MCP worker assertions, observed wait/parallelism telemetry, execution, and comparison metrics.
- The complete 69-node CLI parser surface, command-local help, typo guidance, early cross-option
  validation, stdin nonblocking behavior, stdout/stderr separation, strict JSON failures, TTY and
  redirected rendering, narrow-terminal identifier fidelity, quiet progress, wrapper routing, and
  destructive dry-run reporting.

## Live acceptance

Offline success does not prove that a provider account, entitlement, endpoint, model, or local server is currently available. Run:

```bash
codex-mmo catalog verify --remote --codex
codex-mmo tool-mcp validate
codex-mmo doctor --profile PROFILE --live --probe
codex-mmo profile smoke PROFILE --cwd "$PWD"
```

`catalog verify` compares observed IDs with the release baseline. `tool-mcp validate` performs non-launching command, working-directory, and credential readiness checks. `doctor --live` validates binaries, credentials, Switchyard configuration, routes, the internal Agent MCP handshake, and optionally the root model. `profile smoke` exercises each task through its declared root, native, or Agent-MCP backend; exercise third-party Tool MCP behavior explicitly when it is part of deployment acceptance.

## Reproducible release build

```bash
make release
```

The release builder:

- Runs the full acceptance validator.
- Excludes development state and bytecode.
- Normalizes ownership, permissions, ordering, and timestamps.
- Adds a file-level `PACKAGE-MANIFEST.json` to the archive.
- Builds tar.gz and ZIP artifacts in a private temporary directory.
- Safely extracts each artifact and verifies the exact file set, explicit parent-directory set, file/directory modes, sizes, and SHA-256 values against `PACKAGE-MANIFEST.json`.
- Requires each extracted artifact to match the source tree and each other.
- Builds both artifacts a second time independently and fails if either pair differs byte-for-byte.
- Publishes the verified archives, SHA-256 file, integrity report, and generated validation report as one rollback-safe set.
- Verifies every final artifact while prior artifacts remain recoverable and restores the prior set if final-name verification fails.

The complete requirement-to-implementation mapping is in [docs/ACCEPTANCE.md](docs/ACCEPTANCE.md).
The external dependency and standards evidence baseline is in [docs/EXTERNAL-VERACITY.md](docs/EXTERNAL-VERACITY.md).

## Independent archive verification

```bash
python3 scripts/verify_release.py --source-tree . \
  ../codex-multimodel-orchestrator-8.0.0-linux.tar.gz \
  ../codex-multimodel-orchestrator-8.0.0-linux.zip
```
