Validation

Codex MMO has two validation levels.

Release acceptance

Run from the package source or an extracted release:

PYTHONDONTWRITEBYTECODE=1 python3 scripts/validate_package.py

Write a machine-readable report:

python3 scripts/validate_package.py \
  --report codex-multimodel-orchestrator-8.0.0-VALIDATION.json

The release validator checks:

Required source, documentation, profile, configuration, and evaluation files.
Exact source-tree agreement with the checked-in PACKAGE-MANIFEST.json.
Python AST and bytecode compilation without writing into the source tree.
Shell syntax for launchers and wrapper templates.
TOML and JSON parsing across the package.
No symlinks, special files, bytecode caches, unresolved release placeholders, embedded private keys, or common credential formats.
Static non-executable profile-pack constraints.
Exact twelve-profile inventory—eight featured profiles and four labs—with no retired-ID aliases and coverage of MCP, native, and hybrid orchestration.
Resolution and immutable compilation of every profile.
Closed Tool MCP registry definitions, per-agent subset grants, and secret-free snapshot material.
Low-trust mechanical boundaries.
Complete, hash-checked catalog snapshots: OpenRouter, OpenCode Zen/Go, Z.AI API, Z.AI Coding Plan, and Codex.
Deterministic regeneration of the catalog and upstream inventory.
Evaluation-suite validity, including safe image fixtures and closed worker assertions.
The full offline integration suite unless --skip-tests is specified.

Development quality gates

Run make quality to check Ruff formatting and lint, mypy, Pyright, ShellCheck, and shfmt. These gates are complementary to the behavioral and release-acceptance checks above.

Offline integration suite

make test

The suite uses deterministic local stand-ins rather than external APIs. It covers:

Common inventory-snapshot integrity, catalog baselines, OpenCode Zen/OpenRouter discovery, and runtime Codex discovery.
Authenticated Z.AI discovery without credential leakage.
All profile resolution and content-addressed snapshots.
MCP/native/hybrid configuration generation.
Tool MCP registry layering, profile grant bounds, native-role enable/disable overlays, credential forwarding, and CLI readiness diagnostics.
Native-agent capability tokens and caller authentication.
Persistent interactive and noninteractive app-server resume with immutable snapshot/home/run reuse, ordered top-level root-thread succession, predecessor-ID resolution, stable run-scoped capabilities, stale worker/launcher reconciliation, reusable active-only admission capacity, and token budgets that are never replenished by reattachment.
Asynchronous app-server worker completion, strict outputSchema projection plus full-contract validation, same-thread shape repair, orphaned-turn settlement before exactly one replacement continuation, durable partial evidence, and structured result contracts.
Codex goal-token budgets, clock-free turn execution, warning-only stalls, external harness detachment, finalization grace, slow-provider accounting, durable pending requests, transport recovery, suspended continuation, live controls, graceful stop, and immediate cancellation.
Low-trust task, permission, and backend enforcement.
Resource limits and conflicting write-scope rejection.
Cancellation reasons and descendant lifecycle handling.
Git audit of new and pre-existing dirty files.
Native and MCP smoke paths.
Atomic installation, upgrade backup, credential preservation, paths containing spaces, and uninstallation.
Evaluation image transport, closed MCP worker assertions, observed wait/parallelism telemetry, execution, and comparison metrics.
The complete 69-node CLI parser surface, command-local help, typo guidance, early cross-option validation, stdin nonblocking behavior, stdout/stderr separation, strict JSON failures, TTY and redirected rendering, narrow-terminal identifier fidelity, quiet progress, wrapper routing, and destructive dry-run reporting.

Live acceptance

Offline success does not prove that a provider account, entitlement, endpoint, model, or local server is currently available. Run:

codex-mmo catalog verify --remote --codex
codex-mmo tool-mcp validate
codex-mmo doctor --profile PROFILE --live --probe
codex-mmo profile smoke PROFILE --cwd "$PWD"

catalog verify compares observed IDs with the release baseline. tool-mcp validate performs non-launching command, working-directory, and credential readiness checks. doctor --live validates binaries, credentials, Switchyard configuration, routes, the internal Agent MCP handshake, and optionally the root model. profile smoke exercises each task through its declared root, native, or Agent-MCP backend; exercise third-party Tool MCP behavior explicitly when it is part of deployment acceptance.

Reproducible release build

make release

The release builder:

Runs the full acceptance validator.
Excludes development state and bytecode.
Normalizes ownership, permissions, ordering, and timestamps.
Adds a file-level PACKAGE-MANIFEST.json to the archive.
Builds tar.gz and ZIP artifacts in a private temporary directory.
Safely extracts each artifact and verifies the exact file set, explicit parent-directory set, file/directory modes, sizes, and SHA-256 values against PACKAGE-MANIFEST.json.
Requires each extracted artifact to match the source tree and each other.
Builds both artifacts a second time independently and fails if either pair differs byte-for-byte.
Publishes the verified archives, SHA-256 file, integrity report, and generated validation report as one rollback-safe set.
Verifies every final artifact while prior artifacts remain recoverable and restores the prior set if final-name verification fails.

The complete requirement-to-implementation mapping is in docs/ACCEPTANCE.md. The external dependency and standards evidence baseline is in docs/EXTERNAL-VERACITY.md.

Independent archive verification

python3 scripts/verify_release.py --source-tree . \
  ../codex-multimodel-orchestrator-8.0.0-linux.tar.gz \
  ../codex-multimodel-orchestrator-8.0.0-linux.zip

6.2 KiB Raw Blame History