120 lines
6.2 KiB
Markdown
120 lines
6.2 KiB
Markdown
# Validation
|
|
|
|
Codex MMO has two validation levels.
|
|
|
|
## Release acceptance
|
|
|
|
Run from the package source or an extracted release:
|
|
|
|
```bash
|
|
PYTHONDONTWRITEBYTECODE=1 python3 scripts/validate_package.py
|
|
```
|
|
|
|
Write a machine-readable report:
|
|
|
|
```bash
|
|
python3 scripts/validate_package.py \
|
|
--report codex-multimodel-orchestrator-8.0.0-VALIDATION.json
|
|
```
|
|
|
|
The release validator checks:
|
|
|
|
1. Required source, documentation, profile, configuration, and evaluation files.
|
|
2. Exact source-tree agreement with the checked-in `PACKAGE-MANIFEST.json`.
|
|
3. Python AST and bytecode compilation without writing into the source tree.
|
|
4. Shell syntax for launchers and wrapper templates.
|
|
5. TOML and JSON parsing across the package.
|
|
6. No symlinks, special files, bytecode caches, unresolved release placeholders, embedded private keys, or common credential formats.
|
|
7. Static non-executable profile-pack constraints.
|
|
8. Exact twelve-profile inventory—eight featured profiles and four labs—with no retired-ID aliases and coverage of MCP, native, and hybrid orchestration.
|
|
9. Resolution and immutable compilation of every profile.
|
|
10. Closed Tool MCP registry definitions, per-agent subset grants, and secret-free snapshot material.
|
|
11. Low-trust mechanical boundaries.
|
|
12. Complete, hash-checked catalog snapshots: OpenRouter, OpenCode Zen/Go, Z.AI API, Z.AI Coding Plan, and Codex.
|
|
13. Deterministic regeneration of the catalog and upstream inventory.
|
|
14. Evaluation-suite validity, including safe image fixtures and closed worker assertions.
|
|
15. The full offline integration suite unless `--skip-tests` is specified.
|
|
|
|
## Development quality gates
|
|
|
|
Run `make quality` to check Ruff formatting and lint, mypy, Pyright, ShellCheck, and shfmt. These gates are complementary to the behavioral and release-acceptance checks above.
|
|
|
|
## Offline integration suite
|
|
|
|
```bash
|
|
make test
|
|
```
|
|
|
|
The suite uses deterministic local stand-ins rather than external APIs. It covers:
|
|
|
|
- Common inventory-snapshot integrity, catalog baselines, OpenCode Zen/OpenRouter discovery, and runtime Codex discovery.
|
|
- Authenticated Z.AI discovery without credential leakage.
|
|
- All profile resolution and content-addressed snapshots.
|
|
- MCP/native/hybrid configuration generation.
|
|
- Tool MCP registry layering, profile grant bounds, native-role enable/disable overlays, credential forwarding, and CLI readiness diagnostics.
|
|
- Native-agent capability tokens and caller authentication.
|
|
- Persistent interactive and noninteractive app-server resume with immutable snapshot/home/run
|
|
reuse, ordered top-level root-thread succession, predecessor-ID resolution, stable run-scoped
|
|
capabilities, stale worker/launcher reconciliation, reusable active-only admission capacity,
|
|
and token budgets that are never replenished by reattachment.
|
|
- Asynchronous app-server worker completion, strict `outputSchema` projection plus full-contract
|
|
validation, same-thread shape repair, orphaned-turn settlement before exactly one replacement
|
|
continuation, durable partial evidence, and structured result contracts.
|
|
- Codex goal-token budgets, clock-free turn execution, warning-only stalls, external harness
|
|
detachment, finalization grace, slow-provider accounting, durable pending requests, transport
|
|
recovery, suspended continuation, live controls, graceful stop, and immediate cancellation.
|
|
- Low-trust task, permission, and backend enforcement.
|
|
- Resource limits and conflicting write-scope rejection.
|
|
- Cancellation reasons and descendant lifecycle handling.
|
|
- Git audit of new and pre-existing dirty files.
|
|
- Native and MCP smoke paths.
|
|
- Atomic installation, upgrade backup, credential preservation, paths containing spaces, and uninstallation.
|
|
- Evaluation image transport, closed MCP worker assertions, observed wait/parallelism telemetry, execution, and comparison metrics.
|
|
- The complete 69-node CLI parser surface, command-local help, typo guidance, early cross-option
|
|
validation, stdin nonblocking behavior, stdout/stderr separation, strict JSON failures, TTY and
|
|
redirected rendering, narrow-terminal identifier fidelity, quiet progress, wrapper routing, and
|
|
destructive dry-run reporting.
|
|
|
|
## Live acceptance
|
|
|
|
Offline success does not prove that a provider account, entitlement, endpoint, model, or local server is currently available. Run:
|
|
|
|
```bash
|
|
codex-mmo catalog verify --remote --codex
|
|
codex-mmo tool-mcp validate
|
|
codex-mmo doctor --profile PROFILE --live --probe
|
|
codex-mmo profile smoke PROFILE --cwd "$PWD"
|
|
```
|
|
|
|
`catalog verify` compares observed IDs with the release baseline. `tool-mcp validate` performs non-launching command, working-directory, and credential readiness checks. `doctor --live` validates binaries, credentials, Switchyard configuration, routes, the internal Agent MCP handshake, and optionally the root model. `profile smoke` exercises each task through its declared root, native, or Agent-MCP backend; exercise third-party Tool MCP behavior explicitly when it is part of deployment acceptance.
|
|
|
|
## Reproducible release build
|
|
|
|
```bash
|
|
make release
|
|
```
|
|
|
|
The release builder:
|
|
|
|
- Runs the full acceptance validator.
|
|
- Excludes development state and bytecode.
|
|
- Normalizes ownership, permissions, ordering, and timestamps.
|
|
- Adds a file-level `PACKAGE-MANIFEST.json` to the archive.
|
|
- Builds tar.gz and ZIP artifacts in a private temporary directory.
|
|
- Safely extracts each artifact and verifies the exact file set, explicit parent-directory set, file/directory modes, sizes, and SHA-256 values against `PACKAGE-MANIFEST.json`.
|
|
- Requires each extracted artifact to match the source tree and each other.
|
|
- Builds both artifacts a second time independently and fails if either pair differs byte-for-byte.
|
|
- Publishes the verified archives, SHA-256 file, integrity report, and generated validation report as one rollback-safe set.
|
|
- Verifies every final artifact while prior artifacts remain recoverable and restores the prior set if final-name verification fails.
|
|
|
|
The complete requirement-to-implementation mapping is in [docs/ACCEPTANCE.md](docs/ACCEPTANCE.md).
|
|
The external dependency and standards evidence baseline is in [docs/EXTERNAL-VERACITY.md](docs/EXTERNAL-VERACITY.md).
|
|
|
|
## Independent archive verification
|
|
|
|
```bash
|
|
python3 scripts/verify_release.py --source-tree . \
|
|
../codex-multimodel-orchestrator-8.0.0-linux.tar.gz \
|
|
../codex-multimodel-orchestrator-8.0.0-linux.zip
|
|
```
|