107 lines
9.4 KiB
Markdown
107 lines
9.4 KiB
Markdown
# Profile portfolio coverage
|
||
|
||
This document maps the coordination and workload properties intentionally demonstrated by the implemented portfolio. A check records design intent; it does not claim that a live efficacy gate has already passed.
|
||
|
||
## Workload and coordination coverage
|
||
|
||
| Profile | General engineering | Debugging | Research | Security | Visual | Refactor/contracts | Incident analysis | Cost/access | Route resilience |
|
||
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||
| `adaptive-engineering` | ✓ | | | | | | | | |
|
||
| `codex-harness-team` | ✓ | | | | | | | | |
|
||
| `high-confidence-debugging` | | ✓ | | | | | | | |
|
||
| `research-backed-engineering` | | | ✓ | | | | | | |
|
||
| `secure-change` | | | | ✓ | | | | | |
|
||
| `visual-engineering` | | | | | ✓ | | | | |
|
||
| `contract-first-refactoring` | | | | | | ✓ | | | |
|
||
| `incident-hypothesis-triage` | | | | | | | ✓ | | |
|
||
| `access-efficient-escalation-lab` | | | | | | | | ✓ | |
|
||
| `route-resilience-lab` | | | | | | | | | ✓ |
|
||
| `bounded-research-organization-lab` | | | ✓ | | | | | | |
|
||
| `competing-implementations-lab` | ✓ | | | | | | | | |
|
||
|
||
## Agentic pattern coverage
|
||
|
||
| Profile | Heterogeneous makers | Homogeneous control | Dynamic delegation | Bounded nesting | Parallel work | Adversarial/critic | Independent verification | Weak-model containment | Isolated competing writes |
|
||
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||
| `adaptive-engineering` | ✓ | | ✓ | | ✓ | ✓ | ✓ | | |
|
||
| `codex-harness-team` | | ✓ | ✓ | | ✓ | ✓ | ✓ | | |
|
||
| `high-confidence-debugging` | ✓ | | | | ✓ | ✓ | ✓ | | |
|
||
| `research-backed-engineering` | ✓ | | ✓ | | ✓ | | ✓ | | |
|
||
| `secure-change` | ✓ | | ✓ | | ✓ | ✓ | ✓ | | |
|
||
| `visual-engineering` | | ✓ | ✓ | | ✓ | ✓ | ✓ | | |
|
||
| `contract-first-refactoring` | ✓ | | | | ✓ | | ✓ | | |
|
||
| `incident-hypothesis-triage` | ✓ | | ✓ | | ✓ | ✓ | ✓ | | |
|
||
| `access-efficient-escalation-lab` | ✓ | | ✓ | | ✓ | | ✓ | ✓ | |
|
||
| `route-resilience-lab` | ✓ | | | | ✓ | | ✓ | | |
|
||
| `bounded-research-organization-lab` | ✓ | | ✓ | ✓ | ✓ | | ✓ | | |
|
||
| `competing-implementations-lab` | ✓ | | | | ✓ | | ✓ | | ✓ |
|
||
|
||
`high-confidence-debugging` overlaps diagnosis with independent reproduction, but its repair and verification barriers are sequential. A check under parallel work therefore means useful independent activity exists, not that the entire graph is fan-out.
|
||
|
||
## Access and backend coverage
|
||
|
||
| Profile | Codex/ChatGPT | OpenCode Go | OpenCode Zen | Z.AI Coding Plan | OpenRouter | Local | Native agents | Agent MCP | Hybrid |
|
||
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||
| `adaptive-engineering` | ✓ | ✓ | ✓ | | | | ✓ | ✓ | ✓ |
|
||
| `codex-harness-team` | ✓ | | | | | | ✓ | ✓ | ✓ |
|
||
| `high-confidence-debugging` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||
| `research-backed-engineering` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||
| `secure-change` | | ✓ | ✓ | ✓ | | | | ✓ | |
|
||
| `visual-engineering` | ✓ | | | | | | ✓ | ✓ | ✓ |
|
||
| `contract-first-refactoring` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||
| `incident-hypothesis-triage` | | ✓ | | ✓ | ✓ | | | ✓ | |
|
||
| `access-efficient-escalation-lab` | ✓ | ✓ | | ✓ | | ✓ | | ✓ | |
|
||
| `route-resilience-lab` | ✓ | ✓ | | | ✓ | | | ✓ | |
|
||
| `bounded-research-organization-lab` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||
| `competing-implementations-lab` | ✓ | ✓ | ✓ | | | | | ✓ | |
|
||
|
||
The matrix distinguishes access coverage from maker diversity. A ChatGPT-only profile is valuable as a harness control; a route-resilience profile intentionally uses the same DeepSeek model through different access products. Neither should be mislabeled as the other kind of diversity.
|
||
|
||
## Mechanical guarantees
|
||
|
||
The runtime, not profile prose, enforces the following for Agent-MCP work:
|
||
|
||
- exact route/model/reasoning binding and per-route availability;
|
||
- authenticated lineage, directed spawn edges, per-target action grants, reusable active capacity, and depth;
|
||
- task-kind admission and structured `literal_task` input for low-trust roles;
|
||
- sandbox ceilings, attachment containment, resource units, goal token ceilings or clock-free turns, warning-only stalls, bounded explicit finalization, and cancellation cascades;
|
||
- persistent Unix app-server hosts and exact thread identity, complete event/terminal history, bounded partial evidence, same-thread transport/host recovery, and continuation;
|
||
- opaque-run compare-and-swap inspect/trace/steer/interrupt/pause/continue/detach/stop/finalize/compact/respond/effort/fork controls limited by exact per-target action grants across root, native, and MCP runs;
|
||
- isolated Git worktrees for delegated writers, binary-safe patch capture, artifact/hash correlation, and canonical integration controlled by the root;
|
||
- monotonic `read`, `accept`, `reject`, and `integrate` result states with audit events;
|
||
- strict JSON result shape plus command/artifact correlation where declared;
|
||
- route-policy emission and requested/actual serving telemetry;
|
||
- separate API, subscription, and local-resource ledgers.
|
||
|
||
The runtime does not mechanically prove:
|
||
|
||
- semantic truth of a valid JSON result;
|
||
- that a native role obeyed every prompt-level limit;
|
||
- that every non-wait second was productive;
|
||
- independent infrastructure merely from different model names;
|
||
- model-role superiority without repeated live evaluation.
|
||
|
||
All featured native participants are read-only. Native output contracts are advisory and native spawn telemetry remains Codex-owned. Profiles use Agent MCP whenever strict scopes, lifecycle, result contracts, external routes, or bounded nesting are essential.
|
||
|
||
## Result and write safety
|
||
|
||
Every isolated writer works outside the canonical checkout. A successful worker produces a patch/artifact manifest and a clean or tainted audit state; it does not modify the root workspace. The root must read the result, choose `accept` or `reject`, and call integration explicitly. Integration rechecks patch identity and applies it to the canonical worktree under a lease. Concurrent writers are bundled only in `competing-implementations-lab`, where the worktrees are independent and only one candidate is selected.
|
||
|
||
## Evaluation coverage
|
||
|
||
The twelve suite IDs map one-to-one to the twelve profiles. Schema 8 provides neutral outcomes, optional orchestration diagnostics, configured-root and strongest-single controls, accessible-service controls, root-plus-worker variants, complete worker ablations, three development trials, five release trials, negative controls, difficulty bands, deterministic hidden mutations, and holdout files.
|
||
|
||
The evaluator records success, objective validation, wall time, root wait/activity evidence, overlap, acceptance/rejection/integration, contradictions, retries, route/serving identity, tokens, and non-interchangeable cost ledgers. It publishes no aggregate quality/cost score.
|
||
|
||
The visual suite uses Playwright screenshots and an image-preserving transport. Security, research, debugging, and contract suites include workload-specific evidence assertions. Route and incident suites inject typed credential-loss, rate-limit, and timeout faults without silently rebinding affected roles.
|
||
|
||
## Tool MCP boundary
|
||
|
||
Tool MCP is not Agent MCP. Operators own a finite, machine-specific server registry; profiles can grant only existing server/tool IDs and can only narrow the registry’s approval ceiling. Credentials and endpoints never enter installable profile packs. A role’s read-only workspace permission does not make a third-party MCP tool read-only in its external system.
|
||
|
||
## Honest release boundary
|
||
|
||
Offline validation can establish schema, safety, transport construction, lifecycle, and fixture behavior. It cannot establish current authenticated availability or composition efficacy. A release claim remains blocked until live catalog fingerprints, provider smoke tests, five-trial featured evaluations, complete cost/route telemetry, and all ablation/promotion gates are retained as evidence.
|
||
|
||
The 2026-08-23 external audit repinned the execution contract to Codex 0.149.0 alone and refreshed the public inventories to 29 Go, 64 Zen, and 422 OpenRouter records. Go has one listing-only non-agent record (`hy3-preview`); the Muse contributor route now has a complete executable evidence join. Zen has 58 executable records plus six Google-native catalog-only records; two executable records are explicitly `live-undocumented` because the live listing and Models.dev agree while the provider documentation table omits them. Authenticated Z.AI Coding Plan discovery still establishes the three selected bindings among nine advertised IDs, not support for the six unselected IDs. A no-retry, eight-token probe completed through built-in Luna and through OpenRouter Chat with Mistral Nemo; every Go/Zen protocol pairing returned upstream HTTP 403 and both Z.AI Coding pairings returned upstream HTTP 429. The older pinned DeepSeek OpenRouter endpoint failure also remains unresolved: success on a different reviewed model is not a fallback or evidence for that endpoint. Current provider availability, selected-route endpoint health, updated Z.AI document review, complete route telemetry, and repeated efficacy/ablation trials therefore remain release blockers rather than inferred successes.
|