diff --git a/.opencode/state/status.md b/.opencode/state/status.md index 0847c26..ed1aea8 100644 --- a/.opencode/state/status.md +++ b/.opencode/state/status.md @@ -1,14 +1,16 @@ # Project status -Updated: 2026-10-14 (Phase 1 COMPLETE — quality gate passed) +Updated: 2026-10-14 (2.5 orchestration run PASS — quality gate + end-to-end run both green) ## Current phase -Phase 1 done: plugin installs, loads in OpenCode v2.0.29, skills inject, package builds, -`@haylan/opencode-pocock@0.1.0` published to the Gitea registry (release run 5684), install-from- -registry verified, nightly workflow proven (dispatch run 5685: github.com reachable, no-change -early exit). Remaining: Phase 2.5 (manual token-spending orchestration run) and 2.7 (observe a -week of nightlies). +Phase 1 done AND 2.5 done: plugin installs, loads in OpenCode v2.0.29, skills inject, package +builds, `@haylan/opencode-pocock@0.1.0` published to the Gitea registry (release run 5684), +install-from-registry verified, nightly workflow proven (dispatch run 5685: github.com reachable, +no-change early exit), and a full manual orchestration run of the `pocock` orchestrator against a +3-ticket fixture (omniroute/qwen3.8-27b) is **PASS** — all 3 tickets built by parallel workers, +reviewed, fixed, merged, clean exit, no product bugs. Remaining: 2.7 (observe a week of +nightlies). ## Done @@ -41,8 +43,12 @@ week of nightlies). ## Known gaps / risks -- 2.5 not done: no token-spending end-to-end orchestration run yet (pocock building a fixture - spec with parallel workers). Drift/unit/smoke suites cover the deterministic surface. +- 2.5 DONE (2026-10-14): full orchestration run PASS on omniroute/qwen3.8-27b — see + docs/runtime-checks.md § "2.5 orchestration run". Noted env-only quirks: skill id collision + resolves `code-review` to the user's global `~/.agents/skills` copy (documented, not a plugin + bug); user's codebase-memory "graph-first" rule rejected one orchestrator `glob` (recovered via + `find`). Local 27b took ~63 min; reviewer subagents stream output only on completion (looked + stalled, was not). No product bugs found. - 2.7: nightlies need a week of green runs before "stable" is declared; first scheduled run 2026-10-15 03:30 UTC. - Background subagents in V2 (env-gated in V1) not verified; orchestrator text works either way. @@ -53,10 +59,10 @@ week of nightlies). ## Next steps (exact) -1. 2.5: manual orchestration run — fixture project with spec + 3 tickets (2 independent, 1 - blocked), run `opencode --agent pocock`, verify worktree loop/merges/review/permissions; - record in docs/runtime-checks.md. +1. ~~2.5: manual orchestration run~~ DONE 2026-10-14 (PASS, see docs/runtime-checks.md). 2. Watch nightly runs for a week (2.7); on upstream release (17 pending changesets), verify the full sync→generate→publish→commit-back cycle end-to-end. 3. Then declare stable and announce; keep classification.json current when upstream promotes in-progress skills. +4. (Optional, owner) Decide whether to fold the 2.5 agent-improvement suggestions (truncated-skill + note, non-ff resume reminder) into agents/*.md — they are in the runtime-checks.md report. diff --git a/TODO.md b/TODO.md index 9d865af..77a990b 100644 --- a/TODO.md +++ b/TODO.md @@ -48,8 +48,10 @@ State: `.opencode/state/status.md`. Design: `docs/architecture.md`. permissions (edit/subagent/webfetch deny, git-read shell allow-list, skill allow code-review) - [x] 2.4 orchestrator V2 flow wiring — done in 1.6: agent text covers background AND batch dispatch; slash UX + /tmp default documented in README from runtime findings -- [ ] 2.5 per-skill regression checks (drift suite exists; add manual token-spending orchestration +- [x] 2.5 per-skill regression checks (drift suite exists; add manual token-spending orchestration run: fixture spec + 3 tickets like pocock-agents smoke) + record in docs/runtime-checks.md + (DONE 2026-10-14: full PASS on omniroute/qwen3.8-27b, ~63 min, all 3 tickets + review + fixes, + no product bugs — see docs/runtime-checks.md § "2.5 orchestration run") - [x] 2.6 README usage guide + finalize skill-mapping (matrix generated from classification.json) - [ ] 2.7 one week of green nightlies -> declare stable (BLOCKED on PACKAGE_TOKEN secret) @@ -69,3 +71,8 @@ State: `.opencode/state/status.md`. Design: `docs/architecture.md`. ## Done log - 2026-10-14: phase-0 analysis docs written (AGENTS.md, state/, docs/architecture.md, TODO.md). +- 2026-10-14: 2.5 manual orchestration run — `pocock` orchestrator drove a 3-ticket `greeting` + fixture (omniroute/qwen3.8-27b, ~63 min) end-to-end: parallel workers on isolated worktrees, + non-ff resume path exercised, two-axis code review, review-fix worker, all 16 tests green on + `main`, worktrees cleaned, tickets marked done, `EXIT_CODE=0`. **PASS, no product bugs.** + Recorded in docs/runtime-checks.md § "2.5 orchestration run". diff --git a/docs/runtime-checks.md b/docs/runtime-checks.md index 2361590..c39798c 100644 --- a/docs/runtime-checks.md +++ b/docs/runtime-checks.md @@ -55,3 +55,70 @@ password printed on stdout) in a temp fixture and asserts against the HTTP API. harmless, retried on next run. - The user's global `~/.agents/skills` already contains most Matt Pocock skills — smoke assertions are written to tolerate id collisions (presence + flags, not source path). + +## 2.5 orchestration run (2026-10-14, omniroute/qwen3.8-27b) + +**Result: PASS** — full `implement-spec` flow, all 3 tickets, review, fixes, clean exit. + +Fixture: fresh clone of the pocock-agents `greeting` reference (AGENTS.md `## Agent skills` block → +`docs/agents/issue-tracker.md` local tracker, GLOSSARY.md, `.scratch/greeting/spec.md` + 3 tickets — +01+02 independent, 03 blocked by 01+02, `src/greeting.js` + node:test, no remote, no deps). Agents +copied to `.opencode/agents/`; `opencode.json` = +`{"$schema":…,"plugins":["G:/_DEV/repos/mattpocock-skills-opencode/dist"],"model":"omniroute/qwen3.8-27b"}` +(`model` is a valid top-level v2 config key — confirmed against `opencode.ai/config.json`). + +Command (cwd = fixture, background, `> run.log 2>&1`): + +``` +opencode run --agent pocock --auto "Implement the spec in .scratch/greeting/spec.md using +implement-spec: build all three tickets with parallel workers, review, and finish. The tracker is +local markdown (see docs/agents/issue-tracker.md). Do not ask me anything else; proceed through the +whole flow autonomously." +``` + +Duration: ~63 min (11:00:26Z → 12:03:47Z) on the local 27b model. `EXIT_CODE=0`. + +| # | Check | Result | Evidence | +| --- | -------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Integration + ticket branches exist and are merged | PASS | `spec/greeting` is an ancestor of `main`; `git merge-base --is-ancestor` confirms 01, 02, 03 all in `spec/greeting`. 01 + 03 ff-merged; 02 diverged after 01 landed → resolved via merge commit `0308c30 "Merge branch 'spec/greeting' into ticket/greeting/02"`, exactly the agent's documented resume-worker path. | +| 2 | Worktree dir used + cleaned | PASS | All worktrees under `/tmp/pocock-workers/pocock-25-fixture/greeting/` (`$WT` default). `git worktree list` shows only the main checkout after the run; orchestrator logged `git worktree remove --force … tickets/{01,02,03,review-fixes}` → `CLEANED`. Leftover empty `notes/` + `tickets/` dirs (no worktree admin entries) — harmless. | +| 3 | Tests pass on integration branch | PASS | Fresh worktree of `spec/greeting`: `npm test` = 16/16 pass, 0 fail (also 16/16 at every in-run merge point). | +| 4 | Flow evidence in run.log | PASS | `→ Skill "ask-matt"`, `→ Skill "implement-spec"`, `→ Skill "code-review"`; 4 `pocock-worker` dispatches (ticket 01/02 parallel, "Worker 02: merge integration tip" resume, ticket 03) + 4 `pocock-reviewer` dispatches (Spec axis, Standards axis, + compact re-runs); TDD red→green commit pairs per ticket (`test(…): red…` → `feat(…)`); no `✗` permission denials — zero worker escape attempts (no push/stash/out-of-allow-list skill). Only failure in the whole log: one `✗ Glob` hit by the user's global codebase-memory "graph-first" rule (environment quirk, below). | +| 5 | Model actually served | PASS | Every turn banner: `> pocock · qwen3.8-27b` — top-level `"model"` in opencode.json took effect for orchestrator and all nested subagents. | +| 6 | Ticket statuses updated | PASS | All three `issues/NN-*.md` now read `Status: done` with a `## Comments` note naming the branch — per the local-tracker convention in `docs/agents/issue-tracker.md`. | + +Flow detail worth keeping: 02's divergence was handled by resuming worker 02 to merge the new +integration tip (agent text: "It knows its own change best, so it's the merger") — the non-ff path +worked, not just the happy ff path. Review produced 2 actionable Standards findings + 1 real Spec gap +(farewell error-message criterion untested); orchestrator dispatched a `review-fixes` worker for them +and merged back to `main`. `spec/greeting` then ff-merged to `main` (no remote → delivered on +default branch, per agent text). + +Product bugs: **none**. Everything above is agent/plugin behaving per its documented design. + +Findings / model-weakness notes (not product bugs): + +- **Skill-id collision resolved to the user's global copy.** When the orchestrator loaded + `code-review`, the content came from `~/.agents/skills/code-review/SKILL.md` (pre-existing global + Matt Pocock install), not the plugin-injected copy — same collision documented in the 1.10 note. + The model then said the skill text was "truncated" and hunted for the SKILL.md on disk (found the + global one). Our plugin ships the full body (`code-review` body = 6177 chars, no slicing in + `src/index.ts`), so this is the collision + local-model behaviour, not truncation by the plugin. +- **`graph-first` rule collision.** The user's global codebase-memory-mcp injects a rule that forces a + graph query before any `glob`. One orchestrator `glob` was rejected with that error; the model + recovered by using `find`. Not our product; note for anyone running this with codebase-memory + installed. +- **Local 27b is slower than capable.** The whole run took ~63 min; reviewer subagents were CPU-busy + for ~14 min with no log growth (subagent output streams on completion, not incrementally — expected, + not a stall). No stalls requiring intervention; no permission denials; the only judgment shortcuts + were self-declared in the final summary (skipped a second post-fix review pass; left + spec-prescribed module duplication alone). + +Suggested agent improvements (NOT applied — for the owner): + +- `pocock.md`: consider noting that when a skill's content looks truncated, the model may re-read the + SKILL.md from disk; if the user also has a global Matt Pocock install, the id resolves to the file + copy — worth a one-line "skills may resolve to a global copy; content is authoritative either way". +- `pocock-worker.md`/`pocock.md`: the non-ff resume path works but is model-dependent; a short + reminder that "a ticket that can't ff-merge is normal when a parallel ticket merged first — resume + its worker" could make the 27b class of model take the documented path more reliably.