This commit is contained in:
haylan committed
1 parent
14d6b2ce7a
commit
cc9a0ddf6e
3 files changed
+92
-12
No files matched your search
@@ -55,3 +55,70 @@ password printed on stdout) in a temp fixture and asserts against the HTTP API.
|
||||
harmless, retried on next run.
|
||||
- The user's global `~/.agents/skills` already contains most Matt Pocock skills — smoke assertions
|
||||
are written to tolerate id collisions (presence + flags, not source path).
|
||||
|
||||
## 2.5 orchestration run (2026-10-14, omniroute/qwen3.8-27b)
|
||||
|
||||
**Result: PASS** — full `implement-spec` flow, all 3 tickets, review, fixes, clean exit.
|
||||
|
||||
Fixture: fresh clone of the pocock-agents `greeting` reference (AGENTS.md `## Agent skills` block →
|
||||
`docs/agents/issue-tracker.md` local tracker, GLOSSARY.md, `.scratch/greeting/spec.md` + 3 tickets —
|
||||
01+02 independent, 03 blocked by 01+02, `src/greeting.js` + node:test, no remote, no deps). Agents
|
||||
copied to `.opencode/agents/`; `opencode.json` =
|
||||
`{"$schema":…,"plugins":["G:/_DEV/repos/mattpocock-skills-opencode/dist"],"model":"omniroute/qwen3.8-27b"}`
|
||||
(`model` is a valid top-level v2 config key — confirmed against `opencode.ai/config.json`).
|
||||
|
||||
Command (cwd = fixture, background, `> run.log 2>&1`):
|
||||
|
||||
```
|
||||
opencode run --agent pocock --auto "Implement the spec in .scratch/greeting/spec.md using
|
||||
implement-spec: build all three tickets with parallel workers, review, and finish. The tracker is
|
||||
local markdown (see docs/agents/issue-tracker.md). Do not ask me anything else; proceed through the
|
||||
whole flow autonomously."
|
||||
```
|
||||
|
||||
Duration: ~63 min (11:00:26Z → 12:03:47Z) on the local 27b model. `EXIT_CODE=0`.
|
||||
|
||||
| # | Check | Result | Evidence |
|
||||
| --- | -------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 1 | Integration + ticket branches exist and are merged | PASS | `spec/greeting` is an ancestor of `main`; `git merge-base --is-ancestor` confirms 01, 02, 03 all in `spec/greeting`. 01 + 03 ff-merged; 02 diverged after 01 landed → resolved via merge commit `0308c30 "Merge branch 'spec/greeting' into ticket/greeting/02"`, exactly the agent's documented resume-worker path. |
|
||||
| 2 | Worktree dir used + cleaned | PASS | All worktrees under `/tmp/pocock-workers/pocock-25-fixture/greeting/` (`$WT` default). `git worktree list` shows only the main checkout after the run; orchestrator logged `git worktree remove --force … tickets/{01,02,03,review-fixes}` → `CLEANED`. Leftover empty `notes/` + `tickets/` dirs (no worktree admin entries) — harmless. |
|
||||
| 3 | Tests pass on integration branch | PASS | Fresh worktree of `spec/greeting`: `npm test` = 16/16 pass, 0 fail (also 16/16 at every in-run merge point). |
|
||||
| 4 | Flow evidence in run.log | PASS | `→ Skill "ask-matt"`, `→ Skill "implement-spec"`, `→ Skill "code-review"`; 4 `pocock-worker` dispatches (ticket 01/02 parallel, "Worker 02: merge integration tip" resume, ticket 03) + 4 `pocock-reviewer` dispatches (Spec axis, Standards axis, + compact re-runs); TDD red→green commit pairs per ticket (`test(…): red…` → `feat(…)`); no `✗` permission denials — zero worker escape attempts (no push/stash/out-of-allow-list skill). Only failure in the whole log: one `✗ Glob` hit by the user's global codebase-memory "graph-first" rule (environment quirk, below). |
|
||||
| 5 | Model actually served | PASS | Every turn banner: `> pocock · qwen3.8-27b` — top-level `"model"` in opencode.json took effect for orchestrator and all nested subagents. |
|
||||
| 6 | Ticket statuses updated | PASS | All three `issues/NN-*.md` now read `Status: done` with a `## Comments` note naming the branch — per the local-tracker convention in `docs/agents/issue-tracker.md`. |
|
||||
|
||||
Flow detail worth keeping: 02's divergence was handled by resuming worker 02 to merge the new
|
||||
integration tip (agent text: "It knows its own change best, so it's the merger") — the non-ff path
|
||||
worked, not just the happy ff path. Review produced 2 actionable Standards findings + 1 real Spec gap
|
||||
(farewell error-message criterion untested); orchestrator dispatched a `review-fixes` worker for them
|
||||
and merged back to `main`. `spec/greeting` then ff-merged to `main` (no remote → delivered on
|
||||
default branch, per agent text).
|
||||
|
||||
Product bugs: **none**. Everything above is agent/plugin behaving per its documented design.
|
||||
|
||||
Findings / model-weakness notes (not product bugs):
|
||||
|
||||
- **Skill-id collision resolved to the user's global copy.** When the orchestrator loaded
|
||||
`code-review`, the content came from `~/.agents/skills/code-review/SKILL.md` (pre-existing global
|
||||
Matt Pocock install), not the plugin-injected copy — same collision documented in the 1.10 note.
|
||||
The model then said the skill text was "truncated" and hunted for the SKILL.md on disk (found the
|
||||
global one). Our plugin ships the full body (`code-review` body = 6177 chars, no slicing in
|
||||
`src/index.ts`), so this is the collision + local-model behaviour, not truncation by the plugin.
|
||||
- **`graph-first` rule collision.** The user's global codebase-memory-mcp injects a rule that forces a
|
||||
graph query before any `glob`. One orchestrator `glob` was rejected with that error; the model
|
||||
recovered by using `find`. Not our product; note for anyone running this with codebase-memory
|
||||
installed.
|
||||
- **Local 27b is slower than capable.** The whole run took ~63 min; reviewer subagents were CPU-busy
|
||||
for ~14 min with no log growth (subagent output streams on completion, not incrementally — expected,
|
||||
not a stall). No stalls requiring intervention; no permission denials; the only judgment shortcuts
|
||||
were self-declared in the final summary (skipped a second post-fix review pass; left
|
||||
spec-prescribed module duplication alone).
|
||||
|
||||
Suggested agent improvements (NOT applied — for the owner):
|
||||
|
||||
- `pocock.md`: consider noting that when a skill's content looks truncated, the model may re-read the
|
||||
SKILL.md from disk; if the user also has a global Matt Pocock install, the id resolves to the file
|
||||
copy — worth a one-line "skills may resolve to a global copy; content is authoritative either way".
|
||||
- `pocock-worker.md`/`pocock.md`: the non-ff resume path works but is model-dependent; a short
|
||||
reminder that "a ticket that can't ff-merge is normal when a parallel ticket merged first — resume
|
||||
its worker" could make the 27b class of model take the documented path more reliably.
|
||||
Reference in new issue
Block a user