125 lines
13 KiB
Markdown
125 lines
13 KiB
Markdown
# Runtime checks — verified against real OpenCode v2
|
|
|
|
Environment: Windows 11, `opencode v2.0.29`, `@opencode/plugin@2.0.29`, 2026-10-14.
|
|
Method: `scripts/smoke.ts` boots `opencode serve` (private server, Basic auth user `opencode`,
|
|
password printed on stdout) in a temp fixture and asserts against the HTTP API. Result: **PASS**.
|
|
|
|
## Verified
|
|
|
|
- **Registry install by package name (1.10, 2026-10-14)**: `@haylan/opencode-pocock@0.1.0`
|
|
published to the Gitea npm registry by the dispatched `release` workflow (run 5684, success;
|
|
tag `v0.1.0` created, package linked to repo). Anonymous registry reads work (HTTP 200 without
|
|
auth). A clean fixture with `"plugins": ["@haylan/opencode-pocock"]` + scope registry in
|
|
`~/.npmrc` loads the plugin: `/api/plugin` shows
|
|
`source {type:"package", target:"@haylan/opencode-pocock", version:"0.1.0"}`,
|
|
`state.status "active"`; skills inject from the bun cache path
|
|
(`~/.cache/opencode/npm/@haylan/opencode-pocock@latest/.../skills/...`); `npx -y
|
|
@haylan/opencode-pocock init` installs the three agents (modes decode correctly).
|
|
Id collisions with a pre-existing global `~/.agents/skills` Matt Pocock install resolve to the
|
|
file-based copy (expected; a clean machine gets all 31 from the package).
|
|
- **Plugin loads and activates** from a local directory entry (`plugins: ["<abs path>/dist"]`);
|
|
`GET /api/plugin` shows `haylan.pocock` with `state.status: "active"`.
|
|
- A **file** entry (`file://.../dist/index.js`) is IGNORED — local plugin entries must be
|
|
directories with a root `index`/`server` file (confirms fastcrw research). npm-name entries
|
|
resolve through `exports` (to be re-verified after first registry publish, TODO 1.10).
|
|
- **Skill injection works**: all 31 manifest skills registered (`GET /api/skill`); the 16
|
|
user-invoked ones carry `autoinvoke: false`. The API **omits `autoinvoke` when true** (default).
|
|
- **DMI honored natively**: file-loaded skills with upstream `disable-model-invocation: true`
|
|
(e.g. a global `~/.agents/skills` install) show `autoinvoke: false` without our plugin.
|
|
- **ID collisions**: when the same skill id also exists as a file-based skill (global install), the
|
|
file-based entry is what `/api/skill` reports; plugin injection coexists without errors.
|
|
(`pr`, absent from the tester's global install, was served from our package path.)
|
|
- **Injected content must have frontmatter stripped** to match file-based loading
|
|
(`markdown.content`); the plugin does this since 2026-10-14.
|
|
- **Agent markdown with V2-native frontmatter decodes correctly** from `.opencode/agents/`:
|
|
`pocock` (primary), `pocock-worker` / `pocock-reviewer` (subagent), descriptions intact, and the
|
|
full `permissions` array survived decoding in order — verified on `pocock-worker`:
|
|
`skill * deny` + `tdd/codebase-design/diagnosing-bugs allow`, `subagent * deny`, and all
|
|
git/gh/glab shell denies present.
|
|
- **Default permission floor** (v2.0.29): agents start with `{action:"*",resource:"*",effect:"allow"}`
|
|
plus specific rules, among them `external_directory /tmp` and `/tmp/*` **allow** — the
|
|
`/tmp/pocock-workers` worktree root needs no extra user config on this build (README still
|
|
documents the rule for other setups/subagents).
|
|
- **Injected skills do NOT become slash commands** (`GET /api/command` shows none for them;
|
|
`Skill.Info` has no `slash` field). User-invoked steps are started via conversation with `pocock`
|
|
(propose → yes → load) or by any agent explicitly loading the skill. File-based installs (global
|
|
`~/.agents/skills`, or a `skills: [...]` directory source) keep slash behavior.
|
|
- **Server API**: prefix `/api/*`, Basic auth (`opencode` + printed password); SPA HTML answers
|
|
unknown non-/api paths with 200 — assert JSON, not status. `POST /api/location/reload` exists.
|
|
Plugin loading is lazy (appears seconds after boot); `/openapi.json` requires auth.
|
|
|
|
## Known environment quirks (tester machine, not product issues)
|
|
|
|
- Windows: `server.kill()` under `shell:true` only kills `cmd.exe`; smoke uses
|
|
`taskkill /PID <pid> /T /F`. Temp fixture deletion can lag behind handle release (EPERM warning);
|
|
harmless, retried on next run.
|
|
- The user's global `~/.agents/skills` already contains most Matt Pocock skills — smoke assertions
|
|
are written to tolerate id collisions (presence + flags, not source path).
|
|
|
|
## 2.5 orchestration run (2026-10-14, omniroute/qwen3.8-27b)
|
|
|
|
**Result: PASS** — full `implement-spec` flow, all 3 tickets, review, fixes, clean exit.
|
|
|
|
Fixture: fresh clone of the pocock-agents `greeting` reference (AGENTS.md `## Agent skills` block →
|
|
`docs/agents/issue-tracker.md` local tracker, GLOSSARY.md, `.scratch/greeting/spec.md` + 3 tickets —
|
|
01+02 independent, 03 blocked by 01+02, `src/greeting.js` + node:test, no remote, no deps). Agents
|
|
copied to `.opencode/agents/`; `opencode.json` =
|
|
`{"$schema":…,"plugins":["G:/_DEV/repos/mattpocock-skills-opencode/dist"],"model":"omniroute/qwen3.8-27b"}`
|
|
(`model` is a valid top-level v2 config key — confirmed against `opencode.ai/config.json`).
|
|
|
|
Command (cwd = fixture, background, `> run.log 2>&1`):
|
|
|
|
```
|
|
opencode run --agent pocock --auto "Implement the spec in .scratch/greeting/spec.md using
|
|
implement-spec: build all three tickets with parallel workers, review, and finish. The tracker is
|
|
local markdown (see docs/agents/issue-tracker.md). Do not ask me anything else; proceed through the
|
|
whole flow autonomously."
|
|
```
|
|
|
|
Duration: ~63 min (11:00:26Z → 12:03:47Z) on the local 27b model. `EXIT_CODE=0`.
|
|
|
|
| # | Check | Result | Evidence |
|
|
| --- | -------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| 1 | Integration + ticket branches exist and are merged | PASS | `spec/greeting` is an ancestor of `main`; `git merge-base --is-ancestor` confirms 01, 02, 03 all in `spec/greeting`. 01 + 03 ff-merged; 02 diverged after 01 landed → resolved via merge commit `0308c30 "Merge branch 'spec/greeting' into ticket/greeting/02"`, exactly the agent's documented resume-worker path. |
|
|
| 2 | Worktree dir used + cleaned | PASS | All worktrees under `/tmp/pocock-workers/pocock-25-fixture/greeting/` (`$WT` default). `git worktree list` shows only the main checkout after the run; orchestrator logged `git worktree remove --force … tickets/{01,02,03,review-fixes}` → `CLEANED`. Leftover empty `notes/` + `tickets/` dirs (no worktree admin entries) — harmless. |
|
|
| 3 | Tests pass on integration branch | PASS | Fresh worktree of `spec/greeting`: `npm test` = 16/16 pass, 0 fail (also 16/16 at every in-run merge point). |
|
|
| 4 | Flow evidence in run.log | PASS | `→ Skill "ask-matt"`, `→ Skill "implement-spec"`, `→ Skill "code-review"`; 4 `pocock-worker` dispatches (ticket 01/02 parallel, "Worker 02: merge integration tip" resume, ticket 03) + 4 `pocock-reviewer` dispatches (Spec axis, Standards axis, + compact re-runs); TDD red→green commit pairs per ticket (`test(…): red…` → `feat(…)`); no `✗` permission denials — zero worker escape attempts (no push/stash/out-of-allow-list skill). Only failure in the whole log: one `✗ Glob` hit by the user's global codebase-memory "graph-first" rule (environment quirk, below). |
|
|
| 5 | Model actually served | PASS | Every turn banner: `> pocock · qwen3.8-27b` — top-level `"model"` in opencode.json took effect for orchestrator and all nested subagents. |
|
|
| 6 | Ticket statuses updated | PASS | All three `issues/NN-*.md` now read `Status: done` with a `## Comments` note naming the branch — per the local-tracker convention in `docs/agents/issue-tracker.md`. |
|
|
|
|
Flow detail worth keeping: 02's divergence was handled by resuming worker 02 to merge the new
|
|
integration tip (agent text: "It knows its own change best, so it's the merger") — the non-ff path
|
|
worked, not just the happy ff path. Review produced 2 actionable Standards findings + 1 real Spec gap
|
|
(farewell error-message criterion untested); orchestrator dispatched a `review-fixes` worker for them
|
|
and merged back to `main`. `spec/greeting` then ff-merged to `main` (no remote → delivered on
|
|
default branch, per agent text).
|
|
|
|
Product bugs: **none**. Everything above is agent/plugin behaving per its documented design.
|
|
|
|
Findings / model-weakness notes (not product bugs):
|
|
|
|
- **Skill-id collision resolved to the user's global copy.** When the orchestrator loaded
|
|
`code-review`, the content came from `~/.agents/skills/code-review/SKILL.md` (pre-existing global
|
|
Matt Pocock install), not the plugin-injected copy — same collision documented in the 1.10 note.
|
|
The model then said the skill text was "truncated" and hunted for the SKILL.md on disk (found the
|
|
global one). Our plugin ships the full body (`code-review` body = 6177 chars, no slicing in
|
|
`src/index.ts`), so this is the collision + local-model behaviour, not truncation by the plugin.
|
|
- **`graph-first` rule collision.** The user's global codebase-memory-mcp injects a rule that forces a
|
|
graph query before any `glob`. One orchestrator `glob` was rejected with that error; the model
|
|
recovered by using `find`. Not our product; note for anyone running this with codebase-memory
|
|
installed.
|
|
- **Local 27b is slower than capable.** The whole run took ~63 min; reviewer subagents were CPU-busy
|
|
for ~14 min with no log growth (subagent output streams on completion, not incrementally — expected,
|
|
not a stall). No stalls requiring intervention; no permission denials; the only judgment shortcuts
|
|
were self-declared in the final summary (skipped a second post-fix review pass; left
|
|
spec-prescribed module duplication alone).
|
|
|
|
Suggested agent improvements (NOT applied — for the owner):
|
|
|
|
- `pocock.md`: consider noting that when a skill's content looks truncated, the model may re-read the
|
|
SKILL.md from disk; if the user also has a global Matt Pocock install, the id resolves to the file
|
|
copy — worth a one-line "skills may resolve to a global copy; content is authoritative either way".
|
|
- `pocock-worker.md`/`pocock.md`: the non-ff resume path works but is model-dependent; a short
|
|
reminder that "a ticket that can't ff-merge is normal when a parallel ticket merged first — resume
|
|
its worker" could make the 27b class of model take the documented path more reliably.
|