chore: add project docs, glossary, research and opencode/fastcrw skills
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed
commit
b316daed43
9 files changed
+409
No files matched your search
@@ -0,0 +1,102 @@
|
||||
# fastCRW `POST /v1/search` contract
|
||||
|
||||
Resolves issue #3. Researched 2026-10-03 against `main` of github.com/us/crw (`fastcrw/crw`), docs.fastcrw.com and npm `crw-mcp` 0.37.2.
|
||||
|
||||
## TL;DR for the plugin
|
||||
|
||||
- Endpoint: `POST {CRW_API_URL}/v1/search`, JSON body `{"query": "...", "limit": N}`.
|
||||
- Auth: `Authorization: Bearer $CRW_API_KEY` for hosted. Self-hosted sends no header unless the operator set `CRW_AUTH__API_KEYS`; then it is the same Bearer scheme.
|
||||
- **Parse defensively: `data` is a flat array on hosted, but `{"data": {"results": [...]}}` on self-hosted.** The official CLI accepts both. When `sources` is set, results are grouped as `{web:[],news:[],images:[]}` instead of an array.
|
||||
- Every result has both `description` and `snippet` (snippet is an always-populated alias of description).
|
||||
- Errors: `{"success": false, "error": "...", "error_code": "..."}` with HTTP 4xx/5xx.
|
||||
|
||||
## Auth
|
||||
|
||||
- Hosted (`https://api.fastcrw.com`): `Authorization: Bearer <key>` required. [docs.fastcrw.com REST API]
|
||||
- Self-hosted (`http://localhost:3000`): no header needed by default. Middleware bypasses auth when `api_keys` is empty; otherwise checks the `Bearer ` token with constant-time compare. 401 messages: `Missing Authorization header`, `Invalid API key`. [crates/crw-server/src/middleware.rs]
|
||||
- Server-side keys are set via `CRW_AUTH__API_KEYS` (comma-separated or JSON array). [crates/crw-core/src/config.rs `AuthConfig`]
|
||||
|
||||
## Env var names
|
||||
|
||||
| Var | Used by | Meaning |
|
||||
|---|---|---|
|
||||
| `CRW_API_URL` | crw-mcp, crw CLI | Remote server base URL. Default `https://api.fastcrw.com`; `http://localhost:3000` for self-hosted. |
|
||||
| `CRW_API_KEY` | crw-mcp, crw CLI | Bearer token for the remote server. |
|
||||
| `CRW_LOCAL` | crw-mcp | Truthy forces embedded mode, ignoring `CRW_API_URL`. |
|
||||
| `CRW_AUTH__API_KEYS` | server | Accepted keys (server side). |
|
||||
| `CRW_SEARCH__*` | server | Search config (`enabled`, `search_backend_url`, `timeout_ms`, `default_limit`, `max_limit`). |
|
||||
| `CRW_SEARXNG_URL` | CLI | Legacy local search backend URL. |
|
||||
|
||||
Sources: docs.fastcrw.com (REST API section), crates/crw-core/src/config.rs, crw-mcp README.
|
||||
Plugin should use `CRW_API_URL` / `CRW_API_KEY` to match upstream naming.
|
||||
|
||||
## Request body
|
||||
|
||||
Source: `SearchRequest` in crates/crw-core/src/types.rs (camelCase serde); blog/crw-search-api-release.md.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `query` | string, required | Max 2000 chars (handler validation). |
|
||||
| `limit` | int, optional | Default 5 (`default_limit`); clamped server-side to `1..=max_limit` (max 20). |
|
||||
| `lang` | string | SearXNG language, e.g. `en`, `de`, `auto`; validated as a language tag. |
|
||||
| `tbs` | enum | `qdr:h`, `qdr:d`, `qdr:w`, `qdr:m`, `qdr:y`. |
|
||||
| `sources` | array | `web`, `news`, `images`. Presence switches response to grouped form. |
|
||||
| `categories` | array, max 5 | `github`, `research`, `pdf`, or passthrough string. |
|
||||
| `scrapeOptions` | object | Scrapes each web result (timeout bounds 1..60000 ms). Adds `markdown` etc. to results. |
|
||||
| `answer`, `answerTopN`, `summarizeResults`, `maxCharsPerSource`, `llmApiKey`, `llmProvider`, `llmModel` | | LLM synthesis; need `scrapeOptions`. Not needed by this plugin. |
|
||||
|
||||
Minimal request is `{"query","limit"}`; `lang`/`tbs` are the only extras worth exposing.
|
||||
|
||||
## Response
|
||||
|
||||
Envelope `ApiResponse` (camelCase, absent fields omitted): `success`, `data`, `error`, `error_code`, `code` (v2 compat only), `warning`.
|
||||
|
||||
`data` is `SearchResponseData` = `{results, answer?, citations?, llmUsage?, warnings?}`. `results` is untagged: flat `SearchResult[]` or grouped `{web,news,images}`.
|
||||
|
||||
Hosted example (flat, per docs):
|
||||
|
||||
```json
|
||||
{"success": true, "data": [
|
||||
{"url": "https://...", "title": "...", "description": "...", "snippet": "...", "position": 1, "score": 9.5}
|
||||
]}
|
||||
```
|
||||
|
||||
Self-hosted (CLI parser handles `{"success": true, "data": {"results": [ ... ]}}`).
|
||||
|
||||
`SearchResult`: `url`, `title`, `description`, `snippet` (alias of description, always populated), `position` (u32), optional `score`, `publishedDate`, `category`, and (with scrapeOptions) `markdown`, `html`, `rawHtml`, `links`.
|
||||
|
||||
Conflict note: the docs example and CLI show flat `data[]` for hosted; the struct and CLI show `data.results` for the server. I did not call a live instance, so verify against the target server and keep the dual-shape parser.
|
||||
|
||||
## Errors
|
||||
|
||||
JSON body `{"success": false, "error": "<message>", "error_code": "<code>"}` (crates/crw-server/src/error.rs).
|
||||
|
||||
| HTTP | error_code | Cause |
|
||||
|---|---|---|
|
||||
| 400 | `invalid_request` | bad query/limit/lang |
|
||||
| 401 | (auth middleware message) | missing/invalid key |
|
||||
| 429 | `rate_limited` | rate limit |
|
||||
| 502 | `http_error` | search backend HTTP/JSON failure |
|
||||
| 504 | `timeout` | search timeout (default 15 s) |
|
||||
| 503 | `search_disabled` / `search_degraded` | search off or backend unhealthy |
|
||||
| 422 | `target_unreachable` | backend unreachable |
|
||||
| 500 | `internal_error` | other |
|
||||
|
||||
Backend hostnames/credentials are never leaked in errors.
|
||||
|
||||
## Limits
|
||||
|
||||
- `limit` max 20 (default 5); `query` max 2000 chars; `categories` max 5.
|
||||
- Server defaults: `rate_limit_rps` 10 (0 = unlimited), `request_timeout_secs` 60, search `timeout_ms` 15000.
|
||||
- Hosted credits: 1 per search, +1 per scraped result. Rate/size limits of hosted plan are not documented in sources found.
|
||||
- Self-hosted: search only works when a SearXNG-style backend is configured (`CRW_SEARCH__SEARCH_BACKEND_URL`); Docker Compose ships a search sidecar, the bare single binary does not. If unconfigured, expect 503 `search_disabled`. (An older blog post claimed self-hosted had no search; the current config.rs/self-hosting docs contradict it.)
|
||||
|
||||
## crw-mcp vs REST
|
||||
|
||||
crw-mcp 0.37.2 exposes 9 tools (`crw_scrape`, `crw_crawl`, `crw_check_crawl_status`, `crw_map`, `crw_extract`, `crw_check_extract_status`, `crw_cancel_extract`, `crw_search`, `crw_parse_file`). In proxy mode (`CRW_API_URL` set) it just forwards to the REST API with the Bearer key; in embedded mode `crw_search` only appears if a backend is configured. The REST call makes the MCP layer, its Node/binary install and the other 8 tools unnecessary for a search-only plugin; a plain `fetch` to `/v1/search` suffices.
|
||||
|
||||
## Sources
|
||||
|
||||
- https://docs.fastcrw.com/ (REST API), /self-hosting/, /mcp-clients/
|
||||
- https://github.com/us/crw: crates/crw-core/src/{types.rs,config.rs,error.rs}, crates/crw-server/src/{middleware.rs,error.rs,routes/search.rs}, crates/crw-cli/src/commands/search.rs, crates/crw-mcp README, blog/crw-search-api-release.md
|
||||
- https://registry.npmjs.org/crw-mcp
|
||||
@@ -0,0 +1,75 @@
|
||||
# Research: verified opencode v2 plugin API (issue #2)
|
||||
|
||||
Researched 2026-10-03. Sources (all first-party opencode docs):
|
||||
|
||||
- [S1] https://opencode.ai/v2/docs/build/plugins (build a plugin)
|
||||
- [S2] https://opencode.ai/v2/docs/plugins (configure plugins)
|
||||
- [S3] https://opencode.ai/v2/docs/config
|
||||
- [S4] https://opencode.ai/v2/docs/websearch
|
||||
- [S5] https://opencode.ai/v2/docs/build/plugins/migrate-v1
|
||||
- [S6] https://opencode.ai/v2/docs/api (HTTP API reference)
|
||||
|
||||
Caveat: pages were read through a fetch-and-summarise tool, so quotes are as returned by it, not byte-checked. The opencode GitHub source (anomalyco/opencode) was NOT inspected (code search needs login); anything needing exact TypeScript types is flagged "unverified" below.
|
||||
|
||||
## Verdict on the user's sketch
|
||||
|
||||
| Sketch element | Verdict | Evidence |
|
||||
|---|---|---|
|
||||
| `Plugin.define` from `"@opencode/plugin"` | RIGHT | [S1] `import { Plugin } from "@opencode/plugin"` / `export default Plugin.define({ id: "example", async setup(ctx) {...} })`. V1 package was `@opencode-ai/plugin` [S5]. |
|
||||
| `ctx.websearch.transform` | RIGHT | [S1] `await ctx.websearch.transform((editor) => { ... })` |
|
||||
| `editor.add` | RIGHT | [S1] `editor.add({ id: "internal", name: "Internal search", execute: async ({ query }, { signal }) => {...} })` |
|
||||
| `plugins[].package` / `options` in opencode.json | RIGHT | [S2][S3] key is `"plugins"` (V1 was `"plugin"`, tuples replaced by objects [S5]); object form `{ "package": "...", "options": {...} }` |
|
||||
| `websearch.provider` | RIGHT (config key) | [S3][S4] `{ "websearch": { "provider": "tavily" } }`; also `"random"`. Built-in ids: exa, firecrawl, parallel, tavily, tinyfish [S4]. |
|
||||
|
||||
Things the sketch likely missed / must add:
|
||||
- `id` is required on `Plugin.define` (stable, used for storage scoping/diagnostics) [S5].
|
||||
- A plugin-added provider is made active with `editor.default.set("<id>")` [S1]; `websearch.provider` in config selects by provider id, so `"provider": "<your-id>"` is the expected pairing (inference from [S1]+[S4]; docs do not state it explicitly -> verify at runtime).
|
||||
- Options are read as `ctx.options` (not passed as a setup arg) [S1].
|
||||
|
||||
## Sub-questions
|
||||
|
||||
### 1. Plugin entry / definition shape
|
||||
Each module has one default export with a unique id and `setup`:
|
||||
```ts
|
||||
import { Plugin } from "@opencode/plugin"
|
||||
export default Plugin.define({
|
||||
id: "example",
|
||||
async setup(ctx) { /* may return a cleanup function */ },
|
||||
})
|
||||
```
|
||||
[S1][S5]. Context: `ctx.app.version`, `ctx.location` (`directory`, optional `workspaceID`, project), `ctx.options`, `ctx.storage` (durable JSON, plugin-scoped), plus domains `ctx.session/permission/shell/tool/websearch/...` [S1][S5]. Return cleanup from `setup` for timers/sockets [S5]. V1 plugins do not run in V2; dual-support is possible via spreading `Plugin.define(...)` and adding a V1 `server()` [S1][S5].
|
||||
|
||||
### 2. Registration/loading (local path vs npm)
|
||||
`"plugins"` array in `opencode.json(c)`; entries are strings or objects [S2]:
|
||||
npm `"opencode-acme-plugin"`, `"opencode-acme-plugin@1.2.0"`, `"@acme/opencode-plugin"`; local `"./plugins/local"`, `"../shared/plugin.ts"`, `"/abs/plugin.ts"`, `"file:///..."`. Also auto-discovery of `.ts/.js` files and packages in `.opencode/plugins/` or `~/.config/opencode/plugins/` [S2]. CLI: `opencode plugin add|list|check|update|remove` [S2]. Disable via ids: `["*", "-opencode.provider.*", ...]` [S2]. Published package needs `"type":"module"`, `exports["."]` -> entry, dependency `@opencode/plugin` [S1].
|
||||
|
||||
### 3. Passing options
|
||||
```jsonc
|
||||
{ "plugins": [ { "package": "./plugins/company", "options": { "strict": true } } ] }
|
||||
```
|
||||
read via `ctx.options.strict` in `setup` [S1][S3]. Options type is untyped key/value (validate yourself).
|
||||
|
||||
### 4. Can a plugin provide/replace the websearch provider?
|
||||
Yes. [S1]:
|
||||
```ts
|
||||
await ctx.websearch.transform((editor) => {
|
||||
editor.add({ id: "internal", name: "Internal search",
|
||||
execute: async ({ query }, { signal }) => {
|
||||
const response = await fetch(`https://search.example.com?q=${query}`, { signal })
|
||||
return [{ url: "", title: "", content: "", time: {} }]
|
||||
} })
|
||||
editor.default.set("internal")
|
||||
})
|
||||
```
|
||||
Disable: `editor.default.set(false)`; re-evaluate: `await ctx.websearch.reload()` [S1]. The API reference lists add/default/remove operations [S6] (remove not shown in S1 example; exact editor type unverified).
|
||||
|
||||
### 5. How `websearch.provider` is selected
|
||||
Config `{ "websearch": { "provider": "<id>" | "random" } }`; `"random"` picks an available provider; `"websearch": false` disables search entirely [S3][S4]. Provider on HTTP 429 enters cooldown and search retries another provider [S4]. Provider API keys via `/connect` or env (e.g. `TAVILY_API_KEY`) [S4]. For a plugin provider, `editor.default.set(id)` is the plugin-side selection [S1]; interplay/precedence with the config `provider` key is not documented -> verify.
|
||||
|
||||
### 6. Required search result shape
|
||||
From [S1] the `execute` return is an array of `{ url, title, content, time }`; the example uses `time: {}` (object, inner shape undocumented). [S6] lists the same four fields. Unverified: whether `time` fields are optional, exact `time` keys (e.g. published/updated), and whether extra fields are allowed. `execute` receives `({ query }, { signal })`; `signal` is an AbortSignal and must be passed to fetch [S1].
|
||||
|
||||
## Open items (need source or runtime check)
|
||||
- Exact TS types for editor, `time`, and execute args (read anomalyco/opencode source or `@opencode/plugin` typings after install).
|
||||
- Precedence of config `websearch.provider` vs plugin `editor.default.set`.
|
||||
- [S6] suggested `POST /api/websearch/query` with `{query, provider?}` -> `{results, provider}`; this came from the summariser and is lower confidence.
|
||||
Reference in new issue
Block a user