haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

opencode: context size and compaction settings for the litellm/qwen3.8-27b-local stack #26

New Issue
Open
opened 2026-09-03 04:19:22 +00:00 by haylan · 1 comment
haylan commented 2026-09-03 04:19:22 +00:00
Owner
Copy Link
Copy Source

Destination

opencode (pointed at http://litellm.home, model qwen3.8-27b-local) uses
correct context-window and compaction behavior for the deployed
llama.cpp/Qwen3.8-27B backend — with those numbers derived from this
repo's existing config (LLAMA_CTX_SIZE, litellm's max_tokens) rather
than hand-duplicated and drifting in opencode.json. opencode's provider
config (limit.context/limit.output) has no documented runtime
auto-discovery (confirmed in docs/research/opencode-cli-setup.md §5) —
so "auto-discovery" here means deciding how much of that gap this repo
closes with its own tooling (a generator script, docs, or just accepted
manual upkeep) vs. leaving it manual.

Notes

  • Domain docs: none yet (CONTEXT.md/docs/adr/ don't exist in this repo
    — proceed silently per docs/agents/domain.md).
  • Read first: docs/research/opencode-cli-setup.md (opencode config shape),
    docs/research/qwen3.8-27b-quant.md (KV-cache/VRAM math), and
    docs/research/opencode-auto-compact.md (compaction trigger formula, on
    the unmerged research/opencode-auto-compact branch — merge or
    cherry-pick before relying on it).
  • Live config: ~/.config/opencode/opencode.json on this dev machine
    carries the real litellm provider block, model default, and compaction
    settings — see Decisions so far below.
  • Current backend facts: docker-compose.yml's llama-server runs with
    --ctx-size ${LLAMA_CTX_SIZE:-131072} (merged via #30); litellm's
    max_tokens is 16384. opencode reaches the model only through litellm,
    per issue #15.
  • Invoke /research for opencode-internals questions, /grilling +
    /domain-modeling for decisions.

Decisions so far

  • How does opencode's auto-compact actually work? —
    single global compaction block, triggers at usedTokens >= limit.context - min(20000, limit.output),
    silently disabled if limit.context is unset. Findings: docs/research/opencode-auto-compact.md
    on branch research/opencode-auto-compact.
  • Should context/output limits be generated or hand-maintained? —
    hand-maintained, no generator script: values live in ~/.config/opencode/opencode.json with a
    comment pointing at the source-of-truth files to check by eye when they change.
  • Point opencode at litellm.home with a virtual key —
    done. Provider block added, set as opencode's default model, smoke-tested end to end. Surfaced a
    live bug in the process: LITELLM_MASTER_KEY doesn't start with sk-, so litellm rejects it on
    every API endpoint — blocks minting new virtual keys via the documented API flow until fixed.
  • Context bumped to 128K end to end: LLAMA_CTX_SIZE raised 65536 → 131072 on the server
    directly, ctx-size-128k branch merged into main (PR #30) so the repo's checked-in default now
    matches, and ~/.config/opencode/opencode.json's limit.context updated to match — all three
    reconciled, re-smoke-tested.

Not yet specified

  • Whether litellm's request-priority/caching (router_settings) has any interaction with opencode's
    own context bookkeeping.
  • Whether the LITELLM_MASTER_KEY sk-prefix bug (found while resolving #29) is worth its own fix —
    not yet a ticket; small enough it may just get fixed inline whenever someone's next in
    litellm-config.yaml/.env.

Out of scope

  • Tool-calling reliability of Qwen3.8-27B against llama.cpp/opencode (broken llama.cpp Jinja/tool-
    parser bugs, documented in docs/research/opencode-cli-setup.md §4-5 and
    docs/research/qwen3.8-27b-tool-calling.md) — orthogonal to context-size/compaction.
  • opencode's websearch tool for the local model: confirmed unavailable by opencode's own design
    (hosted-provider-only, or a paid Exa/Parallel key) — not something this stack's config can turn
    on. User chose to leave it off rather than build an MCP bridge to this repo's SearXNG endpoint or
    bring in a paid third-party search API.
## Destination opencode (pointed at http://litellm.home, model `qwen3.8-27b-local`) uses correct context-window and compaction behavior for the deployed llama.cpp/Qwen3.8-27B backend — with those numbers **derived from** this repo's existing config (`LLAMA_CTX_SIZE`, litellm's `max_tokens`) rather than hand-duplicated and drifting in `opencode.json`. opencode's provider config (`limit.context`/`limit.output`) has no documented runtime auto-discovery (confirmed in `docs/research/opencode-cli-setup.md` §5) — so "auto-discovery" here means deciding how much of that gap this repo closes with its own tooling (a generator script, docs, or just accepted manual upkeep) vs. leaving it manual. ## Notes - Domain docs: none yet (`CONTEXT.md`/`docs/adr/` don't exist in this repo — proceed silently per `docs/agents/domain.md`). - Read first: `docs/research/opencode-cli-setup.md` (opencode config shape), `docs/research/qwen3.8-27b-quant.md` (KV-cache/VRAM math), and `docs/research/opencode-auto-compact.md` (compaction trigger formula, on the unmerged `research/opencode-auto-compact` branch — merge or cherry-pick before relying on it). - Live config: `~/.config/opencode/opencode.json` on this dev machine carries the real `litellm` provider block, model default, and compaction settings — see Decisions so far below. - Current backend facts: `docker-compose.yml`'s `llama-server` runs with `--ctx-size ${LLAMA_CTX_SIZE:-131072}` (merged via #30); litellm's `max_tokens` is 16384. opencode reaches the model only through litellm, per issue #15. - Invoke `/research` for opencode-internals questions, `/grilling` + `/domain-modeling` for decisions. ## Decisions so far - [How does opencode's auto-compact actually work?](https://git.arthurerlich.de/haylan/LLM-Server/issues/27) — single global `compaction` block, triggers at `usedTokens >= limit.context - min(20000, limit.output)`, silently disabled if `limit.context` is unset. Findings: `docs/research/opencode-auto-compact.md` on branch `research/opencode-auto-compact`. - [Should context/output limits be generated or hand-maintained?](https://git.arthurerlich.de/haylan/LLM-Server/issues/28) — hand-maintained, no generator script: values live in `~/.config/opencode/opencode.json` with a comment pointing at the source-of-truth files to check by eye when they change. - [Point opencode at litellm.home with a virtual key](https://git.arthurerlich.de/haylan/LLM-Server/issues/29) — done. Provider block added, set as opencode's default model, smoke-tested end to end. Surfaced a live bug in the process: `LITELLM_MASTER_KEY` doesn't start with `sk-`, so litellm rejects it on every API endpoint — blocks minting new virtual keys via the documented API flow until fixed. - **Context bumped to 128K end to end**: `LLAMA_CTX_SIZE` raised 65536 → 131072 on the server directly, `ctx-size-128k` branch merged into `main` (PR #30) so the repo's checked-in default now matches, and `~/.config/opencode/opencode.json`'s `limit.context` updated to match — all three reconciled, re-smoke-tested. ## Not yet specified - Whether litellm's request-priority/caching (`router_settings`) has any interaction with opencode's own context bookkeeping. - Whether the `LITELLM_MASTER_KEY` sk-prefix bug (found while resolving #29) is worth its own fix — not yet a ticket; small enough it may just get fixed inline whenever someone's next in `litellm-config.yaml`/`.env`. ## Out of scope - Tool-calling reliability of Qwen3.8-27B against llama.cpp/opencode (broken llama.cpp Jinja/tool- parser bugs, documented in `docs/research/opencode-cli-setup.md` §4-5 and `docs/research/qwen3.8-27b-tool-calling.md`) — orthogonal to context-size/compaction. - opencode's `websearch` tool for the local model: confirmed unavailable by opencode's own design (hosted-provider-only, or a paid Exa/Parallel key) — not something this stack's config can turn on. User chose to leave it off rather than build an MCP bridge to this repo's SearXNG endpoint or bring in a paid third-party search API.
haylan added the wayfinder:map label 2026-09-03 04:19:22 +00:00
haylan commented 2026-09-03 04:43:40 +00:00
Author
Owner
Copy Link
Copy Source

limit.context in ~/.config/opencode/opencode.json bumped 65536 -> 131072: the server's live LLAMA_CTX_SIZE was changed directly on the box, ahead of merging the ctx-size-128k branch into this repo. Re-smoke-tested (opencode run round-trip), works. Repo's docker-compose.yml/.env.example still show the old default until that branch/PR is merged — worth reconciling so the checked-in default matches what's actually deployed.

`limit.context` in ~/.config/opencode/opencode.json bumped 65536 -> 131072: the server's live LLAMA_CTX_SIZE was changed directly on the box, ahead of merging the ctx-size-128k branch into this repo. Re-smoke-tested (`opencode run` round-trip), works. Repo's docker-compose.yml/.env.example still show the old default until that branch/PR is merged — worth reconciling so the checked-in default matches what's actually deployed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:map
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: haylan/LLM-Server#26