Fix llama-server-fast context-size exhaustion breaking Auto Mode #49

2026-09-06T20:07:17Z

haylan commented

2026-09-06 20:07:17 +00:00

Real failure: "Auto Mode couldn't classify this action (Classifier stage 1 unavailable)". Reproduced directly against the server:

{"error":{"message":"[400]: request (6186 tokens) exceeds the available context size (4096 tokens), try increasing it"...

LLAMA_FAST_CTX_SIZE=8192 is the total across every LLAMA_FAST_PARALLEL slot, not per-request — the main model's own .env.example comment already calls this out; missed it when llama-server-fast was set up (#44). With PARALLEL=2 that's 4096/slot, too small for a real classifier call (hints + environment + recent tool-call history).

Fixed by dropping to a single slot (LLAMA_FAST_PARALLEL=1) rather than raising ctx-size — this service doesn't need concurrent classifier calls the way the main model needs concurrent chat sessions, so this costs no extra VRAM. The full 8192 now goes to the one slot.

Also fixes the deployment gap this exposed: the live server's .env already had LLAMA_FAST_PARALLEL=2 set explicitly from the original deploy, and scripts/update.sh's set_if_blank never touches an already-set value — so this fix wouldn't have taken effect without a manual .env edit. update.sh now syncs tracked config values (LLAMA_*, ports, timeouts — anything with a real default in .env.example) against .env on every run:

Already matches → left alone silently.
Differs → a conflict, shown all together on one screen (via gum — single static binary, no build step, cached in .cache/gum/) — you choose which keys take the new value; unpicked ones keep the server's current value. Falls back to a plain prompt if gum can't be fetched.
No TTY (cron/CI/piped) → hard error listing every conflict, unless --force is passed to accept every new value automatically.

Secrets and host-resolved values (blank in .env.example by design) are completely untouched by any of this, same as before.

docker compose config -q, bash -n, and an isolated-sandbox test of the exact scenario here (stale LLAMA_FAST_PARALLEL=2 vs git's 1) all validated — both the no-TTY-no---force (exits 1, prints the diff) and no-TTY---force (corrects the value, leaves a secret untouched) paths confirmed working. Not verified: the interactive gum path itself needs a real TTY, which couldn't be simulated in this sandbox — worth confirming for real on the server, including that gum's release asset naming matches what the script expects.

Refs #5

🤖 Generated with Claude Code

https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx

Real failure: "Auto Mode couldn't classify this action (Classifier stage 1 unavailable)". Reproduced directly against the server: ``` {"error":{"message":"[400]: request (6186 tokens) exceeds the available context size (4096 tokens), try increasing it"... ``` `LLAMA_FAST_CTX_SIZE=8192` is the **total** across every `LLAMA_FAST_PARALLEL` slot, not per-request — the main model's own `.env.example` comment already calls this out; missed it when `llama-server-fast` was set up (#44). With `PARALLEL=2` that's 4096/slot, too small for a real classifier call (hints + environment + recent tool-call history). Fixed by dropping to a single slot (`LLAMA_FAST_PARALLEL=1`) rather than raising ctx-size — this service doesn't need concurrent classifier calls the way the main model needs concurrent chat sessions, so this costs no extra VRAM. The full 8192 now goes to the one slot. **Also fixes the deployment gap this exposed**: the live server's `.env` already had `LLAMA_FAST_PARALLEL=2` set explicitly from the original deploy, and `scripts/update.sh`'s `set_if_blank` never touches an already-set value — so this fix wouldn't have taken effect without a manual `.env` edit. `update.sh` now syncs tracked config values (`LLAMA_*`, ports, timeouts — anything with a real default in `.env.example`) against `.env` on every run: - Already matches → left alone silently. - Differs → a **conflict**, shown all together on one screen (via `gum` — single static binary, no build step, cached in `.cache/gum/`) — you choose which keys take the new value; unpicked ones keep the server's current value. Falls back to a plain prompt if `gum` can't be fetched. - No TTY (cron/CI/piped) → hard error listing every conflict, unless `--force` is passed to accept every new value automatically. Secrets and host-resolved values (blank in `.env.example` by design) are completely untouched by any of this, same as before. `docker compose config -q`, `bash -n`, and an isolated-sandbox test of the exact scenario here (stale `LLAMA_FAST_PARALLEL=2` vs git's `1`) all validated — both the no-TTY-no-`--force` (exits 1, prints the diff) and no-TTY-`--force` (corrects the value, leaves a secret untouched) paths confirmed working. **Not verified**: the interactive `gum` path itself needs a real TTY, which couldn't be simulated in this sandbox — worth confirming for real on the server, including that gum's release asset naming matches what the script expects. Refs #5 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx

haylan added 1 commit 2026-09-06 20:07:17 +00:00

fix: llama-server-fast context-size exhaustion breaking Auto Mode classifier 52a92f6508

Real failure: "Auto Mode couldn't classify this action (Classifier
stage 1 unavailable)". Reproduced directly against the server:

  {"error":{"message":"[400]: request (6186 tokens) exceeds the
  available context size (4096 tokens)"...

LLAMA_FAST_CTX_SIZE=8192 is the TOTAL across every LLAMA_FAST_PARALLEL
slot, not per-request — the main model's own .env.example comment
already calls this out, missed it when llama-server-fast was set up
(#44). With PARALLEL=2 that's 4096/slot, too small for a real
classifier call (hints + environment + recent tool-call history).

Fixed by dropping to a single slot (LLAMA_FAST_PARALLEL=1) rather than
raising ctx-size — this service doesn't need concurrent classifier
calls the way the main model needs concurrent chat sessions, so this
costs no extra VRAM. The full 8192 now goes to the one slot.

docker compose config -q validated.

Refs #5

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx

haylan added 1 commit 2026-09-06 20:10:54 +00:00

feat: update.sh force-syncs tracked config from .env.example 9def240a8e

Follows directly from the previous commit's caveat: this PR's own fix
(LLAMA_FAST_PARALLEL=2 -> 1) wouldn't have taken effect on the server
without a manual .env edit, because set_if_blank never touches an
already-set value — by design, for secrets, but the same logic was
silently protecting stale copies of ordinary tunable config too.

Every KEY=VALUE line in .env.example with a real (non-blank) default
is now force-synced into .env on every run. Secrets and host-resolved
values are unaffected — .env.example already leaves those blank on
purpose, so the sync loop naturally skips them and they keep going
through set_if_blank as before.

Trade-off, called out in both the script's header and the sync loop's
own comment: there's no such thing as a persistent server-only
override for these keys anymore — a hand-edited value not reflected
in git gets reverted on the next run. That's the intended behavior.

Verified against a simulated stale .env matching the real scenario
from this PR: LLAMA_FAST_PARALLEL correctly overwritten 2 -> 1, an
OMNIROUTE secret left untouched. bash -n and docker compose config -q
both pass.

Refs #5

haylan added 1 commit 2026-09-06 20:20:31 +00:00

feat: interactive per-key conflict resolution in update.sh's config sync ea7b05fb99

Replaces the previous commit's blind force-overwrite with a real
choice, per user feedback: force-overwriting server config without
asking was the wrong default.

- A tracked config value (real default in .env.example) that already
  matches .env is left alone silently — no prompt, no noise.
- A value that DIFFERS is a conflict, shown on one screen (all
  conflicts together, not one prompt per key) via gum
  (charmbracelet/gum) — single static binary fetched as a release
  tarball into .cache/gum/ (gitignored), no build step, no package
  manager dependency. Falls back to a plain read-based prompt if gum
  can't be fetched (offline, unsupported arch).
- Non-interactive (no TTY — cron, CI, piped): any conflict is a hard
  error (exit 1, lists every conflicting key) unless --force is
  passed, which accepts every new value automatically — matches how
  this PR's own fix needs to land unattended.
- Secrets and host-resolved values are completely unaffected either
  way — untouched by this loop, same as before.

Verified in an isolated sandbox against the exact scenario from this
PR (stale LLAMA_FAST_PARALLEL=2 vs git's 1):
- no TTY, no --force: exits 1, prints the diff, doesn't touch .env
- no TTY, --force: LLAMA_FAST_PARALLEL corrected 2 -> 1, an
  OMNIROUTE secret confirmed untouched (not regenerated)
docker compose config -q still passes.

Not verified: the interactive gum path itself (needs a real TTY,
couldn't allocate a pty in this sandbox) — worth confirming for real
on the server, including that gum's release asset naming actually
matches what ensure_gum() expects.

Refs #5

haylan merged commit 5b7548dc7c into main

2026-09-06 20:21:53 +00:00

haylan deleted branch fix-fastmodel-context-size

2026-09-06 20:21:53 +00:00

haylan referenced this pull request

2026-09-06 20:27:21 +00:00

Verify the stack on the real Radeon R9700 box (GPU passthrough + Lazytainer idle-stop) #5

haylan referenced this pull request

2026-09-06 20:33:46 +00:00

Local AI inference stack: llama.cpp + Qwen3.8-27B on Radeon R9700 #1

haylan referenced this pull request

2026-09-06 20:34:29 +00:00