haylan/LLM-Server
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Audit: contradictions and duplicates across docs and code comments #56

New Issue
Closed
opened 2026-09-24 19:08:18 +00:00 by haylan · 1 comment
haylan commented 2026-09-24 19:08:18 +00:00
Owner
Copy link
Copy source

Part of #55

Question

Check every factual claim in README.md, docs/ (excluding docs/research/), and comments in docker-compose.yml, .env.example, scripts/ against the actual source. For each: verdict true / false (what the source says instead) / duplicate-of (where else it's stated). AFK. Output: a table in the resolution comment.

Part of #55 ## Question Check every factual claim in README.md, docs/ (excluding docs/research/), and comments in docker-compose.yml, .env.example, scripts/ against the actual source. For each: verdict true / false (what the source says instead) / duplicate-of (where else it's stated). AFK. Output: a table in the resolution comment.
haylan added the wayfinder:task label 2026-09-24 19:08:18 +00:00
haylan added a new dependency 2026-09-24 19:08:32 +00:00
#59 Glossary: pin down domain terms in CONTEXT.md
haylan added a new dependency 2026-09-24 19:08:33 +00:00
#60 Target doc structure
haylan self-assigned this 2026-09-24 19:09:04 +00:00
haylan commented 2026-09-24 19:11:49 +00:00
Author
Owner
Copy link
Copy source

Resolution: audit of docs and code comments against source

Scope: README.md, docs/ (excluding research/ and agents/), comments in docker-compose.yml, .env.example, scripts/*.sh, checked against compose/scripts as of 8ae3210. Verdicts: TRUE / FALSE / DUPLICATE / STALE / REMOVED-FEATURE (ComfyUI / Qwen-Image / Open WebUI, not audited in depth). Line refs are for this audit only; they don't belong in docs.

Headline findings

  1. The dashboard port is published. Compose maps OMNIROUTE_DASHBOARD_PORT to the host. Four places say it never is.
  2. OMNIROUTE_PORT:-4000 doesn't exist. Compose publishes OMNIROUTE_API_PORT 1:1, so it's 20129:20129 by default with no 4000 mapping. Every client doc and .env.example still point at 4000.
  3. Lazytainer auto-suspend doesn't work in practice (research doc on lazytainer idle-stop, #40, #5 closing comment). README, .env.example and compose still present it as working. After switch-model.sh is removed, nothing in the code will record this.
  4. Two sources explain the classifier's VRAM fix differently. Compose says a missing flash-attn was the real cause. qwen-code.md calls flash-attn a "wrong turn" and credits partial offload.
  5. .env.example says qdrant isn't exposed off-box. That's false: compose publishes 6333 on the host and qdrant has no auth, so anyone on the LAN can reach it without logging in. This is a security issue.

README.md

Location Claim gist Verdict Detail
README:intro Lazytainer auto-suspends llama-server when idle FALSE Labels are set, but the idle-stop never fires: OmniRoute health pings count as traffic (research doc on lazytainer idle-stop, #40, #5)
README:intro Wayfinder map #1 has the architecture rationale STALE #1 is still open but predates OmniRoute (LiteLLM/Open WebUI era)
README:Quickstart What update.sh does: creates .env, fills secrets, resolves SEARXNG_LAN_IP, downloads the model, pulls/builds/brings up TRUE (incomplete) It also runs git pull, syncs non-blank config values from .env.example with a conflict prompt (--force in non-interactive runs), resolves the GPU GIDs, and downloads the classifier model and the ComfyUI files
README:Quickstart Safe to re-run; recreates only what changed TRUE
README:Quickstart llama.cpp API is internal-only TRUE, DUPLICATE Compose uses expose only. See the cluster "llama-server has no host port"
README:Quickstart / risk Qwen3.8 tool-calling isn't verified STALE? #5 closed with inference confirmed end-to-end through OmniRoute. Tool-calling wasn't explicitly re-tested. DUPLICATE of index.md and opencode.md
README:AI gateway update.sh handles most OmniRoute secrets; keys are minted by hand TRUE, DUPLICATE See the key-minting cluster
README:AI gateway Gateway at ${OMNIROUTE_PORT:-4000}/v1 STALE/FALSE That var doesn't exist. Compose publishes OMNIROUTE_API_PORT (default 20129) 1:1
README:AI gateway Dashboard is never published FALSE Compose publishes OMNIROUTE_DASHBOARD_PORT to the host
README:AI gateway Not smoke-tested on real hardware STALE #5 closed after real-hardware verification. #17 closed as obsolete (LiteLLM)
README:AI gateway OmniRoute stealth/MITM features aren't used TRUE Nothing in compose enables them. The upstream claim itself can't be checked from the repo
README:Web search Gateway fronts SearXNG search TRUE (partly unverifiable) extra_hosts and the private-URL flags support it. The provider itself is registered in the dashboard, not in the repo. Its pointer goes to the research doc on LiteLLM + SearXNG search (LiteLLM naming)

docs/network-access.md

Location Claim gist Verdict Detail
intro The stack has no chat UI; every client is a coding CLI FALSE, REMOVED-FEATURE open-webui is published on 8008. The claim becomes true once Open WebUI is removed
intro Publishes the gateway API port and relies on the existing NPM TRUE
llama.cpp LAN-only Inference API on ${LLAMA_PORT:-8080} STALE LLAMA_PORT is gone. Port 8080 is container-internal (expose) and isn't published at all, so "LAN-only" understates it: it isn't on the LAN either
llama.cpp LAN-only Unauthenticated; CLIs go through the gateway TRUE, DUPLICATE
AI gateway Both hostnames point at ${OMNIROUTE_PORT:-4000} STALE/FALSE Should be OMNIROUTE_API_PORT
AI gateway Dashboard is never registered in NPM or published by compose; the split port is "structural, no network route" FALSE Compose publishes the dashboard port, and the compose comment says "both published directly". Not registering it in NPM is the only thing keeping it off the internet. It is reachable from the LAN
AI gateway Every gateway call needs an API key TRUE (upstream behaviour)
Neo4j Browser on 7474 and bolt on 7687, published TRUE
Qdrant Dashboard on raw LAN port 6333 TRUE Also no auth (see .env.example:RAG)
(missing) — gap The doc doesn't mention the published comfyui 8138 or open-webui 8008 (REMOVED-FEATURE)

docs/proxy-key-onboarding.md

Location Claim gist Verdict Detail
intro No in-stack workload needs a key FALSE, REMOVED-FEATURE open-webui needs OPENWEBUI_OMNIROUTE_KEY. True once it's removed. DUPLICATE of update.sh's key-minting comment
intro / closing There's no scripted mint because POST /api/keys needs a session, not verified (#37) TRUE, DUPLICATE update.sh has no mint step. #37 is closed, so the "see #37" pointer is stale
Create the key, step 1 Dashboard port is never published, so use the container IP or an SSH tunnel FALSE It's published, so http://<host>:<OMNIROUTE_DASHBOARD_PORT> works directly. The docker inspect template is also wrong: it uses .Networks.ai_stack, but the network is named ai-stack and Compose prefixes it with the project name (e.g. llm-server_ai-stack)
Create the key, step 1 Hardcodes 20128 — Restates a value. Name the key instead
retire/rotate Revoke by hand TRUE

docs/proxy-request-priority.md

Location Claim gist Verdict Detail
banner The Mechanism section onward is LiteLLM and stale TRUE (self-declared)
intro One local model instance serves every workload FALSE qwen-classifier is a second llama.cpp instance on the same GPU
Tiers Priority is assigned per virtual key STALE No mechanism exists in OmniRoute config or compose
Mechanism / Timeout LiteLLM scheduler, Redis, and timeouts; also a "#14 (compose authoring)" smoke-test task STALE LiteLLM has been removed
Lazytainer interaction Queueing happens inside LiteLLM and Lazytainer wakes the container STALE / FALSE LiteLLM is gone. Lazytainer's detector can't tell OmniRoute health pings from real traffic (#40), and its wake-on-traffic was never confirmed

docs/coding-cli-setup/*

Location Claim gist Verdict Detail
index:intro llama.cpp port is internal-only TRUE, DUPLICATE
index:endpoints and summary table; claude-code, kimi-cli, opencode, qwen-code (x4) ${OMNIROUTE_PORT:-4000} STALE/FALSE Should be OMNIROUTE_API_PORT (20129 by default). Occurs about 12 times
index Model file / qwen3.8-27b-local naming TRUE Model file is the compose default. The model name is set in the dashboard
index One key per CLI; these are high-priority per proxy-request-priority.md TRUE / STALE The priority doc it points to is stale
index Run the #5 / #17 smoke tests before relying on this STALE Both issues are closed (#5 verified, #17 obsolete)
index:further reading Semaphore 30s reject for llama-server / qwen-classifier TRUE (per research doc) Dashboard setting, can't check from source
claude-code / opencode Needs --jinja, already set TRUE, DUPLICATE Both llama-server and qwen-classifier pass --jinja
opencode Per-slot context = CTX/PARALLEL = 131072 TRUE, DUPLICATE Also in qwen-code.md and .env.example:LLAMA_PARALLEL
qwen-code:why a second model Classifier queued behind 2 GPU slots; CPU-only too slow, blew past "OmniRoute's request timeout" TRUE, DUPLICATE; wording conflicts Compose says "OmniRoute's 60s timeout" instead. Neither figure matches the current REQUEST_TIMEOUT_MS, and the 60s was most likely the per-connection timeoutMs. Pick one wording
qwen-code:why a second model The fix was partial offload; batch-size tuning and flash-attn were "wrong turns" FALSE (contradicts compose) The compose qwen-classifier comment says flash-attn was the actual cause and batch-size the wrong lever, and never calls partial offload the lever. --n-gpu-layers 28 is set, but the comment gives no reason for it
qwen-code:settings bullets Classifier --parallel 1, 65536 ctx, qwen-code transcript caps TRUE, DUPLICATE Matches compose. The transcript-cap reasoning is repeated word for word in compose
qwen-code:auto mode tuning "see network-access.md for reaching the dashboard API" FALSE pointer network-access.md has no access steps. They're in proxy-key-onboarding.md, and they're wrong there too (see above)
qwen-code:auto mode tuning Classifier timeoutMs 120000 and "20-minute" main connection unverifiable Dashboard-only settings, nothing in source

docs/server-planing.md

Location Claim gist Verdict Detail
whole 4x R9700 rack build plan not source-checkable A purchasing/planning doc, not config. The running stack is 1x R9700

docker-compose.yml (comments)

Location Claim gist Verdict Detail
llama-server:group_add Numeric GIDs, not names TRUE, DUPLICATE See the GID cluster
llama-server:environment GPU_MAX_HW_QUEUES=1 works around ROCm#5706 TRUE, DUPLICATE Repeated in qwen-classifier. Confirmed fixed in #5
llama-server:--cache-reuse Targets the root cause of the SSE timeout; pairs with promptCacheAffinity TRUE promptCacheAffinity is a dashboard default and can't be checked from source
llama-server:expose No host port; reached only via omniroute (#15) TRUE, DUPLICATE
llama-server:labels Idle-timeout tuning here; raise it if 15 min is too eager STALE The idle-stop never fires (#40), so tuning the timeout has no effect
qwen-classifier header Queueing, CPU-only, context sizing, VRAM, reasoning off TRUE / DUPLICATE / conflict Duplicates qwen-code.md, and the two disagree on the VRAM fix (see above). "Re-verify with rocm-smi after deploy" is still an open to-do
downloader / downloader-classifier One-off job, run by update.sh every time, test -f guard TRUE
downloader-comfyui, comfyui — REMOVED-FEATURE
omniroute header Replaces litellm; provider routing registered in the dashboard, not in the repo TRUE, DUPLICATE Also in update.sh's header comment. The LiteLLM history belongs in docs
omniroute:split port Dashboard and API are both published TRUE Contradicts README, network-access.md, proxy-key-onboarding.md and .env.example
omniroute:private URLs, WS bridge Needed for local providers; WS bridge secret is required TRUE, DUPLICATE (WS bridge also in .env.example)
omniroute:OMNIROUTE_MEMORY_MB "every client here is a coding CLI" FALSE, REMOVED-FEATURE open-webui is a client too. The claim becomes true once it's removed
omniroute:STREAM_IDLE_TIMEOUT_MS "see LLAMA_PARALLEL above" FALSE pointer The LLAMA_PARALLEL explanation is in .env.example, not above in compose. The rest DUPLICATES .env.example's timeout comments
omniroute:REQUEST_TIMEOUT_MS First non-ping SSE event deadline; value raised past the default TRUE, DUPLICATE .env.example repeats it
omniroute:extra_hosts "Same reasoning as litellm's extra_hosts entry below" STALE The litellm service is gone, so there's no entry below. The real reason is only in update.sh's SEARXNG comment
omniroute:mem_limit, stop_grace_period, healthcheck Matches the Docker guide; image has only node TRUE
lazytainer Why not host networking; joins ai-stack TRUE The mechanism is fine, but it doesn't solve the idle-stop problem (#40)
qdrant "see docs/agents/... (wayfinder)"; "published directly like comfyui above" FALSE pointer, REMOVED-FEATURE ref docs/agents/ has no RAG doc. The comfyui comparison will go when comfyui does
open-webui — REMOVED-FEATURE

.env.example (comments)

Location Claim gist Verdict Detail
header Values are compose defaults; "only uncomment/change what you want to override" FALSE Nothing is commented out. Every non-blank value is live, and update.sh syncs them into .env as tracked config (it prompts on conflicts), so editing .env by hand gets flagged on the next run. Compose also has no default for any secret
LLAMA_GPU_LAYERS 999 = all layers; this is the RAM-offload knob; mmap fallback TRUE
LLAMA_CTX_SIZE 262144 is the true max; q8_0 KV fits TRUE (compose sets q8_0)
LLAMA_PARALLEL 2 slots; llama.cpp queue has no timeout; per-slot math TRUE, DUPLICATE Per-slot math is also in opencode.md and qwen-code.md. "Handoff notes in the issue tracker" is a vague pointer
LLAMA_CLASSIFIER_MODEL_FILE See the compose comment TRUE
Lazytainer Seconds before llama-server is stopped FALSE in practice Configured, but it never fires (#40)
SearXNG header Pointer to the research doc on LiteLLM + SearXNG search STALE naming The doc exists but is named after LiteLLM
SEARXNG_LAN_IP Resolved by update.sh TRUE, DUPLICATE
OmniRoute block OMNIROUTE_PORT is published at 4000 and mapped onto API_PORT; the dashboard is never published STALE / FALSE No OMNIROUTE_PORT variable exists. Compose maps OMNIROUTE_API_PORT:OMNIROUTE_API_PORT and publishes the dashboard
timeouts STREAM_IDLE and REQUEST semantics TRUE, DUPLICATE Also in the compose omniroute comments
secrets Filled by update.sh TRUE
OMNIROUTE_STORAGE_ENCRYPTION_KEY Don't change it after the first run TRUE, DUPLICATE Also in update.sh. It's a data-loss warning, so it stays inline. The LiteLLM aside is stale
per-workload keys No /key/generate equivalent; mint by hand (#37) TRUE, DUPLICATE /key/generate is LiteLLM vocabulary
HOST_*_GID Sits under the "ComfyUI" header; shared by "llama-server, llama-server-fast, comfyui" STALE, misfiled llama-server-fast is gone. The actual users are llama-server, qwen-classifier and comfyui (the last is REMOVED-FEATURE). It belongs under its own GPU header
COMFYUI_*, OPENWEBUI_OMNIROUTE_KEY — REMOVED-FEATURE
RAG "No auth on qdrant, same trust boundary as llama-server: ai-stack not exposed off-box" FALSE (security) qdrant publishes 6333 on the host's interfaces, so it's reachable from the LAN without auth. llama-server has no published port at all, so the two aren't in the same trust boundary

scripts/update.sh (comments)

Location Claim gist Verdict Detail
header The one command to run after any change; safe to re-run TRUE
header:config sync Syncs non-blank values; interactive gum prompt; errors without a TTY unless --force; blank secrets go through set_if_blank TRUE The secrets list leaves out NEO4J_PASSWORD, which is also handled by set_if_blank
header:omniroute config Lives in the dashboard, not a checked-in file like litellm-config.yaml TRUE, DUPLICATE Also in compose's omniroute header. The LiteLLM reference is history
header:ponytail No rollback TRUE
git pull first Self-update race (#5) TRUE
gum Vendored tarball checked in; download fallback; cached in .cache/gum TRUE Only the x86_64 tarball is vendored, which is consistent with "other archs" falling back to the download
set_if_blank Handles missing, blank and set TRUE
secrets Never change the storage key TRUE, DUPLICATE
SEARXNG search.home isn't resolvable inside the container TRUE, DUPLICATE This is the only real explanation for compose's extra_hosts
GIDs Numeric GID rationale TRUE, DUPLICATE (x3)
model download Also runs downloader-comfyui REMOVED-FEATURE
key-mint ponytail No in-stack workload needs a key FALSE, REMOVED-FEATURE open-webui needs one. The rest DUPLICATES proxy-key-onboarding.md

scripts/switch-model.sh

Location Claim gist Verdict Detail
whole script Swaps GPU residency between qwen and comfyui REMOVED-FEATURE
header Lazytainer's idle-stop can't work: its source-blind packet counter sees OmniRoute health pings as traffic TRUE This is the only statement in code or docs that lazytainer doesn't work. It has to move into docs before the script is deleted. It contradicts README, .env.example:Lazytainer and compose:lazytainer labels

Outside scope, noticed in passing

  • The .gitignore comment refers to scripts/ingest-memory.sh, which doesn't exist (STALE). The file also has .qwen/tmp without a trailing newline.

Duplicate clusters

  • Dashboard not published (all FALSE): README:AI gateway · network-access:AI gateway · proxy-key-onboarding:step 1 · .env.example:OmniRoute block. Compose's omniroute split-port comment is the only correct one.
  • OMNIROUTE_PORT:-4000 (STALE): README:AI gateway · network-access:AI gateway · .env.example:OmniRoute block · coding-cli-setup index (endpoints and table), claude-code, kimi-cli, opencode, qwen-code (x4).
  • Lazytainer idle-stop: "works" in README:intro · .env.example:Lazytainer · compose:llama-server labels · proxy-request-priority:Lazytainer interaction. "Doesn't work" only in switch-model.sh's header and the research doc on lazytainer idle-stop.
  • llama-server has no host port / internal-only: README:Quickstart · README:AI gateway · network-access:llama.cpp section · coding-cli index:intro · compose:llama-server expose · compose:lazytainer.
  • Numeric GID rationale: compose:llama-server group_add · compose:comfyui group_add · .env.example:HOST_*_GID · update.sh:GIDs.
  • GPU_MAX_HW_QUEUES / ROCm#5706: compose:llama-server · compose:qwen-classifier.
  • Classifier rationale (queueing, CPU-only, context caps, VRAM): qwen-code:why a second model and settings bullets · compose:qwen-classifier header · .env.example:LLAMA_CLASSIFIER_MODEL_FILE (pointer only). The first two conflict on the VRAM fix.
  • Per-slot context math: opencode.md · qwen-code:settings bullets · .env.example:LLAMA_PARALLEL.
  • --jinja required: claude-code.md · opencode.md.
  • Tool-calling risk: README:risk · coding-cli index:warning · opencode:risks.
  • No scripted key mint / session auth (#37): README:AI gateway · proxy-key-onboarding:intro and closing · .env.example:per-workload keys · update.sh:key-mint ponytail.
  • "No in-stack workload needs a key" (FALSE until Open WebUI is removed): proxy-key-onboarding:intro · update.sh:key-mint ponytail.
  • "Every client is a coding CLI / no chat UI" (FALSE until Open WebUI is removed): network-access:intro · compose:OMNIROUTE_MEMORY_MB.
  • Provider config lives in the dashboard, not the repo: compose:omniroute header · update.sh:header · compose:comfyui (REMOVED).
  • OmniRoute timeouts (STREAM_IDLE / REQUEST): compose:omniroute env comments · .env.example:timeouts.
  • Storage encryption key must never change: .env.example · update.sh:secrets. It's a data-loss warning, so it stays inline.
  • WS bridge secret required: compose:omniroute · .env.example.
  • search.home unresolvable in the container: update.sh:SEARXNG · .env.example:SEARXNG_LAN_IP · compose:omniroute extra_hosts (stale litellm pointer) · README:Quickstart.
  • LiteLLM history asides: README:AI gateway · network-access:AI gateway · proxy-key-onboarding · proxy-request-priority · .env.example (OmniRoute block, storage key, key mint) · compose (split port, omniroute header, extra_hosts) · update.sh (header, secrets, key mint).
## Resolution: audit of docs and code comments against source Scope: README.md, docs/ (excluding research/ and agents/), comments in docker-compose.yml, .env.example, scripts/*.sh, checked against compose/scripts as of `8ae3210`. Verdicts: TRUE / FALSE / DUPLICATE / STALE / REMOVED-FEATURE (ComfyUI / Qwen-Image / Open WebUI, not audited in depth). Line refs are for this audit only; they don't belong in docs. ### Headline findings 1. **The dashboard port is published.** Compose maps `OMNIROUTE_DASHBOARD_PORT` to the host. Four places say it never is. 2. **`OMNIROUTE_PORT:-4000` doesn't exist.** Compose publishes `OMNIROUTE_API_PORT` 1:1, so it's 20129:20129 by default with no 4000 mapping. Every client doc and .env.example still point at 4000. 3. **Lazytainer auto-suspend doesn't work in practice** (research doc on lazytainer idle-stop, #40, #5 closing comment). README, .env.example and compose still present it as working. After switch-model.sh is removed, nothing in the code will record this. 4. **Two sources explain the classifier's VRAM fix differently.** Compose says a missing flash-attn was the real cause. qwen-code.md calls flash-attn a "wrong turn" and credits partial offload. 5. **.env.example says qdrant isn't exposed off-box.** That's false: compose publishes 6333 on the host and qdrant has no auth, so anyone on the LAN can reach it without logging in. This is a security issue. ### README.md | Location | Claim gist | Verdict | Detail | |---|---|---|---| | README:intro | Lazytainer auto-suspends llama-server when idle | FALSE | Labels are set, but the idle-stop never fires: OmniRoute health pings count as traffic (research doc on lazytainer idle-stop, #40, #5) | | README:intro | Wayfinder map #1 has the architecture rationale | STALE | #1 is still open but predates OmniRoute (LiteLLM/Open WebUI era) | | README:Quickstart | What update.sh does: creates .env, fills secrets, resolves SEARXNG_LAN_IP, downloads the model, pulls/builds/brings up | TRUE (incomplete) | It also runs `git pull`, syncs non-blank config values from .env.example with a conflict prompt (`--force` in non-interactive runs), resolves the GPU GIDs, and downloads the classifier model and the ComfyUI files | | README:Quickstart | Safe to re-run; recreates only what changed | TRUE | | | README:Quickstart | llama.cpp API is internal-only | TRUE, DUPLICATE | Compose uses `expose` only. See the cluster "llama-server has no host port" | | README:Quickstart / risk | Qwen3.8 tool-calling isn't verified | STALE? | #5 closed with inference confirmed end-to-end through OmniRoute. Tool-calling wasn't explicitly re-tested. DUPLICATE of index.md and opencode.md | | README:AI gateway | update.sh handles most OmniRoute secrets; keys are minted by hand | TRUE, DUPLICATE | See the key-minting cluster | | README:AI gateway | Gateway at `${OMNIROUTE_PORT:-4000}/v1` | STALE/FALSE | That var doesn't exist. Compose publishes `OMNIROUTE_API_PORT` (default 20129) 1:1 | | README:AI gateway | Dashboard is never published | FALSE | Compose publishes `OMNIROUTE_DASHBOARD_PORT` to the host | | README:AI gateway | Not smoke-tested on real hardware | STALE | #5 closed after real-hardware verification. #17 closed as obsolete (LiteLLM) | | README:AI gateway | OmniRoute stealth/MITM features aren't used | TRUE | Nothing in compose enables them. The upstream claim itself can't be checked from the repo | | README:Web search | Gateway fronts SearXNG search | TRUE (partly unverifiable) | extra_hosts and the private-URL flags support it. The provider itself is registered in the dashboard, not in the repo. Its pointer goes to the research doc on LiteLLM + SearXNG search (LiteLLM naming) | ### docs/network-access.md | Location | Claim gist | Verdict | Detail | |---|---|---|---| | intro | The stack has no chat UI; every client is a coding CLI | FALSE, REMOVED-FEATURE | open-webui is published on 8008. The claim becomes true once Open WebUI is removed | | intro | Publishes the gateway API port and relies on the existing NPM | TRUE | | | llama.cpp LAN-only | Inference API on `${LLAMA_PORT:-8080}` | STALE | LLAMA_PORT is gone. Port 8080 is container-internal (`expose`) and isn't published at all, so "LAN-only" understates it: it isn't on the LAN either | | llama.cpp LAN-only | Unauthenticated; CLIs go through the gateway | TRUE, DUPLICATE | | | AI gateway | Both hostnames point at `${OMNIROUTE_PORT:-4000}` | STALE/FALSE | Should be `OMNIROUTE_API_PORT` | | AI gateway | Dashboard is never registered in NPM or published by compose; the split port is "structural, no network route" | FALSE | Compose publishes the dashboard port, and the compose comment says "both published directly". Not registering it in NPM is the only thing keeping it off the internet. It is reachable from the LAN | | AI gateway | Every gateway call needs an API key | TRUE (upstream behaviour) | | | Neo4j | Browser on 7474 and bolt on 7687, published | TRUE | | | Qdrant | Dashboard on raw LAN port 6333 | TRUE | Also no auth (see .env.example:RAG) | | (missing) | — | gap | The doc doesn't mention the published comfyui 8138 or open-webui 8008 (REMOVED-FEATURE) | ### docs/proxy-key-onboarding.md | Location | Claim gist | Verdict | Detail | |---|---|---|---| | intro | No in-stack workload needs a key | FALSE, REMOVED-FEATURE | open-webui needs `OPENWEBUI_OMNIROUTE_KEY`. True once it's removed. DUPLICATE of update.sh's key-minting comment | | intro / closing | There's no scripted mint because `POST /api/keys` needs a session, not verified (#37) | TRUE, DUPLICATE | update.sh has no mint step. #37 is closed, so the "see #37" pointer is stale | | Create the key, step 1 | Dashboard port is never published, so use the container IP or an SSH tunnel | FALSE | It's published, so `http://<host>:<OMNIROUTE_DASHBOARD_PORT>` works directly. The `docker inspect` template is also wrong: it uses `.Networks.ai_stack`, but the network is named `ai-stack` and Compose prefixes it with the project name (e.g. `llm-server_ai-stack`) | | Create the key, step 1 | Hardcodes 20128 | — | Restates a value. Name the key instead | | retire/rotate | Revoke by hand | TRUE | | ### docs/proxy-request-priority.md | Location | Claim gist | Verdict | Detail | |---|---|---|---| | banner | The Mechanism section onward is LiteLLM and stale | TRUE (self-declared) | | | intro | One local model instance serves every workload | FALSE | qwen-classifier is a second llama.cpp instance on the same GPU | | Tiers | Priority is assigned per virtual key | STALE | No mechanism exists in OmniRoute config or compose | | Mechanism / Timeout | LiteLLM scheduler, Redis, and timeouts; also a "#14 (compose authoring)" smoke-test task | STALE | LiteLLM has been removed | | Lazytainer interaction | Queueing happens inside LiteLLM and Lazytainer wakes the container | STALE / FALSE | LiteLLM is gone. Lazytainer's detector can't tell OmniRoute health pings from real traffic (#40), and its wake-on-traffic was never confirmed | ### docs/coding-cli-setup/* | Location | Claim gist | Verdict | Detail | |---|---|---|---| | index:intro | llama.cpp port is internal-only | TRUE, DUPLICATE | | | index:endpoints and summary table; claude-code, kimi-cli, opencode, qwen-code (x4) | `${OMNIROUTE_PORT:-4000}` | STALE/FALSE | Should be `OMNIROUTE_API_PORT` (20129 by default). Occurs about 12 times | | index | Model file / `qwen3.8-27b-local` naming | TRUE | Model file is the compose default. The model name is set in the dashboard | | index | One key per CLI; these are high-priority per proxy-request-priority.md | TRUE / STALE | The priority doc it points to is stale | | index | Run the #5 / #17 smoke tests before relying on this | STALE | Both issues are closed (#5 verified, #17 obsolete) | | index:further reading | Semaphore 30s reject for llama-server / qwen-classifier | TRUE (per research doc) | Dashboard setting, can't check from source | | claude-code / opencode | Needs `--jinja`, already set | TRUE, DUPLICATE | Both llama-server and qwen-classifier pass `--jinja` | | opencode | Per-slot context = CTX/PARALLEL = 131072 | TRUE, DUPLICATE | Also in qwen-code.md and .env.example:LLAMA_PARALLEL | | qwen-code:why a second model | Classifier queued behind 2 GPU slots; CPU-only too slow, blew past "OmniRoute's request timeout" | TRUE, DUPLICATE; wording conflicts | Compose says "OmniRoute's 60s timeout" instead. Neither figure matches the current `REQUEST_TIMEOUT_MS`, and the 60s was most likely the per-connection `timeoutMs`. Pick one wording | | qwen-code:why a second model | The fix was partial offload; batch-size tuning and flash-attn were "wrong turns" | FALSE (contradicts compose) | The compose qwen-classifier comment says flash-attn was the actual cause and batch-size the wrong lever, and never calls partial offload the lever. `--n-gpu-layers 28` is set, but the comment gives no reason for it | | qwen-code:settings bullets | Classifier `--parallel 1`, 65536 ctx, qwen-code transcript caps | TRUE, DUPLICATE | Matches compose. The transcript-cap reasoning is repeated word for word in compose | | qwen-code:auto mode tuning | "see network-access.md for reaching the dashboard API" | FALSE pointer | network-access.md has no access steps. They're in proxy-key-onboarding.md, and they're wrong there too (see above) | | qwen-code:auto mode tuning | Classifier timeoutMs 120000 and "20-minute" main connection | unverifiable | Dashboard-only settings, nothing in source | ### docs/server-planing.md | Location | Claim gist | Verdict | Detail | |---|---|---|---| | whole | 4x R9700 rack build plan | not source-checkable | A purchasing/planning doc, not config. The running stack is 1x R9700 | ### docker-compose.yml (comments) | Location | Claim gist | Verdict | Detail | |---|---|---|---| | llama-server:group_add | Numeric GIDs, not names | TRUE, DUPLICATE | See the GID cluster | | llama-server:environment | GPU_MAX_HW_QUEUES=1 works around ROCm#5706 | TRUE, DUPLICATE | Repeated in qwen-classifier. Confirmed fixed in #5 | | llama-server:--cache-reuse | Targets the root cause of the SSE timeout; pairs with promptCacheAffinity | TRUE | promptCacheAffinity is a dashboard default and can't be checked from source | | llama-server:expose | No host port; reached only via omniroute (#15) | TRUE, DUPLICATE | | | llama-server:labels | Idle-timeout tuning here; raise it if 15 min is too eager | STALE | The idle-stop never fires (#40), so tuning the timeout has no effect | | qwen-classifier header | Queueing, CPU-only, context sizing, VRAM, reasoning off | TRUE / DUPLICATE / conflict | Duplicates qwen-code.md, and the two disagree on the VRAM fix (see above). "Re-verify with rocm-smi after deploy" is still an open to-do | | downloader / downloader-classifier | One-off job, run by update.sh every time, `test -f` guard | TRUE | | | downloader-comfyui, comfyui | — | REMOVED-FEATURE | | | omniroute header | Replaces litellm; provider routing registered in the dashboard, not in the repo | TRUE, DUPLICATE | Also in update.sh's header comment. The LiteLLM history belongs in docs | | omniroute:split port | Dashboard and API are both published | TRUE | Contradicts README, network-access.md, proxy-key-onboarding.md and .env.example | | omniroute:private URLs, WS bridge | Needed for local providers; WS bridge secret is required | TRUE, DUPLICATE (WS bridge also in .env.example) | | | omniroute:OMNIROUTE_MEMORY_MB | "every client here is a coding CLI" | FALSE, REMOVED-FEATURE | open-webui is a client too. The claim becomes true once it's removed | | omniroute:STREAM_IDLE_TIMEOUT_MS | "see LLAMA_PARALLEL above" | FALSE pointer | The LLAMA_PARALLEL explanation is in .env.example, not above in compose. The rest DUPLICATES .env.example's timeout comments | | omniroute:REQUEST_TIMEOUT_MS | First non-ping SSE event deadline; value raised past the default | TRUE, DUPLICATE | .env.example repeats it | | omniroute:extra_hosts | "Same reasoning as litellm's extra_hosts entry below" | STALE | The litellm service is gone, so there's no entry below. The real reason is only in update.sh's SEARXNG comment | | omniroute:mem_limit, stop_grace_period, healthcheck | Matches the Docker guide; image has only node | TRUE | | | lazytainer | Why not host networking; joins ai-stack | TRUE | The mechanism is fine, but it doesn't solve the idle-stop problem (#40) | | qdrant | "see docs/agents/... (wayfinder)"; "published directly like comfyui above" | FALSE pointer, REMOVED-FEATURE ref | docs/agents/ has no RAG doc. The comfyui comparison will go when comfyui does | | open-webui | — | REMOVED-FEATURE | | ### .env.example (comments) | Location | Claim gist | Verdict | Detail | |---|---|---|---| | header | Values are compose defaults; "only uncomment/change what you want to override" | FALSE | Nothing is commented out. Every non-blank value is live, and update.sh syncs them into .env as tracked config (it prompts on conflicts), so editing .env by hand gets flagged on the next run. Compose also has no default for any secret | | LLAMA_GPU_LAYERS | 999 = all layers; this is the RAM-offload knob; mmap fallback | TRUE | | | LLAMA_CTX_SIZE | 262144 is the true max; q8_0 KV fits | TRUE (compose sets q8_0) | | | LLAMA_PARALLEL | 2 slots; llama.cpp queue has no timeout; per-slot math | TRUE, DUPLICATE | Per-slot math is also in opencode.md and qwen-code.md. "Handoff notes in the issue tracker" is a vague pointer | | LLAMA_CLASSIFIER_MODEL_FILE | See the compose comment | TRUE | | | Lazytainer | Seconds before llama-server is stopped | FALSE in practice | Configured, but it never fires (#40) | | SearXNG header | Pointer to the research doc on LiteLLM + SearXNG search | STALE naming | The doc exists but is named after LiteLLM | | SEARXNG_LAN_IP | Resolved by update.sh | TRUE, DUPLICATE | | | OmniRoute block | `OMNIROUTE_PORT` is published at 4000 and mapped onto API_PORT; the dashboard is never published | STALE / FALSE | No OMNIROUTE_PORT variable exists. Compose maps `OMNIROUTE_API_PORT`:`OMNIROUTE_API_PORT` and publishes the dashboard | | timeouts | STREAM_IDLE and REQUEST semantics | TRUE, DUPLICATE | Also in the compose omniroute comments | | secrets | Filled by update.sh | TRUE | | | OMNIROUTE_STORAGE_ENCRYPTION_KEY | Don't change it after the first run | TRUE, DUPLICATE | Also in update.sh. It's a data-loss warning, so it stays inline. The LiteLLM aside is stale | | per-workload keys | No `/key/generate` equivalent; mint by hand (#37) | TRUE, DUPLICATE | `/key/generate` is LiteLLM vocabulary | | HOST_*_GID | Sits under the "ComfyUI" header; shared by "llama-server, llama-server-fast, comfyui" | STALE, misfiled | llama-server-fast is gone. The actual users are llama-server, qwen-classifier and comfyui (the last is REMOVED-FEATURE). It belongs under its own GPU header | | COMFYUI_*, OPENWEBUI_OMNIROUTE_KEY | — | REMOVED-FEATURE | | | RAG | "No auth on qdrant, same trust boundary as llama-server: ai-stack not exposed off-box" | FALSE (security) | qdrant publishes 6333 on the host's interfaces, so it's reachable from the LAN without auth. llama-server has no published port at all, so the two aren't in the same trust boundary | ### scripts/update.sh (comments) | Location | Claim gist | Verdict | Detail | |---|---|---|---| | header | The one command to run after any change; safe to re-run | TRUE | | | header:config sync | Syncs non-blank values; interactive gum prompt; errors without a TTY unless `--force`; blank secrets go through set_if_blank | TRUE | The secrets list leaves out NEO4J_PASSWORD, which is also handled by set_if_blank | | header:omniroute config | Lives in the dashboard, not a checked-in file like litellm-config.yaml | TRUE, DUPLICATE | Also in compose's omniroute header. The LiteLLM reference is history | | header:ponytail | No rollback | TRUE | | | git pull first | Self-update race (#5) | TRUE | | | gum | Vendored tarball checked in; download fallback; cached in .cache/gum | TRUE | Only the x86_64 tarball is vendored, which is consistent with "other archs" falling back to the download | | set_if_blank | Handles missing, blank and set | TRUE | | | secrets | Never change the storage key | TRUE, DUPLICATE | | | SEARXNG | search.home isn't resolvable inside the container | TRUE, DUPLICATE | This is the only real explanation for compose's extra_hosts | | GIDs | Numeric GID rationale | TRUE, DUPLICATE (x3) | | | model download | Also runs downloader-comfyui | REMOVED-FEATURE | | | key-mint ponytail | No in-stack workload needs a key | FALSE, REMOVED-FEATURE | open-webui needs one. The rest DUPLICATES proxy-key-onboarding.md | ### scripts/switch-model.sh | Location | Claim gist | Verdict | Detail | |---|---|---|---| | whole script | Swaps GPU residency between qwen and comfyui | REMOVED-FEATURE | | | header | Lazytainer's idle-stop can't work: its source-blind packet counter sees OmniRoute health pings as traffic | TRUE | **This is the only statement in code or docs that lazytainer doesn't work.** It has to move into docs before the script is deleted. It contradicts README, .env.example:Lazytainer and compose:lazytainer labels | ### Outside scope, noticed in passing - The `.gitignore` comment refers to `scripts/ingest-memory.sh`, which doesn't exist (STALE). The file also has `.qwen/tmp` without a trailing newline. ## Duplicate clusters - **Dashboard not published (all FALSE):** README:AI gateway · network-access:AI gateway · proxy-key-onboarding:step 1 · .env.example:OmniRoute block. Compose's omniroute split-port comment is the only correct one. - **`OMNIROUTE_PORT:-4000` (STALE):** README:AI gateway · network-access:AI gateway · .env.example:OmniRoute block · coding-cli-setup index (endpoints and table), claude-code, kimi-cli, opencode, qwen-code (x4). - **Lazytainer idle-stop:** "works" in README:intro · .env.example:Lazytainer · compose:llama-server labels · proxy-request-priority:Lazytainer interaction. "Doesn't work" only in switch-model.sh's header and the research doc on lazytainer idle-stop. - **llama-server has no host port / internal-only:** README:Quickstart · README:AI gateway · network-access:llama.cpp section · coding-cli index:intro · compose:llama-server expose · compose:lazytainer. - **Numeric GID rationale:** compose:llama-server group_add · compose:comfyui group_add · .env.example:HOST_*_GID · update.sh:GIDs. - **GPU_MAX_HW_QUEUES / ROCm#5706:** compose:llama-server · compose:qwen-classifier. - **Classifier rationale (queueing, CPU-only, context caps, VRAM):** qwen-code:why a second model and settings bullets · compose:qwen-classifier header · .env.example:LLAMA_CLASSIFIER_MODEL_FILE (pointer only). The first two conflict on the VRAM fix. - **Per-slot context math:** opencode.md · qwen-code:settings bullets · .env.example:LLAMA_PARALLEL. - **`--jinja` required:** claude-code.md · opencode.md. - **Tool-calling risk:** README:risk · coding-cli index:warning · opencode:risks. - **No scripted key mint / session auth (#37):** README:AI gateway · proxy-key-onboarding:intro and closing · .env.example:per-workload keys · update.sh:key-mint ponytail. - **"No in-stack workload needs a key" (FALSE until Open WebUI is removed):** proxy-key-onboarding:intro · update.sh:key-mint ponytail. - **"Every client is a coding CLI / no chat UI" (FALSE until Open WebUI is removed):** network-access:intro · compose:OMNIROUTE_MEMORY_MB. - **Provider config lives in the dashboard, not the repo:** compose:omniroute header · update.sh:header · compose:comfyui (REMOVED). - **OmniRoute timeouts (STREAM_IDLE / REQUEST):** compose:omniroute env comments · .env.example:timeouts. - **Storage encryption key must never change:** .env.example · update.sh:secrets. It's a data-loss warning, so it stays inline. - **WS bridge secret required:** compose:omniroute · .env.example. - **search.home unresolvable in the container:** update.sh:SEARXNG · .env.example:SEARXNG_LAN_IP · compose:omniroute extra_hosts (stale litellm pointer) · README:Quickstart. - **LiteLLM history asides:** README:AI gateway · network-access:AI gateway · proxy-key-onboarding · proxy-request-priority · .env.example (OmniRoute block, storage key, key mint) · compose (split port, omniroute header, extra_hosts) · update.sh (header, secrets, key mint).
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:task
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees haylan
1 Participants
Notifications
Due Date
No due date set.
Blocks
#59 Glossary: pin down domain terms in CONTEXT.md
haylan/LLM-Server
#60 Target doc structure
haylan/LLM-Server
Reference: haylan/LLM-Server#56