Verify the AI proxy stack on the real R9700 box (LiteLLM scheduler smoke test) #17

New Issue

2026-08-25T05:11:03Z

haylan commented

2026-08-25 05:11:03 +00:00

Part of #9

Question

Deploy and verify the docker-compose changes from #14 (litellm + litellm-db services, litellm-config.yaml) on the real Radeon R9700 hardware — mirrors map #1's real-hardware verification ticket (#5).

Specifically confirm:

The stack comes up cleanly (docker compose up -d with real LITELLM_MASTER_KEY/LITELLM_SALT_KEY set).
A virtual key created via the Admin UI (per docs/proxy-key-onboarding.md) can call the proxy and reach llama.cpp end-to-end.
The shadow cost estimate (docs/proxy-shadow-pricing.md) actually appears correctly on /key/info and the Admin UI dashboard after a real call.
LiteLLM's priority scheduler smoke test (per #16's resolution, docs/proxy-request-priority.md): confirm the priority field doesn't leak into llama.cpp's request, and that high-priority requests actually get dispatched ahead of low-priority ones under concurrent load. If broken, flag it — #16's fallback (a queuing shim) becomes live work.
proxy.ai.home / proxy.ai.haylan.ch NPM Proxy Hosts route correctly, and the external host's /ui deny rule actually blocks Admin UI access externally (per docs/network-access.md).

Part of #9 ## Question Deploy and verify the docker-compose changes from #14 (litellm + litellm-db services, litellm-config.yaml) on the real Radeon R9700 hardware — mirrors map #1's real-hardware verification ticket (#5). Specifically confirm: - The stack comes up cleanly (`docker compose up -d` with real `LITELLM_MASTER_KEY`/`LITELLM_SALT_KEY` set). - A virtual key created via the Admin UI (per docs/proxy-key-onboarding.md) can call the proxy and reach llama.cpp end-to-end. - The shadow cost estimate (docs/proxy-shadow-pricing.md) actually appears correctly on `/key/info` and the Admin UI dashboard after a real call. - **LiteLLM's priority scheduler smoke test** (per #16's resolution, docs/proxy-request-priority.md): confirm the `priority` field doesn't leak into llama.cpp's request, and that high-priority requests actually get dispatched ahead of low-priority ones under concurrent load. If broken, flag it — #16's fallback (a queuing shim) becomes live work. - `proxy.ai.home` / `proxy.ai.haylan.ch` NPM Proxy Hosts route correctly, and the external host's `/ui` deny rule actually blocks Admin UI access externally (per docs/network-access.md).

haylan added the wayfinder:task label 2026-08-25 05:11:03 +00:00

haylan added a new dependency 2026-08-25 05:11:08 +00:00

#14 Author the docker-compose service for the chosen proxy

haylan referenced this issue from a commit

2026-08-25 05:11:17 +00:00

Author LiteLLM AI proxy service (resolves #14)

haylan referenced this issue

2026-08-25 05:11:24 +00:00

Author the docker-compose service for the chosen proxy #14

haylan referenced this issue

2026-08-25 05:11:36 +00:00

AI gateway/proxy: routing, per-workload keys, and cloud-cost estimation for the local LLM stack #9