Author the docker-compose service for the chosen proxy #14

New Issue

2026-08-25T04:40:05Z

haylan commented

2026-08-25 04:40:05 +00:00

Part of #9

Question

Add the proxy tool chosen in #10 to docker-compose.yml (new service + .env.example entries), wired in front of llama.cpp per the network plan from #13, with the key-provisioning workflow from #12 and the shadow-cost config from #11 in place. Document it (README.md / a new docs page), following the pattern of map #1's #4 and #6.

Part of #9 ## Question Add the proxy tool chosen in #10 to docker-compose.yml (new service + .env.example entries), wired in front of llama.cpp per the network plan from #13, with the key-provisioning workflow from #12 and the shadow-cost config from #11 in place. Document it (README.md / a new docs page), following the pattern of map #1's #4 and #6.

haylan added the wayfinder:task label 2026-08-25 04:40:05 +00:00

haylan referenced this issue

2026-08-25 04:40:06 +00:00

Migrate Open WebUI (and existing consumers) to route through the new proxy #15

haylan added a new dependency 2026-08-25 04:40:14 +00:00

#10 Which self-hosted AI gateway/proxy tool fits this effort's needs?

haylan added a new dependency 2026-08-25 04:40:14 +00:00

#11 Reference cloud model/pricing for the shadow-cost estimate

haylan added a new dependency 2026-08-25 04:40:14 +00:00

#12 How should per-workload API keys/accounts be provisioned, rotated, and documented?

haylan added a new dependency 2026-08-25 04:40:15 +00:00

#13 Network/hostname plan for exposing the proxy (LAN + external)

haylan added a new dependency 2026-08-25 04:40:15 +00:00

#15 Migrate Open WebUI (and existing consumers) to route through the new proxy

haylan referenced this issue

2026-08-25 04:40:25 +00:00

AI gateway/proxy: routing, per-workload keys, and cloud-cost estimation for the local LLM stack #9

haylan added a new dependency 2026-08-25 04:41:14 +00:00

#16 How should the proxy queue and prioritize concurrent requests across workloads?

haylan referenced this issue

2026-08-25 04:45:40 +00:00

Which self-hosted AI gateway/proxy tool fits this effort's needs? #10

haylan referenced this issue

2026-08-25 04:50:14 +00:00

Reference cloud model/pricing for the shadow-cost estimate #11

haylan referenced this issue

2026-08-25 05:07:28 +00:00

How should the proxy queue and prioritize concurrent requests across workloads? #16

haylan self-assigned this 2026-08-25 05:09:46 +00:00

haylan referenced this issue

2026-08-25 05:11:03 +00:00

Verify the AI proxy stack on the real R9700 box (LiteLLM scheduler smoke test) #17

haylan added a new dependency 2026-08-25 05:11:08 +00:00

#17 Verify the AI proxy stack on the real R9700 box (LiteLLM scheduler smoke test)

haylan referenced this issue from a commit

2026-08-25 05:11:16 +00:00

Author LiteLLM AI proxy service (resolves #14)

haylan closed this issue

2026-08-25 05:11:17 +00:00

haylan commented

2026-08-25 05:11:23 +00:00

Resolved: authored the docker-compose service.

Added litellm + litellm-db (Postgres) services to docker-compose.yml, a new litellm-config.yaml wiring the local llama.cpp model with #11's shadow-cost pricing (Claude Sonnet 5 rate, model_info.input_cost_per_token/output_cost_per_token) and a priority-scheduling stub per #16, and required LITELLM_MASTER_KEY/LITELLM_SALT_KEY/LITELLM_DB_PASSWORD env vars in .env.example. Documented in README.md.

Untested on real hardware — split out as issue #17, which also carries #16's required priority-scheduler smoke test. Open WebUI/coding CLIs still hit llama.cpp directly — that migration is #15, unblocked by this.

Committed on main.

Resolved: authored the docker-compose service. Added `litellm` + `litellm-db` (Postgres) services to docker-compose.yml, a new `litellm-config.yaml` wiring the local llama.cpp model with #11's shadow-cost pricing (Claude Sonnet 5 rate, `model_info.input_cost_per_token`/`output_cost_per_token`) and a priority-scheduling stub per #16, and required `LITELLM_MASTER_KEY`/`LITELLM_SALT_KEY`/`LITELLM_DB_PASSWORD` env vars in .env.example. Documented in README.md. Untested on real hardware — split out as [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17), which also carries #16's required priority-scheduler smoke test. Open WebUI/coding CLIs still hit llama.cpp directly — that migration is #15, unblocked by this. Committed on main.

Sign in to join this conversation.