OmniRoute: can it route to arbitrary OpenAI-compatible local endpoints (llama-server), not just Ollama? #32

New Issue

2026-09-03T17:17:39Z

haylan commented

2026-09-03 17:17:39 +00:00

Part of #31

Question

This stack has no Ollama — llama-server and embedding-server are raw llama.cpp instances each exposing an OpenAI-compatible /v1 (see docker-compose.yml, litellm-config.yaml's model_list, both using openai/<name> + a plain api_base). LiteLLM accepts this directly.

Does OmniRoute support pointing a model entry at an arbitrary OpenAI-compatible api_base (chat and embeddings) the way LiteLLM's model_list does — or is its local-model support genuinely Ollama-only (per its docs' "first-class Ollama integration" framing)? If Ollama-only, what would sit in front of llama-server/embedding-server to bridge this — and is that even viable given llama.cpp serves one model per process?

This is load-bearing: if OmniRoute can't reach these two containers directly (or via a thin bridge), the rest of the migration doesn't make sense to plan.

Part of #31 ## Question This stack has no Ollama — `llama-server` and `embedding-server` are raw llama.cpp instances each exposing an OpenAI-compatible `/v1` (see `docker-compose.yml`, `litellm-config.yaml`'s `model_list`, both using `openai/<name>` + a plain `api_base`). LiteLLM accepts this directly. Does OmniRoute support pointing a model entry at an arbitrary OpenAI-compatible `api_base` (chat *and* embeddings) the way LiteLLM's `model_list` does — or is its local-model support genuinely Ollama-only (per its docs' "first-class Ollama integration" framing)? If Ollama-only, what would sit in front of `llama-server`/`embedding-server` to bridge this — and is that even viable given llama.cpp serves one model per process? This is load-bearing: if OmniRoute can't reach these two containers directly (or via a thin bridge), the rest of the migration doesn't make sense to plan.

haylan added the wayfinder:research label 2026-09-03 17:17:39 +00:00

haylan self-assigned this 2026-09-03 17:23:39 +00:00

haylan closed this issue

2026-09-03 17:23:55 +00:00

haylan reopened this issue

2026-09-03 17:24:04 +00:00

haylan commented

2026-09-03 17:24:07 +00:00

Answer: No, OmniRoute is not Ollama-only. It supports arbitrary OpenAI-compatible api_base for both chat and embeddings, verified from primary source (github.com/diegosouzapw/OmniRoute, main branch).

1. llama.cpp is a named local-provider preset, src/shared/constants/providers/local.ts:

"llama-cpp": {
  id: "llama-cpp", alias: "llamacpp", name: "llama.cpp",
  website: "https://github.com/ggml-org/llama.cpp",
  authHint: "API key optional ... Configure the llama-server OpenAI-compatible base URL (default: http://127.0.0.1:8080/v1)...",
  localDefault: "http://127.0.0.1:8080/v1",
  passthroughModels: true,
},

tests/unit/llama-cpp-local-url.test.ts (regression for issues #3136/#3197) proves the configured providerSpecificData.baseUrl is actually honored for chat routing and does NOT silently fall back to api.openai.com. tests/unit/llamacpp-model-delete.test.ts shows multiple independent named connections of the same provider (conn-a, conn-b) are supported — i.e. you can add more than one llama-cpp connection pointed at different ports.

2. Embeddings use a separate, fully generic mechanism — open-sse/config/embeddingRegistry.ts exports buildDynamicEmbeddingProvider(node), which builds an EmbeddingProvider from a user-supplied "provider_node" (arbitrary prefix + baseUrl, /embeddings appended). Backed by src/lib/db/providers/nodes.ts::createProviderNode({ type: "openai-compatible-embeddings", prefix, baseUrl, apiType: "embeddings", ... }) — a generic provider_nodes table (type, prefix, baseUrl, chatPath, modelsPath, customHeaders), not Ollama-specific.

3. This generic mechanism is not restricted to localhost/Ollama. tests/unit/embeddings-lan-noauth-6925.test.ts explicitly tests a keyless OpenAI-compatible embeddings provider_node at LAN IPs (10.10.0.181, 192.168.1.10) and confirms requests hit that exact host with no forced Authorization header; only the cloud-metadata IP (169.254.169.254) is blocked as an SSRF guard. So a Docker-network hostname (e.g. http://embedding-server:8081/v1) is the same shape of request this code path already handles — worth a quick smoke test in our compose network, but nothing in the source gates it to Ollama or to localhost specifically.

Conclusion for the migration: register two connections — a llama-cpp (or generic openai-compatible) chat connection pointed at llama-server's /v1, and an openai-compatible-embeddings provider_node pointed at embedding-server's /v1/embeddings — mirroring LiteLLM's model_list + api_base pattern exactly. No Ollama, and no bridge, is needed. The single-model-per-process nature of llama.cpp is naturally handled since each provider_node/connection carries its own independent baseUrl.

Sources: raw.githubusercontent.com/diegosouzapw/OmniRoute/main/src/shared/constants/providers/local.ts, .../open-sse/config/embeddingRegistry.ts, .../src/lib/db/providers/nodes.ts, .../tests/unit/llama-cpp-local-url.test.ts, .../tests/unit/llamacpp-model-delete.test.ts, .../tests/unit/embeddings-lan-noauth-6925.test.ts, .../docs/reference/PROVIDER_REFERENCE.md

Answer: No, OmniRoute is not Ollama-only. It supports arbitrary OpenAI-compatible api_base for both chat and embeddings, verified from primary source (github.com/diegosouzapw/OmniRoute, main branch). **1. llama.cpp is a named local-provider preset**, `src/shared/constants/providers/local.ts`: ``` "llama-cpp": { id: "llama-cpp", alias: "llamacpp", name: "llama.cpp", website: "https://github.com/ggml-org/llama.cpp", authHint: "API key optional ... Configure the llama-server OpenAI-compatible base URL (default: http://127.0.0.1:8080/v1)...", localDefault: "http://127.0.0.1:8080/v1", passthroughModels: true, }, ``` `tests/unit/llama-cpp-local-url.test.ts` (regression for issues #3136/#3197) proves the configured `providerSpecificData.baseUrl` is actually honored for chat routing and does NOT silently fall back to api.openai.com. `tests/unit/llamacpp-model-delete.test.ts` shows multiple independent named connections of the same provider (`conn-a`, `conn-b`) are supported — i.e. you can add more than one llama-cpp connection pointed at different ports. **2. Embeddings use a separate, fully generic mechanism** — `open-sse/config/embeddingRegistry.ts` exports `buildDynamicEmbeddingProvider(node)`, which builds an `EmbeddingProvider` from a user-supplied "provider_node" (arbitrary `prefix` + `baseUrl`, `/embeddings` appended). Backed by `src/lib/db/providers/nodes.ts::createProviderNode({ type: "openai-compatible-embeddings", prefix, baseUrl, apiType: "embeddings", ... })` — a generic provider_nodes table (type, prefix, baseUrl, chatPath, modelsPath, customHeaders), not Ollama-specific. **3. This generic mechanism is not restricted to localhost/Ollama.** `tests/unit/embeddings-lan-noauth-6925.test.ts` explicitly tests a keyless OpenAI-compatible embeddings provider_node at LAN IPs (10.10.0.181, 192.168.1.10) and confirms requests hit that exact host with no forced Authorization header; only the cloud-metadata IP (169.254.169.254) is blocked as an SSRF guard. So a Docker-network hostname (e.g. `http://embedding-server:8081/v1`) is the same shape of request this code path already handles — worth a quick smoke test in our compose network, but nothing in the source gates it to Ollama or to localhost specifically. **Conclusion for the migration:** register two connections — a `llama-cpp` (or generic openai-compatible) chat connection pointed at llama-server's `/v1`, and an `openai-compatible-embeddings` provider_node pointed at embedding-server's `/v1/embeddings` — mirroring LiteLLM's model_list + api_base pattern exactly. No Ollama, and no bridge, is needed. The single-model-per-process nature of llama.cpp is naturally handled since each provider_node/connection carries its own independent baseUrl. Sources: raw.githubusercontent.com/diegosouzapw/OmniRoute/main/src/shared/constants/providers/local.ts, .../open-sse/config/embeddingRegistry.ts, .../src/lib/db/providers/nodes.ts, .../tests/unit/llama-cpp-local-url.test.ts, .../tests/unit/llamacpp-model-delete.test.ts, .../tests/unit/embeddings-lan-noauth-6925.test.ts, .../docs/reference/PROVIDER_REFERENCE.md

haylan closed this issue

2026-09-03 17:24:11 +00:00

haylan referenced this issue

2026-09-03 17:25:01 +00:00

Migrate the AI gateway from LiteLLM to OmniRoute #31

haylan referenced this issue

2026-09-03 17:28:21 +00:00

OmniRoute: per-workload virtual keys, admin UI, and deployment shape (Docker/compose) #36

Sign in to join this conversation.