diff --git a/docs/research/litellm-searxng-search.md b/docs/research/litellm-searxng-search.md new file mode 100644 index 0000000..66829e1 --- /dev/null +++ b/docs/research/litellm-searxng-search.md @@ -0,0 +1,260 @@ +# Research: Wiring the local SearXNG instance into LiteLLM's web-search feature + +**Question:** How does LiteLLM's web-search integration +(https://docs.litellm.ai/docs/search) actually work, and what does wiring the +local SearXNG instance (`http://search.home/`) into this repo's +`litellm-config.yaml` require? + +**Answer, short version:** SearXNG **is** a natively supported provider for +LiteLLM's `/v1/search` feature — no custom-endpoint workaround needed. But the +feature is **not** a model-callable tool and **not** automatic +context-injection into chat completions either — it's a **separate REST API** +(`/v1/search/{search_tool_name}`) that a caller (Open WebUI, a script, a +future MCP wrapper) must invoke directly, independent of any LLM call. That +sidesteps this project's known-flaky Qwen3.8-27B tool-calling entirely, as +long as nothing wraps the endpoint back into model-driven tool-calling. +Reachability is the real blocker: `search.home` is a LAN mDNS/local-DNS name +that the `litellm` container cannot resolve by default — needs an +`extra_hosts` entry in `docker-compose.yml`. + +## 1. What LiteLLM's search feature actually is + +LiteLLM ships a **unified search API** (`/v1/search` and +`/v1/search/{search_tool_name}`) that wraps multiple search-provider backends +behind one Perplexity-compatible request/response shape. +Source: https://docs.litellm.ai/docs/search + +Config shape in `config.yaml` (LiteLLM's own documented example, Perplexity +shown, same shape for every provider): + +```yaml +search_tools: + - search_tool_name: perplexity-search + litellm_params: + search_provider: perplexity + api_key: os.environ/PERPLEXITYAI_API_KEY +``` + +Call shape: + +```bash +curl http://0.0.0.0:4000/v1/search/searxng-search \ + -H "Authorization: Bearer sk-1234" \ + -H "Content-Type: application/json" \ + -d '{"query": "latest AI developments", "max_results": 5}' +``` + +Source: https://docs.litellm.ai/docs/search + +**18 providers are listed as supported**, including Perplexity, Tavily, Exa +AI, Brave, Parallel AI, Google PSE, DataForSEO, Firecrawl, **SearXNG**, +Linkup, Serper, DuckDuckGo, SearchAPI.io, You.com, APISerpent, Bedrock +AgentCore, Nimble, and Bing Grounding. +Source: https://docs.litellm.ai/docs/search + +## 2. Is SearXNG natively supported? — Yes + +SearXNG is one of the 18 built-in `search_provider` values, added by +BerriAI/litellm PR #16259 ("[Feat] add serxng search API provider"). +Source: https://github.com/BerriAI/litellm/pull/16259 + +Config shape for SearXNG specifically: + +```yaml +search_tools: + - search_tool_name: searxng-search + litellm_params: + search_provider: searxng + api_base: https://your-searxng-instance.com +``` + +Equivalently, the base URL can be supplied via the `SEARXNG_API_BASE` +environment variable instead of an inline `api_base` key — SearXNG has no API +key of its own (it's an unauthenticated local meta-search engine), so this is +the one provider in the list where `litellm_params` doesn't need a secret. +Sources: https://github.com/BerriAI/litellm/pull/16259, +https://docs.litellm.ai/docs/search + +**For this repo**, the addition to `litellm-config.yaml` (research only — not +applied here) would look like: + +```yaml +search_tools: + - search_tool_name: searxng-search + litellm_params: + search_provider: searxng + api_base: http://search.home/ +``` + +No custom-endpoint or "generic OpenAI-compatible /v1/web_search" fallback is +needed — the concern in the ticket that LiteLLM's docs "may assume a hosted +provider like Tavily/Serper" turned out not to apply; SearXNG is a first-class +`search_provider` value, same shape as every hosted one. + +## 3. Tool-call vs. automatic injection vs. a third thing + +The ticket asked to determine whether this is (a) a tool the model must +explicitly call, or (b) automatic pre-retrieval/context-injection like +Perplexity's own search-augmented answers. **It's neither** — it's a +**standalone REST endpoint** that sits alongside `/v1/chat/completions`, not +wired into it: + +> The documentation indicates this is a separate REST endpoint the +> application calls directly. The page presents `/search` as a standalone API +> endpoint alongside chat completions, not as an automatic injection feature. +> Users explicitly invoke the search endpoint; LiteLLM does not automatically +> inject search results into completions. + +Source: https://docs.litellm.ai/docs/search (fetched content, describing the +`/v1/search` and `/v1/search/{search_tool_name}` endpoints as siblings of +`/v1/chat/completions`, not a chat-completion parameter or automatic +retrieval step) + +Practical effect: whatever calls this endpoint — Open WebUI's own web-search +feature, a shell script, a future MCP server — does so with a plain HTTP call. +**LiteLLM's model routing and Qwen3.8-27B's tool-calling reliability are not +in that path at all**, unless something downstream chooses to expose this +endpoint back to the model *as* a function-calling tool (e.g. an MCP wrapper +that hands the model a `web_search` tool definition backed by this endpoint — +that would reintroduce the model-must-emit-a-correct-tool-call problem, but +that's a choice made one layer up, not something LiteLLM's `/v1/search` +feature forces). + +## 4. Network reachability: `search.home` from inside the `litellm` container + +`docker-compose.yml`'s `litellm` service joins only the `ai-stack` bridge +network (`networks: [ai-stack]`, line 116) and gets DNS resolution from +Docker's embedded DNS server for that network — which resolves other +containers by service/container name (`llama-server`, `qdrant`, etc., as +already used at `api_base: http://llama-server:8080/v1` in +`litellm-config.yaml` line 8) but has **no visibility into the LAN's mDNS/ +local-DNS namespace** that resolves `search.home` on the host machine or on +LAN clients. So `http://search.home/` will not resolve from inside the +`litellm` container as configured today — this matches the ticket's +suspicion, and is standard Docker bridge-networking behavior, not specific to +this repo. +Source: `g:\_DEV\repos\LLM-Server\docker-compose.yml` (litellm service, lines +94–124; `networks:` block, lines 154–156) + +No `extra_hosts`, `host.docker.internal`, or `network_mode: host` pattern +exists yet anywhere in this compose file to crib from — this would be the +first. (One service, `lazytainer`, already uses `network_mode: host`, but for +an unrelated reason — Docker-socket/host-port introspection — and switching +`litellm` to host networking would be a much bigger blast-radius change than +this ticket needs, dropping the `ai-stack` network isolation for every other +port `litellm` exposes.) +Source: `g:\_DEV\repos\LLM-Server\docker-compose.yml` lines 144–152 + +**Recommendation: `extra_hosts` on the `litellm` service**, mapping +`search.home` to its LAN IP, e.g.: + +```yaml +litellm: + ... + extra_hosts: + - "search.home:192.0.2.10" # replace with SearXNG's actual LAN IP +``` + +This is the smallest, most local fix: one line, scoped to the one service +that needs it, no change to network topology or isolation, and it keeps +`litellm-config.yaml`'s `api_base: http://search.home/` value human-readable +(matching this repo's existing preference for symbolic hostnames like +`ai.home` / `proxy.ai.home` documented in `docs/network-access.md`) rather +than hardcoding the LAN IP directly into the YAML config. The IP needs to stay +in sync if SearXNG's host ever gets a new DHCP lease — same caveat that would +apply to any hardcoded-IP alternative, just isolated to one `extra_hosts` +line instead of buried in the search config. + +`host.docker.internal` is not applicable here: that special hostname +resolves to the Docker **host's** own IP (useful for reaching a service +running directly on the host machine's loopback), not to arbitrary LAN mDNS +names — it wouldn't help resolve `search.home` unless SearXNG happens to run +on the same physical host as this compose stack. + +## 5. Interaction with this project's known-flaky Qwen3.8-27B tool-calling + +`docs/research/qwen3.8-27b-tool-calling.md` (2026-08-24) found, with medium- +to-high confidence, that Qwen3.8-27B's tool-calling through llama.cpp +inherits open/partially-fixed upstream parser bugs from the Qwen3.5 lineage +(issues #21158, #20837 in `ggml-org/llama.cpp`) — tool calls can be emitted +but not recognized, or land as inert XML inside a reasoning block, especially +with thinking enabled. +Source: `g:\_DEV\repos\LLM-Server\docs\research\qwen3.8-27b-tool-calling.md` +(section 3, "Bottom line" section) + +Given section 3 above (`/v1/search` is a standalone endpoint, not a +model-tool), **that flakiness has no bearing on the recommended integration +path**: nothing about calling `POST /v1/search/searxng-search` from Open +WebUI or a script asks Qwen3.8-27B to emit a tool call at all. The search +happens (or doesn't) independent of the model's tool-calling grammar/parser +entirely. + +The risk **would** resurface only if a *different* design choice is made +later — e.g. wrapping this same SearXNG-backed endpoint as an MCP tool or a +`tools=[...]` function definition handed to Qwen3.8-27B in a chat-completion +request, so the model itself decides when to search. That path would inherit +every bug documented in `qwen3.8-27b-tool-calling.md` (calls silently dropped, +calls trapped inside `` blocks, etc.) and would need the live smoke +test that doc recommends before being trusted unattended. **That is not what +LiteLLM's `/v1/search` feature itself requires** — it's a choice a caller +could additionally make on top of it. + +## Recommendation + +1. Add a `search_tools` block to `litellm-config.yaml` using + `search_provider: searxng` and `api_base: http://search.home/` (see + section 2) — no custom/generic-endpoint workaround needed, this is a + first-class supported provider. +2. Add `extra_hosts: ["search.home:"]` to the `litellm` service in + `docker-compose.yml` (see section 4) so the container can resolve the + hostname; confirm the IP is stable (static DHCP reservation) since + `extra_hosts` is a static mapping baked in at container start. +3. Treat `/v1/search` as a plain HTTP integration point, not a model tool — + whatever calls it (Open WebUI, a script) should call the REST endpoint + directly rather than exposing it to Qwen3.8-27B as a function-calling + tool, to avoid inheriting this project's documented tool-calling + flakiness (section 5). If model-driven search-tool-calling is wanted + later, that's a separate decision that should be smoke-tested against the + caveats in `qwen3.8-27b-tool-calling.md` first. + +This is research only — `litellm-config.yaml` and `docker-compose.yml` are +not modified by this doc. + +## Sources + +- https://docs.litellm.ai/docs/search — LiteLLM search feature docs: + endpoints, `search_tools` config shape, provider list, standalone-endpoint + behavior. +- https://github.com/BerriAI/litellm/pull/16259 — SearXNG provider + implementation: `search_provider: searxng`, `api_base` / + `SEARXNG_API_BASE` config. +- `g:\_DEV\repos\LLM-Server\docker-compose.yml` — `litellm` service + definition (lines 94–124), `ai-stack` network block (lines 154–156), + `lazytainer`'s `network_mode: host` precedent (lines 144–152). +- `g:\_DEV\repos\LLM-Server\litellm-config.yaml` — current proxy config + conventions (`model_list`, `litellm_params`, `api_base` usage at line 8). +- `g:\_DEV\repos\LLM-Server\docs\network-access.md` — this repo's existing + `*.home` / NPM hostname conventions. +- `g:\_DEV\repos\LLM-Server\docs\research\qwen3.8-27b-tool-calling.md` — + prior findings on Qwen3.8-27B tool-calling reliability via llama.cpp. + +## Confidence/uncertainty summary + +- **High confidence:** SearXNG is a native, first-class `search_provider` in + LiteLLM's `/v1/search` feature (directly documented and confirmed via the + implementing PR); the feature is a standalone REST endpoint separate from + chat completions, not automatic context-injection and not itself a + model-callable tool. +- **Medium confidence:** the exact field name (`api_base` vs. relying solely + on `SEARXNG_API_BASE`) — two independent fetch passes against + docs.litellm.ai returned slightly different renderings of the same example + (one showed `api_key: os.environ/SEARXNG_API_BASE`, the other and the PR + fetch showed `api_base: `); the PR-sourced `api_base` form is treated + as authoritative here since it comes from the implementing code change, but + this should be smoke-tested against the actual deployed LiteLLM image + (`ghcr.io/berriai/litellm:main-stable`) before being relied on verbatim. +- **Not independently verified:** SearXNG's actual LAN IP/hostname stability + on this network, and whether the deployed LiteLLM version + (`main-stable`, per `docker-compose.yml` line 95) already includes PR + #16259 — worth a quick `docker exec litellm pip show litellm` / changelog + check before wiring this in for real.