haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

research: LiteLLM web-search + SearXNG wiring (issue #22)

Browse Source 
Answers whether SearXNG is natively supported by LiteLLM's /v1/search
feature, whether it's a model-tool or automatic retrieval, what
docker-compose.yml networking change is needed for the litellm container
to reach the LAN's search.home host, and how it interacts with this
project's known-flaky Qwen3.8-27B tool-calling.

Research only — litellm-config.yaml and docker-compose.yml are unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
haylan
2026-09-02 21:25:30 +02:00
co-authored by Claude-Bot
parent 949802fb2b
commit 8c42f2518b
1 changed files with 260 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/litellm-searxng-search.md
+260
View File
@@ -0,0 +1,260 @@
# Research: Wiring the local SearXNG instance into LiteLLM's web-search feature
**Question:** How does LiteLLM's web-search integration
(https://docs.litellm.ai/docs/search) actually work, and what does wiring the
local SearXNG instance (`http://search.home/`) into this repo's
`litellm-config.yaml` require?
**Answer, short version:** SearXNG **is** a natively supported provider for
LiteLLM's `/v1/search` feature — no custom-endpoint workaround needed. But the
feature is **not** a model-callable tool and **not** automatic
context-injection into chat completions either — it's a **separate REST API**
(`/v1/search/{search_tool_name}`) that a caller (Open WebUI, a script, a
future MCP wrapper) must invoke directly, independent of any LLM call. That
sidesteps this project's known-flaky Qwen3.8-27B tool-calling entirely, as
long as nothing wraps the endpoint back into model-driven tool-calling.
Reachability is the real blocker: `search.home` is a LAN mDNS/local-DNS name
that the `litellm` container cannot resolve by default — needs an
`extra_hosts` entry in `docker-compose.yml`.
## 1. What LiteLLM's search feature actually is
LiteLLM ships a **unified search API** (`/v1/search` and
`/v1/search/{search_tool_name}`) that wraps multiple search-provider backends
behind one Perplexity-compatible request/response shape.
Source: https://docs.litellm.ai/docs/search
Config shape in `config.yaml` (LiteLLM's own documented example, Perplexity
shown, same shape for every provider):
```yaml
search_tools:
- search_tool_name: perplexity-search
litellm_params:
search_provider: perplexity
api_key: os.environ/PERPLEXITYAI_API_KEY
```
Call shape:
```bash
curl http://0.0.0.0:4000/v1/search/searxng-search \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{"query": "latest AI developments", "max_results": 5}'
```
Source: https://docs.litellm.ai/docs/search
**18 providers are listed as supported**, including Perplexity, Tavily, Exa
AI, Brave, Parallel AI, Google PSE, DataForSEO, Firecrawl, **SearXNG**,
Linkup, Serper, DuckDuckGo, SearchAPI.io, You.com, APISerpent, Bedrock
AgentCore, Nimble, and Bing Grounding.
Source: https://docs.litellm.ai/docs/search
## 2. Is SearXNG natively supported? — Yes
SearXNG is one of the 18 built-in `search_provider` values, added by
BerriAI/litellm PR #16259 ("[Feat] add serxng search API provider").
Source: https://github.com/BerriAI/litellm/pull/16259
Config shape for SearXNG specifically:
```yaml
search_tools:
- search_tool_name: searxng-search
litellm_params:
search_provider: searxng
api_base: https://your-searxng-instance.com
```
Equivalently, the base URL can be supplied via the `SEARXNG_API_BASE`
environment variable instead of an inline `api_base` key — SearXNG has no API
key of its own (it's an unauthenticated local meta-search engine), so this is
the one provider in the list where `litellm_params` doesn't need a secret.
Sources: https://github.com/BerriAI/litellm/pull/16259,
https://docs.litellm.ai/docs/search
**For this repo**, the addition to `litellm-config.yaml` (research only — not
applied here) would look like:
```yaml
search_tools:
- search_tool_name: searxng-search
litellm_params:
search_provider: searxng
api_base: http://search.home/
```
No custom-endpoint or "generic OpenAI-compatible /v1/web_search" fallback is
needed — the concern in the ticket that LiteLLM's docs "may assume a hosted
provider like Tavily/Serper" turned out not to apply; SearXNG is a first-class
`search_provider` value, same shape as every hosted one.
## 3. Tool-call vs. automatic injection vs. a third thing
The ticket asked to determine whether this is (a) a tool the model must
explicitly call, or (b) automatic pre-retrieval/context-injection like
Perplexity's own search-augmented answers. **It's neither** — it's a
**standalone REST endpoint** that sits alongside `/v1/chat/completions`, not
wired into it:
> The documentation indicates this is a separate REST endpoint the
> application calls directly. The page presents `/search` as a standalone API
> endpoint alongside chat completions, not as an automatic injection feature.
> Users explicitly invoke the search endpoint; LiteLLM does not automatically
> inject search results into completions.
Source: https://docs.litellm.ai/docs/search (fetched content, describing the
`/v1/search` and `/v1/search/{search_tool_name}` endpoints as siblings of
`/v1/chat/completions`, not a chat-completion parameter or automatic
retrieval step)
Practical effect: whatever calls this endpoint — Open WebUI's own web-search
feature, a shell script, a future MCP server — does so with a plain HTTP call.
**LiteLLM's model routing and Qwen3.8-27B's tool-calling reliability are not
in that path at all**, unless something downstream chooses to expose this
endpoint back to the model *as* a function-calling tool (e.g. an MCP wrapper
that hands the model a `web_search` tool definition backed by this endpoint —
that would reintroduce the model-must-emit-a-correct-tool-call problem, but
that's a choice made one layer up, not something LiteLLM's `/v1/search`
feature forces).
## 4. Network reachability: `search.home` from inside the `litellm` container
`docker-compose.yml`'s `litellm` service joins only the `ai-stack` bridge
network (`networks: [ai-stack]`, line 116) and gets DNS resolution from
Docker's embedded DNS server for that network — which resolves other
containers by service/container name (`llama-server`, `qdrant`, etc., as
already used at `api_base: http://llama-server:8080/v1` in
`litellm-config.yaml` line 8) but has **no visibility into the LAN's mDNS/
local-DNS namespace** that resolves `search.home` on the host machine or on
LAN clients. So `http://search.home/` will not resolve from inside the
`litellm` container as configured today — this matches the ticket's
suspicion, and is standard Docker bridge-networking behavior, not specific to
this repo.
Source: `g:\_DEV\repos\LLM-Server\docker-compose.yml` (litellm service, lines
94–124; `networks:` block, lines 154–156)
No `extra_hosts`, `host.docker.internal`, or `network_mode: host` pattern
exists yet anywhere in this compose file to crib from — this would be the
first. (One service, `lazytainer`, already uses `network_mode: host`, but for
an unrelated reason — Docker-socket/host-port introspection — and switching
`litellm` to host networking would be a much bigger blast-radius change than
this ticket needs, dropping the `ai-stack` network isolation for every other
port `litellm` exposes.)
Source: `g:\_DEV\repos\LLM-Server\docker-compose.yml` lines 144–152
**Recommendation: `extra_hosts` on the `litellm` service**, mapping
`search.home` to its LAN IP, e.g.:
```yaml
litellm:
...
extra_hosts:
- "search.home:192.0.2.10" # replace with SearXNG's actual LAN IP
```
This is the smallest, most local fix: one line, scoped to the one service
that needs it, no change to network topology or isolation, and it keeps
`litellm-config.yaml`'s `api_base: http://search.home/` value human-readable
(matching this repo's existing preference for symbolic hostnames like
`ai.home` / `proxy.ai.home` documented in `docs/network-access.md`) rather
than hardcoding the LAN IP directly into the YAML config. The IP needs to stay
in sync if SearXNG's host ever gets a new DHCP lease — same caveat that would
apply to any hardcoded-IP alternative, just isolated to one `extra_hosts`
line instead of buried in the search config.
`host.docker.internal` is not applicable here: that special hostname
resolves to the Docker **host's** own IP (useful for reaching a service
running directly on the host machine's loopback), not to arbitrary LAN mDNS
names — it wouldn't help resolve `search.home` unless SearXNG happens to run
on the same physical host as this compose stack.
## 5. Interaction with this project's known-flaky Qwen3.8-27B tool-calling
`docs/research/qwen3.8-27b-tool-calling.md` (2026-08-24) found, with medium-
to-high confidence, that Qwen3.8-27B's tool-calling through llama.cpp
inherits open/partially-fixed upstream parser bugs from the Qwen3.5 lineage
(issues #21158, #20837 in `ggml-org/llama.cpp`) — tool calls can be emitted
but not recognized, or land as inert XML inside a reasoning block, especially
with thinking enabled.
Source: `g:\_DEV\repos\LLM-Server\docs\research\qwen3.8-27b-tool-calling.md`
(section 3, "Bottom line" section)
Given section 3 above (`/v1/search` is a standalone endpoint, not a
model-tool), **that flakiness has no bearing on the recommended integration
path**: nothing about calling `POST /v1/search/searxng-search` from Open
WebUI or a script asks Qwen3.8-27B to emit a tool call at all. The search
happens (or doesn't) independent of the model's tool-calling grammar/parser
entirely.
The risk **would** resurface only if a *different* design choice is made
later — e.g. wrapping this same SearXNG-backed endpoint as an MCP tool or a
`tools=[...]` function definition handed to Qwen3.8-27B in a chat-completion
request, so the model itself decides when to search. That path would inherit
every bug documented in `qwen3.8-27b-tool-calling.md` (calls silently dropped,
calls trapped inside `<think>` blocks, etc.) and would need the live smoke
test that doc recommends before being trusted unattended. **That is not what
LiteLLM's `/v1/search` feature itself requires** — it's a choice a caller
could additionally make on top of it.
## Recommendation
1. Add a `search_tools` block to `litellm-config.yaml` using
`search_provider: searxng` and `api_base: http://search.home/` (see
section 2) — no custom/generic-endpoint workaround needed, this is a
first-class supported provider.
2. Add `extra_hosts: ["search.home:<LAN IP>"]` to the `litellm` service in
`docker-compose.yml` (see section 4) so the container can resolve the
hostname; confirm the IP is stable (static DHCP reservation) since
`extra_hosts` is a static mapping baked in at container start.
3. Treat `/v1/search` as a plain HTTP integration point, not a model tool —
whatever calls it (Open WebUI, a script) should call the REST endpoint
directly rather than exposing it to Qwen3.8-27B as a function-calling
tool, to avoid inheriting this project's documented tool-calling
flakiness (section 5). If model-driven search-tool-calling is wanted
later, that's a separate decision that should be smoke-tested against the
caveats in `qwen3.8-27b-tool-calling.md` first.
This is research only — `litellm-config.yaml` and `docker-compose.yml` are
not modified by this doc.
## Sources
- https://docs.litellm.ai/docs/search — LiteLLM search feature docs:
endpoints, `search_tools` config shape, provider list, standalone-endpoint
behavior.
- https://github.com/BerriAI/litellm/pull/16259 — SearXNG provider
implementation: `search_provider: searxng`, `api_base` /
`SEARXNG_API_BASE` config.
- `g:\_DEV\repos\LLM-Server\docker-compose.yml` — `litellm` service
definition (lines 94–124), `ai-stack` network block (lines 154–156),
`lazytainer`'s `network_mode: host` precedent (lines 144–152).
- `g:\_DEV\repos\LLM-Server\litellm-config.yaml` — current proxy config
conventions (`model_list`, `litellm_params`, `api_base` usage at line 8).
- `g:\_DEV\repos\LLM-Server\docs\network-access.md` — this repo's existing
`*.home` / NPM hostname conventions.
- `g:\_DEV\repos\LLM-Server\docs\research\qwen3.8-27b-tool-calling.md` —
prior findings on Qwen3.8-27B tool-calling reliability via llama.cpp.
## Confidence/uncertainty summary
- **High confidence:** SearXNG is a native, first-class `search_provider` in
LiteLLM's `/v1/search` feature (directly documented and confirmed via the
implementing PR); the feature is a standalone REST endpoint separate from
chat completions, not automatic context-injection and not itself a
model-callable tool.
- **Medium confidence:** the exact field name (`api_base` vs. relying solely
on `SEARXNG_API_BASE`) — two independent fetch passes against
docs.litellm.ai returned slightly different renderings of the same example
(one showed `api_key: os.environ/SEARXNG_API_BASE`, the other and the PR
fetch showed `api_base: <url>`); the PR-sourced `api_base` form is treated
as authoritative here since it comes from the implementing code change, but
this should be smoke-tested against the actual deployed LiteLLM image
(`ghcr.io/berriai/litellm:main-stable`) before being relied on verbatim.
- **Not independently verified:** SearXNG's actual LAN IP/hostname stability
on this network, and whether the deployed LiteLLM version
(`main-stable`, per `docker-compose.yml` line 95) already includes PR
#16259 — worth a quick `docker exec litellm pip show litellm` / changelog
check before wiring this in for real.