docs(research): confirm lazytainer/omniroute idle-stop conflict (#40)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015zCwaWJQuKgXDUfRPBqDS7
This commit is contained in:
@@ -0,0 +1,194 @@
|
|||||||
|
# Research: Why lazytainer's idle-stop on llama-server doesn't fire, and what switch-model.sh should do about it
|
||||||
|
|
||||||
|
**Question:** lazytainer is configured on `llama-server` (`docker-compose.yml`
|
||||||
|
`lazytainer.group.llamaserver.*` labels) but its idle-stop never triggers in
|
||||||
|
practice — OmniRoute appears to keep the container looking "active" to
|
||||||
|
lazytainer's packet-threshold detector. Confirm the mechanism, find root
|
||||||
|
cause, and recommend how the future `scripts/switch-model.sh` (#43, blocked)
|
||||||
|
should handle GPU-residency swaps between `llama-server` and a new `comfyui`
|
||||||
|
service given this.
|
||||||
|
|
||||||
|
**Answer:** Confirmed. lazytainer's detector is a dumb per-port packet
|
||||||
|
counter with no traffic classification — it cannot tell OmniRoute's
|
||||||
|
background provider health-check pings apart from real inference traffic,
|
||||||
|
and there is no config knob in lazytainer or a per-provider one in OmniRoute
|
||||||
|
that fixes this. **`switch-model.sh` should bypass lazytainer entirely** for
|
||||||
|
the swap: drive `docker compose stop`/`up -d` directly on both services,
|
||||||
|
rather than trying to make lazytainer's idle-stop cooperate.
|
||||||
|
|
||||||
|
## Current config (`docker-compose.yml`)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
labels:
|
||||||
|
- "lazytainer.group.llamaserver.sleepMethod=stop"
|
||||||
|
- "lazytainer.group.llamaserver.ports=8080"
|
||||||
|
- "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
|
||||||
|
- "lazytainer.group.llamaserver.minPacketThreshold=2"
|
||||||
|
```
|
||||||
|
|
||||||
|
`ports=8080` matches the container's real internal port (`expose: ["8080"]`,
|
||||||
|
confirmed in the same file) — not a misconfiguration. `minPacketThreshold=2`
|
||||||
|
is already far *below* lazytainer's own documented default of `30`, i.e. this
|
||||||
|
deployment already tried loosening the threshold to make idle-stop easier to
|
||||||
|
reach, not harder.
|
||||||
|
|
||||||
|
## How lazytainer's detector actually works (primary source: `vmorganp/Lazytainer`)
|
||||||
|
|
||||||
|
Confirmed against the project's README and Go source
|
||||||
|
(`src/group.go`) on [github.com/vmorganp/Lazytainer](https://github.com/vmorganp/Lazytainer):
|
||||||
|
|
||||||
|
- It captures packets with **gopacket/libpcap directly on the configured
|
||||||
|
`netInterface`** (default `eth0`), applying a BPF filter built from the
|
||||||
|
group's `ports` list (`"port 8080"` here, per the source's filter-string
|
||||||
|
construction, e.g. `"port 80 or port 81 or etc."` in the general case).
|
||||||
|
- The filter matches **every packet to or from the port** — SYN, ACK,
|
||||||
|
data, FIN, everything. It is not restricted to new-connection SYNs.
|
||||||
|
- Every `pollRate` seconds (default `30`; not overridden in this repo's
|
||||||
|
config) it samples a rolling packet counter (`rxHistory`) and compares the
|
||||||
|
delta against `minPacketThreshold`:
|
||||||
|
`rxHistory[0]+minPacketThreshold > rxHistory[len(rxHistory)-1]` → treated as
|
||||||
|
active, `inactiveSeconds` resets to 0.
|
||||||
|
- `ignoreActiveClients` (default `false`, not set here) only changes whether
|
||||||
|
an ESTABLISHED-connection count is also checked; it does not add any
|
||||||
|
content- or source-based filtering.
|
||||||
|
- **There is no mechanism anywhere in lazytainer to exclude specific traffic
|
||||||
|
(by source IP, path, header, or request type) from the packet count.** The
|
||||||
|
README's config table (`ports`, `inactiveTimeout`, `minPacketThreshold`,
|
||||||
|
`ignoreActiveClients`, `pollRate`, `sleepMethod`, `netInterface`) is
|
||||||
|
exhaustive — nothing else exists to tune this per-caller.
|
||||||
|
|
||||||
|
Consequence: a single TCP connection to port 8080 — a bare connect + one
|
||||||
|
small HTTP exchange + close — already produces well over `minPacketThreshold=2`
|
||||||
|
packets purely from the handshake and teardown (SYN, SYN-ACK, ACK, ..., FIN,
|
||||||
|
ACK), regardless of payload size or purpose. At this threshold, essentially
|
||||||
|
*any* connection to the port counts as "active" and resets `inactiveTimeout`.
|
||||||
|
Raising the threshold wouldn't help either — the fix would need to be
|
||||||
|
"ignore packets from OmniRoute's health-checker," which the tool has no way
|
||||||
|
to express; it only counts packets on a port, source-blind.
|
||||||
|
|
||||||
|
## How OmniRoute actually touches registered providers (primary source: `diegosouzapw/OmniRoute`)
|
||||||
|
|
||||||
|
Confirmed against
|
||||||
|
[`docs/reference/ENVIRONMENT.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/ENVIRONMENT.md)
|
||||||
|
in the OmniRoute repo:
|
||||||
|
|
||||||
|
- OmniRoute runs a **background credential/connection health-check
|
||||||
|
scheduler** (`src/lib/credentialHealth/scheduler.ts`) on
|
||||||
|
`CREDENTIAL_HEALTH_CHECK_INTERVAL`, default `300000` ms (5 min), minimum
|
||||||
|
`10000` ms (10s) — this periodically re-tests each registered provider's
|
||||||
|
connection, which for a provider like `llama-server` (a plain HTTP base
|
||||||
|
URL, no API key) means an actual request/connection to
|
||||||
|
`llama-server:8080`.
|
||||||
|
- Results are cached for `CREDENTIAL_HEALTH_CACHE_TTL` (default also 5 min).
|
||||||
|
- **Only one exclusion exists, and it's hardcoded by provider category, not
|
||||||
|
configurable per-provider**: search providers
|
||||||
|
(`SEARCH_VALIDATOR_CONFIGS` in
|
||||||
|
`src/lib/providers/validation/searchProviders.ts`, e.g. `tavily-search`)
|
||||||
|
are permanently skipped because their validation call is a real billed
|
||||||
|
upstream query. `llama-server` is an inference provider, not a search
|
||||||
|
provider — it is not in this exclusion list.
|
||||||
|
- The only toggle that actually stops the sweep is global:
|
||||||
|
`OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK=1`/`true`, which "disable[s]
|
||||||
|
background periodic testing of provider connections" for **every**
|
||||||
|
registered provider at once. There is no documented per-provider
|
||||||
|
disable/pause flag in
|
||||||
|
[`docs/reference/PROVIDER_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md) —
|
||||||
|
the dashboard's `/dashboard/providers` page is described only as where you
|
||||||
|
"enable, configure, and test each provider," with no documented
|
||||||
|
independent "pause health checks for this one provider" control.
|
||||||
|
|
||||||
|
So: OmniRoute is not the sole cause, but it is a live, recurring cause. Every
|
||||||
|
5 minutes (at most — could also be triggered ad hoc by dashboard/API use) it
|
||||||
|
opens a connection to `llama-server:8080` purely to check the provider is
|
||||||
|
alive, which is exactly the kind of traffic lazytainer's port-level counter
|
||||||
|
cannot distinguish from real inference calls. With `inactiveTimeout=900`
|
||||||
|
(15 min) and a health-check every ≤300s, the container practically always
|
||||||
|
sees qualifying traffic before its idle timer would expire.
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
|
||||||
|
Two independent, both-true facts combine to defeat idle-stop:
|
||||||
|
|
||||||
|
1. **lazytainer's detector is fundamentally traffic-blind** — it counts raw
|
||||||
|
packets on a port with no way to exclude any specific caller or traffic
|
||||||
|
class. This is a property of the tool, not a misconfiguration in this
|
||||||
|
repo (`ports=8080` is correct; `minPacketThreshold=2` is already at the
|
||||||
|
permissive end).
|
||||||
|
2. **OmniRoute periodically pings every registered non-search provider**
|
||||||
|
(default every ≤5 min) to keep its health/availability status current,
|
||||||
|
and that ping is indistinguishable, at the packet level, from a real
|
||||||
|
inference request.
|
||||||
|
|
||||||
|
Neither side offers a targeted fix: lazytainer has no allowlist/denylist by
|
||||||
|
source, and OmniRoute's only "stop pinging" lever
|
||||||
|
(`OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK`) is all-or-nothing across every
|
||||||
|
provider, not scoped to just `llama-server`. Tuning `minPacketThreshold`
|
||||||
|
higher or lower doesn't change the outcome either way, since the health-check
|
||||||
|
traffic and real traffic land on the exact same port with no distinguishing
|
||||||
|
packet-level signature.
|
||||||
|
|
||||||
|
## Recommendation for `scripts/switch-model.sh` (#43)
|
||||||
|
|
||||||
|
**Bypass lazytainer entirely for the GPU-residency swap.** Drive both
|
||||||
|
services directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose stop llama-server
|
||||||
|
docker compose up -d comfyui
|
||||||
|
# ...and the reverse when swapping back
|
||||||
|
```
|
||||||
|
|
||||||
|
Justification:
|
||||||
|
|
||||||
|
- The swap is a **deliberate, scripted, known-in-advance** event — the
|
||||||
|
script always knows exactly which service should go up and which should
|
||||||
|
go down. Idle-stop detection exists to handle the case where nobody knows
|
||||||
|
when a service last had traffic; that's not this case, so routing the
|
||||||
|
swap through a passive heuristic (lazytainer's idle timer) that this
|
||||||
|
research shows is already unreliable for `llama-server` adds a point of
|
||||||
|
failure for no benefit. Direct `docker compose stop`/`up -d` is
|
||||||
|
deterministic and immune to the packet-counting confound described above.
|
||||||
|
- Reconfiguring lazytainer's thresholds was considered and rejected: no
|
||||||
|
threshold value fixes a detector that cannot distinguish OmniRoute's
|
||||||
|
keepalive traffic from real traffic on the same port (see Root cause).
|
||||||
|
This is a ceiling in the tool itself, not a tuning problem.
|
||||||
|
- Pausing OmniRoute's polling for the swap window was also considered.
|
||||||
|
It's the one lever available (`OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK`),
|
||||||
|
but it is global — it would blind OmniRoute's health status for *every*
|
||||||
|
provider (including `searxng-search`, if registered) for the duration of
|
||||||
|
the swap, and adds an extra env-toggle-and-restart step to the script for
|
||||||
|
a problem that direct compose control sidesteps completely. It's worth
|
||||||
|
flagging for #43's implementation as a *secondary* safety measure — briefly
|
||||||
|
disabling the sweep (or accepting that OmniRoute may show `llama-server` as
|
||||||
|
errored/offline for up to `CREDENTIAL_HEALTH_CHECK_INTERVAL` after it's
|
||||||
|
stopped) — but it should not be the primary mechanism the swap relies on.
|
||||||
|
- This does **not** require removing the existing `lazytainer.group.llamaserver.*`
|
||||||
|
labels — they can stay for whatever idle-stop benefit they still provide
|
||||||
|
between swaps (e.g. genuinely idle periods where nothing, including
|
||||||
|
OmniRoute, has recently touched the container long enough to matter) while
|
||||||
|
`switch-model.sh` simply never depends on lazytainer to do the actual
|
||||||
|
stop/start for a swap.
|
||||||
|
|
||||||
|
## Bottom line for #43 (blocked ticket, once unblocked)
|
||||||
|
|
||||||
|
- `switch-model.sh` should call `docker compose stop <from-service>` /
|
||||||
|
`docker compose up -d <to-service>` directly — never rely on lazytainer's
|
||||||
|
idle-stop to free the GPU as part of a swap.
|
||||||
|
- No lazytainer config change (threshold, ports, poll rate) is a viable fix;
|
||||||
|
the detector has no way to exclude OmniRoute's traffic by source.
|
||||||
|
- Optionally, as a secondary hygiene step, the script may toggle
|
||||||
|
`OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK` around the swap (or simply
|
||||||
|
tolerate a stale "errored" status in OmniRoute's dashboard for up to one
|
||||||
|
`CREDENTIAL_HEALTH_CHECK_INTERVAL`) to avoid OmniRoute flagging the
|
||||||
|
just-stopped provider as failed mid-swap — but this is cosmetic/status
|
||||||
|
hygiene, not what makes the swap itself work.
|
||||||
|
|
||||||
|
Sources: [`vmorganp/Lazytainer`](https://github.com/vmorganp/Lazytainer)
|
||||||
|
(README config table; `src/group.go` packet-capture and threshold-comparison
|
||||||
|
logic), [`diegosouzapw/OmniRoute` —
|
||||||
|
`docs/reference/ENVIRONMENT.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/ENVIRONMENT.md)
|
||||||
|
(credential health-check scheduler env vars), [`diegosouzapw/OmniRoute` —
|
||||||
|
`docs/reference/PROVIDER_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md)
|
||||||
|
(provider dashboard controls), this repo's `docker-compose.yml`
|
||||||
|
(`lazytainer.group.llamaserver.*` labels, `llama-server`/`omniroute` service
|
||||||
|
definitions).
|
||||||
Reference in New Issue
Block a user