OmniRoute: memory/knowledgebase parity with litellm-pgvector #34

New Issue

2026-09-03T17:17:58Z

haylan commented

2026-09-03 17:17:58 +00:00

Part of #31

Question

Current setup: litellm-pgvector (vendored, vendor/litellm-pgvector/) backed by pgvector-db, holding a memory-and-notes vector store loaded from data/memory.md/data/claude-legacy-memory.md via scripts/ingest-memory.sh, queried via the OpenAI Assistants-style file_search tool or POST /v1/vector_stores/{id}/search — see docs/memory-knowledgebase.md.

OmniRoute advertises "optional int8 vector quantization-based memory with typed decay, opt-in per-request with x-omniroute-no-memory" but its docs frame it as not a RAG platform.

Determine:

Can OmniRoute's built-in memory ingest this stack's existing fact files and be queried the same way (assistants-style tool call, or an equivalent), using the local embedding-server for embeddings (no cloud embedding API)?
Does "typed decay" mean facts silently expire — is that acceptable for a knowledgebase meant to persist (vs. LiteLLM's flat store)?
If parity isn't there: does OmniRoute support pointing at an external Postgres+pgvector store (keep pgvector-db + litellm-pgvector, just re-pointed), or does the knowledgebase have to move onto Qdrant (already running for Open WebUI) instead?

Answer feeds the "Not yet specified" fog item on whether vendor/litellm-pgvector/ is dropped or kept.

Part of #31 ## Question Current setup: `litellm-pgvector` (vendored, `vendor/litellm-pgvector/`) backed by `pgvector-db`, holding a `memory-and-notes` vector store loaded from `data/memory.md`/`data/claude-legacy-memory.md` via `scripts/ingest-memory.sh`, queried via the OpenAI Assistants-style `file_search` tool or `POST /v1/vector_stores/{id}/search` — see `docs/memory-knowledgebase.md`. OmniRoute advertises "optional int8 vector quantization-based memory with typed decay, opt-in per-request with `x-omniroute-no-memory`" but its docs frame it as not a RAG platform. Determine: - Can OmniRoute's built-in memory ingest this stack's existing fact files and be queried the same way (assistants-style tool call, or an equivalent), using the local `embedding-server` for embeddings (no cloud embedding API)? - Does "typed decay" mean facts silently expire — is that acceptable for a knowledgebase meant to persist (vs. LiteLLM's flat store)? - If parity isn't there: does OmniRoute support pointing at an external Postgres+pgvector store (keep `pgvector-db` + `litellm-pgvector`, just re-pointed), or does the knowledgebase have to move onto Qdrant (already running for Open WebUI) instead? Answer feeds the "Not yet specified" fog item on whether `vendor/litellm-pgvector/` is dropped or kept.

haylan added the wayfinder:research label 2026-09-03 17:17:58 +00:00

haylan self-assigned this 2026-09-03 17:19:42 +00:00

haylan commented

2026-09-03 17:20:46 +00:00

No parity — keep vendor/litellm-pgvector as-is. Researched directly from github.com/diegosouzapw/OmniRoute (docs/frameworks/MEMORY.md and docs/frameworks/LOCAL_CORPUS_CONTEXT.md, raw.githubusercontent.com, current as of release v3.8.51).

1. Ingestion / query parity — no. OmniRoute's memory has no bulk fact-file import. Ingestion is: (a) regex-based auto-extraction from assistant replies (not LLM-based, non-blocking via setImmediate), (b) manual POST /api/memory of single records, or (c) MCP tools (omniroute_memory_add/_search/_clear). There's no assistants-style file_search tool or equivalent bulk endpoint — retrieval is query-driven (chat-injected system message, POST /api/memory/retrieve-preview, or the MCP search tool), not file-centric. A separate feature, Local Corpus Context (docs/frameworks/LOCAL_CORPUS_CONTEXT.md), does read local .md files from an approved directory, but it's keyword/line-scoped text search only (no embeddings, no vector index) exposed via MCP tools, not file_search. Embedding backends for memory are transformers (local ONNX MiniLM, in-process), static (cached lookup), or remote (OpenAI/Cohere/Voyage cloud APIs) — no documented way to point it at an arbitrary OpenAI-compatible endpoint like our llama.cpp embedding-server.

2. 'Typed decay' — marketing overstatement, not a real mechanism. No type-based decay rules exist (e.g. nothing makes 'episodic' facts expire faster than 'factual' ones). Actual expiry is: an optional per-record expiresAt (only if the caller sets it), plus a default 30-day retentionDays cutoff that silently excludes older entries from retrieval (created_at >= cutoff), plus opt-in manual summarize-and-delete of old entries. So by default, facts do go stale after 30 days unless retentionDays is raised — not acceptable for a knowledgebase meant to persist indefinitely, and not how our flat litellm-pgvector store behaves today.

3. External backend — Qdrant only, not Postgres/pgvector. Memory storage is a 3-tier resolution: Tier 0 SQLite FTS5 (always on), Tier 1 sqlite-vec (local), Tier 2 optional external Qdrant (host/port/collection/API key, with health checks and auto-degrade if unhealthy). Postgres+pgvector is not supported as a backend anywhere in the memory system.

Conclusion: Keep vendor/litellm-pgvector/pgvector-db for the fact-file knowledgebase — OmniRoute's memory is a different feature (auto-captured conversational memory with a retention cutoff), not a RAG/file-ingestion platform, confirming its own docs' framing. If OmniRoute is adopted for other reasons, its memory's optional Tier-2 backend is Qdrant, not pgvector — so if this knowledgebase ever needed to leave litellm-pgvector, the only OmniRoute-native external path is Qdrant, not a re-pointed pgvector-db.

No parity — keep vendor/litellm-pgvector as-is. Researched directly from github.com/diegosouzapw/OmniRoute (docs/frameworks/MEMORY.md and docs/frameworks/LOCAL_CORPUS_CONTEXT.md, raw.githubusercontent.com, current as of release v3.8.51). **1. Ingestion / query parity — no.** OmniRoute's memory has no bulk fact-file import. Ingestion is: (a) regex-based auto-extraction from assistant replies (not LLM-based, non-blocking via setImmediate), (b) manual `POST /api/memory` of single records, or (c) MCP tools (`omniroute_memory_add`/`_search`/`_clear`). There's no assistants-style `file_search` tool or equivalent bulk endpoint — retrieval is query-driven (chat-injected system message, `POST /api/memory/retrieve-preview`, or the MCP search tool), not file-centric. A separate feature, Local Corpus Context (`docs/frameworks/LOCAL_CORPUS_CONTEXT.md`), does read local `.md` files from an approved directory, but it's keyword/line-scoped text search only (no embeddings, no vector index) exposed via MCP tools, not file_search. Embedding backends for memory are `transformers` (local ONNX MiniLM, in-process), `static` (cached lookup), or `remote` (OpenAI/Cohere/Voyage cloud APIs) — no documented way to point it at an arbitrary OpenAI-compatible endpoint like our llama.cpp embedding-server. **2. 'Typed decay' — marketing overstatement, not a real mechanism.** No type-based decay rules exist (e.g. nothing makes 'episodic' facts expire faster than 'factual' ones). Actual expiry is: an optional per-record `expiresAt` (only if the caller sets it), plus a default 30-day `retentionDays` cutoff that silently excludes older entries from retrieval (`created_at >= cutoff`), plus opt-in manual summarize-and-delete of old entries. So by default, facts do go stale after 30 days unless retentionDays is raised — not acceptable for a knowledgebase meant to persist indefinitely, and not how our flat litellm-pgvector store behaves today. **3. External backend — Qdrant only, not Postgres/pgvector.** Memory storage is a 3-tier resolution: Tier 0 SQLite FTS5 (always on), Tier 1 sqlite-vec (local), Tier 2 optional external Qdrant (host/port/collection/API key, with health checks and auto-degrade if unhealthy). Postgres+pgvector is not supported as a backend anywhere in the memory system. **Conclusion:** Keep `vendor/litellm-pgvector`/`pgvector-db` for the fact-file knowledgebase — OmniRoute's memory is a different feature (auto-captured conversational memory with a retention cutoff), not a RAG/file-ingestion platform, confirming its own docs' framing. If OmniRoute is adopted for other reasons, its memory's optional Tier-2 backend is Qdrant, not pgvector — so if this knowledgebase ever needed to leave litellm-pgvector, the only OmniRoute-native external path is Qdrant, not a re-pointed pgvector-db.

haylan closed this issue

2026-09-03 17:20:50 +00:00

haylan referenced this issue

2026-09-03 17:21:18 +00:00

Migrate the AI gateway from LiteLLM to OmniRoute #31

haylan referenced this issue

2026-09-03 17:38:36 +00:00

OmniRoute: docker-compose service design and secrets/config plan #37

haylan referenced this issue

2026-09-03 17:49:53 +00:00

OmniRoute: docker-compose service design and secrets/config plan #37

Sign in to join this conversation.