haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

LangChain+pgvector direct RAG vs. the litellm-pgvector connector — which fits this stack? #25

New Issue
Closed
opened 2026-09-02 19:55:46 +00:00 by haylan · 2 comments
haylan commented 2026-09-02 19:55:46 +00:00
Owner
Copy Link
Copy Source

Part of #21

Question

The already-implemented knowledgebase path (#23's resolution: litellm-pgvector companion service, vendored into vendor/litellm-pgvector/ after a remote git-context build failed on the server — see docker-compose.yml, commits f508f56/7247674) is unpopular with the user ("does not work for me, I don't like it") and heavier than it needs to be: a 20KB custom FastAPI connector, Prisma migrations, and guessed vector_store_registry/pg_vector config fields never confirmed against a live deploy.

A simpler alternative surfaced: https://markaicode.com/stack/litellm-langchain-stack/ describes a LiteLLM + LangChain + pgvector stack that skips a custom connector entirely — LangChain's PGVector class talks to Postgres/pgvector directly for retrieval, and LiteLLM is used only for its actual job (provider-abstracted chat/embedding calls over HTTP, base_url="http://litellm:4000/v1"). No vector_store_registry, no bespoke connector service, no vendored third-party source tree.

Resolve: is the LangChain+pgvector-direct architecture actually a better fit for this stack than the litellm-pgvector connector path already built? Specifically:

  • Does it still deliver "memory served from LiteLLM" in spirit (LiteLLM remains the model-facing gateway; only routes through LangChain add retrieval) — check this doesn't contradict the standing decision in #21 that memory must be served from the gateway layer, not a client.
  • What would data/memory.md/data/claude-legacy-memory.md ingestion, and the embedding-server (already built, nomic-embed-text-v1.5) look like against LangChain's PGVector instead of litellm-pgvector's batch-embeddings endpoint?
  • What does this mean for the embedding-server, pgvector-db services already in docker-compose.yml — reusable as-is, or does the schema/connection story change?
  • Concretely: does adopting this mean removing litellm-pgvector and vendor/litellm-pgvector/ and replacing them with a small LangChain-based retrieval service (or a LangServe app, per the article) — sketch what that replacement service actually needs to be (a real service definition, not just "use LangChain").
  • Verify the markaicode article's claims against LangChain's actual current docs (langchain-postgres / PGVector package) before committing to anything — don't take a blog post's shape at face value.

If this comes back in favor of the LangChain-direct path, it invalidates part of #23's resolution and the current litellm-pgvector implementation on main — say so explicitly and scope what needs to be torn out vs. kept (pgvector-db, embedding-server likely survive; litellm-pgvector likely doesn't).

Part of #21 ## Question The already-implemented knowledgebase path (#23's resolution: `litellm-pgvector` companion service, vendored into `vendor/litellm-pgvector/` after a remote git-context build failed on the server — see docker-compose.yml, commits f508f56/7247674) is unpopular with the user ("does not work for me, I don't like it") and heavier than it needs to be: a 20KB custom FastAPI connector, Prisma migrations, and guessed `vector_store_registry`/`pg_vector` config fields never confirmed against a live deploy. A simpler alternative surfaced: https://markaicode.com/stack/litellm-langchain-stack/ describes a **LiteLLM + LangChain + pgvector** stack that skips a custom connector entirely — LangChain's `PGVector` class talks to Postgres/pgvector directly for retrieval, and LiteLLM is used only for its actual job (provider-abstracted chat/embedding calls over HTTP, `base_url="http://litellm:4000/v1"`). No `vector_store_registry`, no bespoke connector service, no vendored third-party source tree. **Resolve**: is the LangChain+pgvector-direct architecture actually a better fit for this stack than the `litellm-pgvector` connector path already built? Specifically: - Does it still deliver "memory served from LiteLLM" in spirit (LiteLLM remains the model-facing gateway; only routes through LangChain add retrieval) — check this doesn't contradict the standing decision in #21 that memory must be served from the gateway layer, not a client. - What would `data/memory.md`/`data/claude-legacy-memory.md` ingestion, and the embedding-server (already built, nomic-embed-text-v1.5) look like against LangChain's `PGVector` instead of `litellm-pgvector`'s batch-embeddings endpoint? - What does this mean for the `embedding-server`, `pgvector-db` services already in `docker-compose.yml` — reusable as-is, or does the schema/connection story change? - Concretely: does adopting this mean **removing** `litellm-pgvector` and `vendor/litellm-pgvector/` and replacing them with a small LangChain-based retrieval service (or a LangServe app, per the article) — sketch what that replacement service actually needs to be (a real service definition, not just "use LangChain"). - Verify the markaicode article's claims against LangChain's actual current docs (`langchain-postgres` / `PGVector` package) before committing to anything — don't take a blog post's shape at face value. If this comes back in favor of the LangChain-direct path, it invalidates part of #23's resolution and the current `litellm-pgvector` implementation on main — say so explicitly and scope what needs to be torn out vs. kept (pgvector-db, embedding-server likely survive; litellm-pgvector likely doesn't).
haylan added the wayfinder:research label 2026-09-02 19:55:46 +00:00
haylan commented 2026-09-02 20:03:17 +00:00
Author
Owner
Copy Link
Copy Source

Verdict: switch to LangChain-direct. Full research + a concrete replacement-service spec: docs/research/langchain-pgvector-vs-litellm-pgvector.md on branch research/langchain-pgvector-vs-litellm-pgvector (committed locally, not pushed/merged — research only).

  • Complexity: vendor/litellm-pgvector/ is 793 lines (Python + Prisma schema) of hand-rolled raw-SQL endpoints plus a Prisma migration/codegen build step (a second network dependency in the Docker build, on top of the git-context build that already failed once). A LangChain-based replacement — a thin FastAPI wrapper around langchain_postgres.PGVector (verified against current LangChain docs, not the deprecated langchain_community class the markaicode article actually shows) — is an estimated ~90-120 lines: 2 endpoints (/ingest, /query) instead of 5, no ORM/migrations (PGVector auto-creates its own 2 tables), no unverified raw-SQL field-name config.
  • Gateway architecture holds: LiteLLM stays the only model-facing gateway for chat/embeddings — the new service still calls back into http://litellm:4000/v1/embeddings (via langchain-openai's OpenAIEmbeddings) exactly like litellm-pgvector does today. Only the retrieval connector's internals change; no client talks to Postgres directly. One real capability loss: the OpenAI-style file_search tool call against /chat/completions goes away (no vector_store_registry entry for a non-native provider) — but that mechanism was never confirmed working (issue #24) in the first place.
  • embedding-server / pgvector-db: both reusable completely as-is. embedding-server's /v1/embeddings route is already OpenAI-compatible and already wired through LiteLLM as local-embedding. pgvector-db is plain pgvector/pgvector:pg16 — same image, only the schema-management story changes (PGVector's implicit table creation vs. Prisma migrations).
  • Removed if adopted: litellm-pgvector service block in docker-compose.yml, all of vendor/litellm-pgvector/, the vector_store_registry block in litellm-config.yaml, LITELLM_PGVECTOR_* env vars.
  • scripts/ingest-memory.sh changes to POST once to the new service's /ingest instead of litellm-pgvector's two-step create-store + batch-embeddings calls; chunking logic (one fact/paragraph per line) is unchanged.

This invalidates part of #23's resolution — scoped explicitly in the doc's §6 (what's torn out vs. survives). Not implemented here per this ticket's scope; a follow-up implementation ticket should build services/memory-retrieval/ per the doc's §5 spec (compose block, Dockerfile, requirements, env vars, endpoints all sketched there).

**Verdict: switch to LangChain-direct.** Full research + a concrete replacement-service spec: `docs/research/langchain-pgvector-vs-litellm-pgvector.md` on branch `research/langchain-pgvector-vs-litellm-pgvector` (committed locally, not pushed/merged — research only). - **Complexity**: `vendor/litellm-pgvector/` is 793 lines (Python + Prisma schema) of hand-rolled raw-SQL endpoints plus a Prisma migration/codegen build step (a second network dependency in the Docker build, on top of the git-context build that already failed once). A LangChain-based replacement — a thin FastAPI wrapper around `langchain_postgres.PGVector` (verified against current LangChain docs, not the deprecated `langchain_community` class the markaicode article actually shows) — is an estimated ~90-120 lines: 2 endpoints (`/ingest`, `/query`) instead of 5, no ORM/migrations (PGVector auto-creates its own 2 tables), no unverified raw-SQL field-name config. - **Gateway architecture holds**: LiteLLM stays the only model-facing gateway for chat/embeddings — the new service still calls back into `http://litellm:4000/v1/embeddings` (via `langchain-openai`'s `OpenAIEmbeddings`) exactly like `litellm-pgvector` does today. Only the retrieval connector's internals change; no client talks to Postgres directly. One real capability loss: the OpenAI-style `file_search` tool call against `/chat/completions` goes away (no `vector_store_registry` entry for a non-native provider) — but that mechanism was never confirmed working (issue #24) in the first place. - **embedding-server / pgvector-db**: both reusable completely as-is. `embedding-server`'s `/v1/embeddings` route is already OpenAI-compatible and already wired through LiteLLM as `local-embedding`. `pgvector-db` is plain `pgvector/pgvector:pg16` — same image, only the schema-management story changes (PGVector's implicit table creation vs. Prisma migrations). - **Removed if adopted**: `litellm-pgvector` service block in `docker-compose.yml`, all of `vendor/litellm-pgvector/`, the `vector_store_registry` block in `litellm-config.yaml`, `LITELLM_PGVECTOR_*` env vars. - **`scripts/ingest-memory.sh`** changes to POST once to the new service's `/ingest` instead of `litellm-pgvector`'s two-step create-store + batch-embeddings calls; chunking logic (one fact/paragraph per line) is unchanged. This invalidates part of #23's resolution — scoped explicitly in the doc's §6 (what's torn out vs. survives). Not implemented here per this ticket's scope; a follow-up implementation ticket should build `services/memory-retrieval/` per the doc's §5 spec (compose block, Dockerfile, requirements, env vars, endpoints all sketched there).
haylan commented 2026-09-02 20:09:49 +00:00
Author
Owner
Copy Link
Copy Source

Implemented — switched from litellm-pgvector to a LangChain-direct service (services/memory-retrieval/), per this ticket's verdict. Details in the commit on main (abeadc4): new memory-retrieval compose service replaces litellm-pgvector, vendor/litellm-pgvector/ removed, vector_store_registry removed from litellm-config.yaml, scripts/ingest-memory.sh rewritten for the new /ingest endpoint, .env vars renamed (MEMORY_RETRIEVAL_API_KEY/MEMORY_RETRIEVAL_EMBEDDING_KEY), docs/memory-knowledgebase.md updated.

This partially supersedes #23's original resolution — pgvector-db and embedding-server survive as decided there, but the connector approach doesn't. Still unverified on real hardware — rolled into #24.

Implemented — switched from litellm-pgvector to a LangChain-direct service (`services/memory-retrieval/`), per this ticket's verdict. Details in the commit on main (`abeadc4`): new `memory-retrieval` compose service replaces `litellm-pgvector`, `vendor/litellm-pgvector/` removed, `vector_store_registry` removed from `litellm-config.yaml`, `scripts/ingest-memory.sh` rewritten for the new `/ingest` endpoint, `.env` vars renamed (`MEMORY_RETRIEVAL_API_KEY`/`MEMORY_RETRIEVAL_EMBEDDING_KEY`), `docs/memory-knowledgebase.md` updated. This partially supersedes #23's original resolution — `pgvector-db` and `embedding-server` survive as decided there, but the connector approach doesn't. Still unverified on real hardware — rolled into #24.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:research
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: haylan/LLM-Server#25