LLM-Server/docs/research/langchain-pgvector-vs-litellm-pgvector.md

Research: LangChain+pgvector-direct RAG vs. the litellm-pgvector connector (for Gitea issue #25)

Question: Is a LangChain + pgvector-direct retrieval architecture (LangChain's PGVector class talking to Postgres directly, LiteLLM used only for provider-abstracted chat/embedding calls) a better fit for this stack than the already-built litellm-pgvector connector service (vendor/litellm-pgvector/, issue #23's resolution)?

Bottom line: yes — switch. Replace litellm-pgvector / vendor/litellm-pgvector/ with a small hand-written FastAPI service (roughly 80–120 lines, one new dependency set: langchain-postgres, langchain-openai, fastapi, uvicorn) that wraps LangChain's PGVector class and calls LiteLLM's OpenAI-compatible /v1/embeddings endpoint for embeddings. This keeps "memory served from the gateway layer" in spirit (LiteLLM still does all chat/embedding model calls; only the retrieval query now goes through a LangChain-based service instead of a vendored 754-line Prisma-backed connector), removes 754 lines of unvetted vendored source and a Prisma migration dependency, and — critically — removes the guessed and unverified vector_store_registry / custom_llm_provider: pg_vector config that issue #24 flagged as never confirmed against a live deploy. pgvector-db (plain pgvector/pgvector:pg16) is reusable completely as-is; only the schema management strategy changes (LangChain manages its own two tables instead of Prisma migrations).

1. Verifying the markaicode article against LangChain's actual docs

The article (https://markaicode.com/stack/litellm-langchain-stack/) proposes three services — LangServe (port 8000, wraps a LangChain chain as REST), LiteLLM (port 4000), and Postgres+pgvector (port 5432) — and shows:

vectorstore = PGVector(
    connection_string="postgresql://user:pass@pgvector:5432/vectordb",
    embedding_function=your_embeddings,
    collection_name="documents",
)

This snippet uses the deprecated class. connection_string and embedding_function are constructor args of langchain_community.vectorstores.pgvector.PGVector, the older integration that ships inside the general-purpose langchain-community package. The current, actively-maintained integration lives in its own package, langchain-postgres, as langchain_postgres.vectorstores.PGVector, with a different constructor:

from langchain_postgres import PGVector
from langchain_openai import OpenAIEmbeddings

vector_store = PGVector(
    embeddings=embeddings,          # not embedding_function
    collection_name="my_docs",
    connection="postgresql+psycopg://user:pass@host:5432/db",  # not connection_string; psycopg3 DSN
    use_jsonb=True,
)

Confirmed constructor signature (embeddings, connection, collection_name, use_jsonb, create_extension, async_mode) via reference.langchain.com/python/langchain-postgres/vectorstores/PGVector and the setup walkthrough at docs.langchain.com/oss/python/integrations/vectorstores/pgvector. Two concrete corrections to the article's shape:

• Package: langchain-postgres is a separate pip install (pip install -qU langchain-postgres), not part of langchain_community. The docs page does not say the community class is formally deprecated, but it is the older/legacy path — the actively-documented integration page for Postgres+pgvector points at langchain_postgres.vectorstores.PGVector.
• Driver: langchain-postgres requires the psycopg (psycopg3) driver and a postgresql+psycopg:// DSN, not psycopg2 / plain postgresql://. This repo's pgvector-db service doesn't care (it's just Postgres), but the new service's requirements.txt needs psycopg[binary], not psycopg2-binary (which vendor/litellm-pgvector/requirements.txt uses).
• create_extension defaults to True — PGVector runs CREATE EXTENSION IF NOT EXISTS vector itself on first connect. Source: same API reference page above. This is the same requirement litellm-pgvector's Prisma schema has (Postgres user needs CREATE privilege) — no change in privilege requirements, just who issues the DDL.
• The article's LangServe wrapper (add_routes()) is a real LangChain component but adds a fourth abstraction (LangServe app → LangChain chain/retriever → PGVector → Postgres) for what this stack needs from only two HTTP endpoints (ingest, query). A plain FastAPI app calling PGVector methods directly is simpler and is what's specced below (§5) — LangServe is overkill for a two-endpoint internal service.

There is also a v2, engine-based API, langchain_postgres.v2.vectorstores.PGVectorStore (with native async via PGEngine/AsyncPGVectorStore), documented at reference.langchain.com/python/integrations/langchain_postgres. It's newer and adds first-class async, but the plain PGVector class is the one shown on LangChain's main pgvector integration page, is simpler to set up (no separate PGEngine object), and this service has no need for the async throughput PGVectorStore targets (single ingest script, occasional queries). Recommendation: use langchain_postgres.vectorstores.PGVector, not PGVectorStore — revisit only if query volume ever demands async throughput.

2. Complexity comparison: vendor/litellm-pgvector vs. a LangChain-direct service

vendor/litellm-pgvector/ as it exists in this repo today:

File	Lines
main.py	521
embedding_service.py	89
models.py	85
config.py	59
prisma/schema.prisma	39
Total (Python + schema)	793

Plus: a Dockerfile that runs prisma generate at build time, a requirements.txt pulling in prisma==0.11.0 (which itself vendors a Rust query-engine binary download at install time — a second network dependency inside the Docker build, on top of the git-context build that already failed once per the vendoring rationale in docker-compose.yml/VENDORED.md), and hand-rolled raw-SQL string building for every endpoint (see main.py lines 86–128, 219–320, 322–402, 405–511 — table/column names are f-string-interpolated from settings.table_names/settings.db_fields, not parameterized, though the values are parameterized). None of this has been smoke-tested against a live deploy (issue #24).

A LangChain-direct replacement needs to expose exactly two HTTP endpoints (POST /ingest, POST /query — see §5) as a thin wrapper around library calls that already do the embedding + SQL work:

• langchain_postgres.PGVector.add_texts(texts, metadatas) — ingestion, replaces main.py's create_embedding/create_embeddings_batch (176 lines) and all of models.py's embedding request/response schemas.
• langchain_postgres.PGVector.similarity_search_with_relevance_scores(query, k) — query, replaces search_vector_store (100 lines) and generate_query_embedding/embedding_service.py (89 lines) — LangChain's OpenAIEmbeddings (from langchain-openai, pointed at base_url=http://litellm:4000/v1) does the embedding call instead of a hand-written httpx call to LiteLLM.
• No create_vector_store/list_vector_stores needed — one Postgres "collection" (LangChain's term, one row in its langchain_pg_collection table) is enough for this repo's single memory-and-notes use case; collection_name="memory-and-notes" passed once at construction covers it.
• No Prisma, no generated query-engine binary, no prisma/schema.prisma — PGVector creates and manages its own two tables (langchain_pg_collection, langchain_pg_embedding) via SQLAlchemy on first use.

Estimated replacement size: ~90–120 lines of Python across the FastAPI app

• Pydantic request/response models (a /ingest and /query endpoint, an Embeddings-backed PGVector instance built once at startup, an API-key Depends() check reused almost verbatim from vendor/litellm-pgvector/main.py lines 50–55) — roughly an 85% reduction from the vendored connector's 793 lines, with zero Prisma/migration surface and no hand-built raw SQL (parameter binding, vector-string formatting, and JSONB metadata handling are all internal to PGVector, not re-implemented per endpoint as in main.py).

This does not eliminate the need for a small custom service — a bare PGVector.similarity_search() call is a Python library call, not an HTTP endpoint, and this stack's callers (the ingest script, Claude Code hooks described in docs/memory-knowledgebase.md) need HTTP. Something still has to be that thin wrapper; it's just ~85% smaller and has no ORM/migration layer.

3. Does this still count as "memory served from the gateway layer"?

Yes, with the same division of responsibility issue #21 established, just a narrower one for the connector. LiteLLM (litellm container) remains the only thing every client (Open WebUI, Claude Code, anything else behind the proxy) talks to for chat and embedding model calls — nothing changes there. What moves is which service holds the retrieval logic: today it's litellm-pgvector (itself calling back into litellm:4000 for embeddings, per docker-compose.yml lines 241–246); under this alternative it's the new LangChain-based service, which also calls back into litellm:4000/v1 for embeddings (via langchain-openai's OpenAIEmbeddings(base_url="http://litellm:4000/v1"), since LiteLLM exposes an OpenAI-compatible API — confirmed by this repo's existing local-embedding model_list entry in litellm-config.yaml already being OpenAI-route-compatible). The client-facing shape is identical: no client (Open WebUI, a Claude Code hook, ingest-memory.sh) talks to Postgres directly, and no client bypasses LiteLLM for the actual embedding/generation model calls. Only the internal retrieval connector's implementation changes, from a vendored third-party FastAPI+Prisma app to an in-repo FastAPI+LangChain app. This does not contradict #21's standing decision.

One nuance: today, vector_store_registry makes LiteLLM itself the thing a chat completion's tools: [{"type": "file_search", ...}] call reaches (per docs/research/litellm-knowledgebase.md §1) — LiteLLM intercepts the tool call and proxies to the registered pg_vector backend. Under the alternative, that specific mechanism goes away: the new service is not a vector_store_registry provider (there is no langchain_postgres/custom provider type in LiteLLM's vector_store_registry, and building one would reintroduce exactly the custom-connector complexity this ticket is trying to shed). Retrieval becomes a direct call to the new service's /query endpoint (or a client-side RAG step: query the service, splice results into the prompt before calling litellm) rather than an in-band file_search tool call LiteLLM resolves itself. This is a real, if modest, capability reduction: {"type": "file_search", "vector_store_ids": [...]} inside a /chat/completions call to litellm stops working; callers move to a two-step "search then send" pattern instead. Given vector_store_registry's exact field shape was never confirmed working (issue #24) and this is the only capability lost, this is judged an acceptable trade for the complexity and reliability win.

4. Ingestion, embedding-server, pgvector-db reuse

embedding-server (llama.cpp, nomic-embed-text-v1.5, docker-compose.yml lines 38–66): fully reusable as-is. It already serves an OpenAI-compatible /v1/embeddings route (llama.cpp's --embeddings flag, per docs/research/litellm-knowledgebase.md §3), and litellm-config.yaml already exposes it through LiteLLM as the local-embedding model (api_base: http://embedding-server:8080/v1). The new service reaches it the same way litellm-pgvector does today — indirectly, by calling http://litellm:4000/v1/embeddings with model=local-embedding — via langchain_openai.OpenAIEmbeddings(model="local-embedding", base_url="http://litellm:4000/v1", api_key=<virtual key>). OpenAIEmbeddings is LangChain's standard Embeddings implementation for any OpenAI-compatible endpoint (no custom Embeddings subclass needed) — confirmed via the constructor example in docs.langchain.com/oss/python/integrations/vectorstores/pgvector, which pairs OpenAIEmbeddings with PGVector directly.

scripts/ingest-memory.sh changes, but modestly: same per-line chunking of data/memory.md/data/claude-legacy-memory.md (the ponytail comment in the current script — one chunk per non-empty, non-heading line, no real chunking logic needed — still holds), but it now curls the new service's POST /ingest instead of litellm-pgvector's two-step POST /v1/vector_stores + POST /v1/vector_stores/{id}/embeddings/batch. Concretely: build one JSON array of {content, metadata} per file (same shape the script already builds) and POST it once to /ingest; the new service calls PGVector.add_texts(texts, metadatas) internally, which embeds and inserts in one call — no separate "create the store" step, since collection_name is fixed at service startup and PGVector creates the collection row implicitly on first add_texts. The no-dedup caveat in the current script's header comment carries over unchanged (add_texts always inserts; nothing in PGVector deduplicates by content).

pgvector-db (pgvector/pgvector:pg16, docker-compose.yml lines 198–214): reusable completely as-is — same image, same "separate Postgres instance from litellm-db" rationale, no compose changes needed to the service definition itself. Only the schema changes: litellm-pgvector's Prisma schema (vendor/litellm-pgvector/prisma/schema.prisma, 39 lines) defines its own vector_stores/embeddings tables with configurable table/field names (config.py's settings.table_names/db_fields), applied via Prisma migrations at container startup (the exact behavior of which, per issue #24's ponytail note, was never smoke-tested). PGVector instead auto-creates two fixed tables — langchain_pg_collection and langchain_pg_embedding — via SQLAlchemy Base.metadata.create_all()-style setup on first connection, no separate migration step or migration tool needed. This is a strictly simpler schema-management story: no Prisma CLI, no prisma generate build step, no migration files to track. The existing litellm_pgvector/PGVECTOR_DB_PASSWORD database and role stay — the new service just needs its own DATABASE_URL pointed at the same database (or a fresh one; reusing litellm_pgvector is simplest since no data has been proven-loaded into it yet per issue #24).

litellm-pgvector (container) and vendor/litellm-pgvector/ (source tree): removed if this verdict is adopted. Nothing else in the repo depends on the vendored source once the new service replaces it.

5. Concrete replacement spec (not implemented — spec only)

New compose service, replacing the litellm-pgvector block:

  memory-retrieval:                     # name TBD; "litellm-pgvector" freed up
    build:
      context: ./services/memory-retrieval    # new small in-repo dir, not vendored
    container_name: memory-retrieval
    depends_on:
      pgvector-db:
        condition: service_healthy
      litellm:
        condition: service_healthy
    environment:
      - DATABASE_URL=postgresql+psycopg://litellm_pgvector:${PGVECTOR_DB_PASSWORD}@pgvector-db:5432/litellm_pgvector
      - LITELLM_BASE_URL=http://litellm:4000/v1
      - LITELLM_API_KEY=${MEMORY_RETRIEVAL_EMBEDDING_KEY}   # new virtual key, same pattern as LITELLM_PGVECTOR_EMBEDDING_KEY today
      - EMBEDDING_MODEL=local-embedding
      - SERVER_API_KEY=${MEMORY_RETRIEVAL_API_KEY}          # replaces LITELLM_PGVECTOR_API_KEY
      - COLLECTION_NAME=memory-and-notes
    expose:
      - "8000"
    restart: unless-stopped
    networks: [ai-stack]

Note DATABASE_URL changes scheme to postgresql+psycopg:// (psycopg3 DSN langchain-postgres requires — see §1), not the postgresql:// scheme litellm-pgvector uses today.

Dockerfile shape (services/memory-retrieval/Dockerfile): same base image family as today's connector — python:3.11-slim — pip install the requirements, copy the app, CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]. No prisma generate step, no build-essential/postgresql-client needed at the OS-package level (no Prisma binary to build against; psycopg[binary] ships prebuilt wheels).

requirements.txt: fastapi, uvicorn[standard], langchain-postgres, langchain-openai, psycopg[binary], pydantic — five packages vs. the current ten in vendor/litellm-pgvector/requirements.txt, and no prisma.

Endpoints (2, vs. today's 5 — create store, list stores, search, create embedding, create embeddings batch):

• POST /ingest — body: {"chunks": [{"content": "...", "metadata": {...}}]}. Calls PGVector.add_texts(texts=[...], metadatas=[...]). Replaces both ingest-memory.sh's "create store" call (no longer needed — collection is implicit) and its batch-embeddings call.
• POST /query — body: {"query": "...", "k": 5}. Calls PGVector.similarity_search_with_relevance_scores(query, k=k), returns {"results": [{"content": ..., "metadata": ..., "score": ...}]}.
• GET /health — same trivial liveness check the current connector has (main.py lines 514–517), kept for compose depends_on/healthcheck use if desired.

Bearer-token auth via SERVER_API_KEY, same Depends() pattern as vendor/litellm-pgvector/main.py lines 47–55 — that part of the existing connector is fine as-is and can be copied over close to verbatim.

litellm-config.yaml change: remove the vector_store_registry block entirely (lines 47–61 today) — per §3, the new service isn't a vector_store_registry provider, so there's nothing to register. The local-embedding model_list entry (lines 25–35) is unchanged and still needed — it's what the new service's OpenAIEmbeddings client calls through litellm.

6. Verdict

Switch to LangChain-direct. This does invalidate part of issue #23's resolution and the litellm-pgvector container currently in docker-compose.yml — say so explicitly for scoping a follow-up implementation ticket:

• Torn out: litellm-pgvector service block in docker-compose.yml, vendor/litellm-pgvector/ (entire tree), the vector_store_registry block in litellm-config.yaml, LITELLM_PGVECTOR_API_KEY/ LITELLM_PGVECTOR_EMBEDDING_KEY env vars (replaced by new-service equivalents).
• Survives unchanged: pgvector-db (compose block as-is), embedding-server (compose block as-is), the local-embedding model_list entry in litellm-config.yaml, the overall "LiteLLM does all model calls" gateway boundary from issue #21.
• New: a small in-repo services/memory-retrieval/ FastAPI+LangChain app (~90–120 lines, spec in §5), a new compose service block, a rewritten scripts/ingest-memory.sh pointed at /ingest instead of litellm-pgvector's two-step API, and one new virtual key (MEMORY_RETRIEVAL_EMBEDDING_KEY) issued the same way LITELLM_PGVECTOR_EMBEDDING_KEY was per docs/proxy-key-onboarding.md.
• Capability trade-off (§3): the OpenAI Assistants-style file_search tool call directly against litellm's /chat/completions stops working (no vector_store_registry entry to resolve it); callers move to querying the new service directly and splicing results into the prompt themselves. This mechanism was never confirmed working in the first place (issue #24), so it's a loss of unverified capability, not working functionality.

This is not implemented here — this document is research/spec only, per the ticket's scope. A follow-up implementation ticket should: write services/memory-retrieval/, update docker-compose.yml and litellm-config.yaml per §5, rewrite scripts/ingest-memory.sh, delete vendor/litellm-pgvector/, and update docs/memory-knowledgebase.md.

7. Repo context read for this research

• docker-compose.yml — embedding-server, pgvector-db, litellm-pgvector service definitions (build context, env vars, network).
• litellm-config.yaml — local-embedding model entry, vector_store_registry block.
• docs/research/litellm-knowledgebase.md — prior research establishing LiteLLM's native vector-store feature has no Qdrant provider and that litellm-pgvector was the only self-hosted path found at the time; its §1, §3, §4 are cited above for the file_search tool-call mechanism and embedding-model wiring.
• docs/memory-knowledgebase.md — current user-facing setup doc for the knowledgebase/memory feature.
• scripts/ingest-memory.sh — current ingestion mechanism and its no-dedup caveat.
• vendor/litellm-pgvector/{main.py,embedding_service.py,models.py,config.py, prisma/schema.prisma,requirements.txt,Dockerfile} — read in full to characterize size (793 total lines Python+schema) and shape (raw-SQL f-string-built queries per endpoint, Prisma-managed schema, calls back to LiteLLM's /embeddings for query embedding).
• Gitea issue #25 (this ticket) and its parent #21 — standing decision that memory is served from the gateway layer, not a client.
• Gitea issue #23/#24 (referenced by #25) — original litellm-pgvector adoption and its unverified-against-live-deploy caveat.

8. Primary sources consulted

• markaicode.com/stack/litellm-langchain-stack/ — the alternative surfaced by the user; verified and corrected above (§1).
• reference.langchain.com/python/langchain-postgres/vectorstores/PGVector — PGVector constructor signature, create_extension default.
• docs.langchain.com/oss/python/integrations/vectorstores/pgvector — install instructions, postgresql+psycopg:// DSN requirement, OpenAIEmbeddings + PGVector pairing example.
• github.com/langchain-ai/langchain-postgres — package README; async-function-pairing note; PGVector deprecated in v0.0.14+ in favor of PGVectorStore (v2 API).
• reference.langchain.com/python/integrations/langchain_postgres/ — v2 PGVectorStore/PGEngine/AsyncPGVectorStore API, compared against PGVector in §1.
• vendor/litellm-pgvector/README.md and source files (this repo) — connector's existing endpoint/config shape, cited for the LOC comparison in §2.