Building RAG Systems with LangChain and Vector Databases

Introduction

Retrieval-Augmented Generation, commonly called RAG, has become one of the most practical ways to build intelligent applications with large language models. Instead of relying solely on the parametric knowledge embedded in an LLM, a RAG system retrieves relevant external documents at query time and feeds them into the model context. This approach reduces hallucination, enables factual grounding, and allows organizations to query private data without fine-tuning.

In this article we focus on building RAG systems with LangChain and vector databases. LangChain provides a composable framework for chaining prompts, retrievers, and models. Vector databases store embeddings and enable fast similarity search. Together they form the backbone of most modern knowledge assistants, support bots, and document Q&A tools.

By the end of this guide you will understand the core concepts, see a reference architecture, build a working pipeline with code, compare vector stores, and learn production best practices for deployment, security, and performance.

Table of Contents

• Core Concepts
• Architecture Overview
• Step-by-Step Guide
• Real-World Examples
• Production Code Examples
• Comparison Table
• Best Practices
• Common Mistakes
• Performance Tips
• Security Considerations
• Deployment Notes
• Debugging Tips
• FAQ
• Conclusion

Core Concepts

Before writing code, you should understand the building blocks of a RAG system. Each component has a distinct responsibility and can be swapped without rewriting the entire application.

Embeddings

Embeddings are dense vector representations of text produced by a model such as OpenAI text-embedding-3-small or open-source alternatives like all-MiniLM. A sentence and its semantically similar sentence will have nearby vectors in high-dimensional space. This property enables similarity search.

Vector Database

A vector database stores these embeddings along with metadata and provides approximate nearest neighbor search. Popular options include pgvector for PostgreSQL, Pinecone, Weaviate, Qdrant, and Milvus. The database returns the most relevant chunks for a given query embedding.

Chunking

Documents are split into smaller chunks before embedding. Chunk size affects retrieval quality. Too small loses context; too large adds noise and exceeds context limits. A common starting point is 500 to 1000 tokens with overlap.

Retriever

The retriever takes a user question, embeds it, queries the vector store, and returns the top-k documents. LangChain abstracts this through the VectorStoreRetriever interface.

Generator

The generator is the LLM. It receives a system prompt, the retrieved context, and the user question, then produces a grounded answer.

Architecture Overview

A typical RAG architecture contains an ingestion pipeline and a query pipeline. The ingestion pipeline runs offline or on a schedule. The query pipeline runs per user request.

Ingestion pipeline: Source documents are fetched, cleaned, split into chunks, embedded via an embedding model, and written to a vector database. Metadata such as source URL, timestamp, and owner is stored alongside each vector.

Query pipeline: The user question is embedded using the same model. The vector database returns relevant chunks. A prompt template assembles the question and context. The LLM generates the answer. Optionally, a reranker improves precision, and citations are returned to the user.

LangChain sits in the middle as the orchestration layer. It connects loaders, text splitters, embedding models, vector stores, prompts, and chat models using declarative chains or the newer LangChain Expression Language (LCEL).

Step-by-Step Guide

We will build a minimal but realistic RAG system using LangChain, OpenAI embeddings, and pgvector. You can replace pgvector with Pinecone by changing the vector store class.

Step 1: Install Dependencies

pip install langchain langchain-openai langchain-postgres pgvector psycopg2-binary tiktoken

Step 2: Set Environment Variables

export OPENAI_API_KEY="sk-..."export DATABASE_URL="postgresql://user:pass@localhost:5432/ragdb"

Step 3: Enable pgvector

CREATE EXTENSION IF NOT EXISTS vector;CREATE TABLE documents (  id UUID PRIMARY KEY,  content TEXT,  embedding vector(1536),  metadata JSONB);CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

Step 4: Ingest Documents

from langchain_community.document_loaders import TextLoaderfrom langchain_text_splitters import RecursiveCharacterTextSplitterfrom langchain_openai import OpenAIEmbeddingsfrom langchain_postgres import PGVectorloader = TextLoader("company_handbook.txt")docs = loader.load() splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)chunks = splitter.split_documents(docs)embeddings = OpenAIEmbeddings(model="text-embedding-3-small")vector_store = PGVector(    embeddings=embeddings,    collection_name="handbook",    connection=DATABASE_URL,    use_jsonb=True,)vector_store.add_documents(chunks)print("Ingested", len(chunks), "chunks")

Step 5: Query the System

from langchain_core.prompts import ChatPromptTemplatefrom langchain_openai import ChatOpenAIfrom langchain_core.output_parsers import StrOutputParserfrom langchain_core.runnables import RunnablePassthroughretriever = vector_store.as_retriever(search_kwargs={"k": 4})prompt = ChatPromptTemplate.from_template("""You are a helpful assistant. Use the context to answer.Context:{context}Question: {question}""")llm = ChatOpenAI(model="gpt-4o-mini")def format_docs(docs):    return "".join(d.page_content for d in docs)rag_chain = (    {"context": retriever | format_docs, "question": RunnablePassthrough()}    | prompt    | llm    | StrOutputParser())answer = rag_chain.invoke("What is the remote work policy?")print(answer)

Real-World Examples

Customer support assistants use RAG to answer policy questions from internal wikis. Legal teams query contracts by embedding clauses and retrieving similar language across thousands of PDFs. Engineering organizations build internal docs bots that ground answers in the latest runbooks.

In a healthcare setting, a RAG system can retrieve clinical guidelines and present them to a physician with citations. In finance, compliance teams use RAG to map new regulations against existing controls. The pattern is the same; only the data sources and access controls change.

Production Code Examples

Below is a more robust ingestion function with metadata and idempotent upserts.

import uuidfrom langchain_core.documents import Documentdef ingest_file(path: str, source: str):    loader = TextLoader(path)    raw = loader.load()    chunks = splitter.split_documents(raw)    docs = []    for c in chunks:        docs.append(Document(            page_content=c.page_content,            metadata={                "source": source,                "chunk_id": str(uuid.uuid4()),                "doc_hash": hash(c.page_content),            }        ))    vector_store.add_documents(docs)    return len(docs)

This example adds a reranking step using a cross-encoder for higher precision.

from langchain.retrievers import ContextualCompressionRetrieverfrom langchain.retrievers.document_compressors import CrossEncoderRerankerfrom langchain_community.cross_encoders import HuggingFaceCrossEncodermodel = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-base")compressor = CrossEncoderReranker(model=model, top_n=3)compression_retriever = ContextualCompressionRetriever(    base_compressor=compressor, base_retriever=retriever)

Comparison Table

Choosing a vector database depends on scale, latency, and operational preference. The table below compares common options.

Best Practices

• Use the same embedding model for ingestion and query.
• Store source metadata to enable citations and audits.
• Chunk by semantic boundaries where possible, not arbitrary sizes.
• Add a reranker when precision matters more than latency.
• Log queries and retrieved chunks to evaluate quality.
• Version your prompt templates and embedding models.

Common Mistakes

• Switching embedding models between ingest and query, causing mismatch.
• Embedding entire large documents without chunking.
• Ignoring metadata filtering, returning irrelevant namespaces.
• Overstuffing context with too many chunks, increasing cost.
• Not evaluating answers against a golden set.

Performance Tips

Use IVF or HNSW indexes appropriate to your store. In pgvector, tune lists based on row count. Cache frequent queries and their retrieved contexts. Reduce chunk overlap if storage grows too fast. Use smaller embedding models when acceptable. Batch embedding calls during ingestion to reduce API round trips.

Security Considerations

RAG systems expose your data to the retrieval and generation path. Apply row-level metadata filters so users only retrieve documents they are authorized to see. Never log raw API keys. Redact PII before embedding if compliance requires. Rate limit the query endpoint to prevent abuse and cost spikes. Validate and sanitize user input before placing it in prompts to reduce injection risk.

Deployment Notes

Run ingestion as a scheduled job or event-driven worker. Deploy the query API behind an authenticated gateway. Use a connection pool for Postgres. Monitor token usage per request. Set timeouts on LLM calls. Containerize the service and scale horizontally behind a load balancer.

Debugging Tips

If answers are wrong, first inspect retrieved chunks. Use LangChain tracing or simple print statements to view context. Verify embedding dimensions match the table definition. Check that the retriever returns non-empty results. Test the prompt with fixed context to isolate model vs retrieval issues.

FAQ

What is RAG with LangChain?

RAG with LangChain is the practice of using LangChain components to retrieve relevant documents from a vector store and pass them into an LLM prompt for grounded generation.

Which vector database should I use?

If you already use PostgreSQL, start with pgvector. If you need managed scale without ops, Pinecone is a strong choice. Qdrant and Weaviate are good self-hosted options.

Do I need to fine-tune the LLM?

No. RAG grounds the model through context. Fine-tuning is optional and usually unnecessary for knowledge retrieval use cases.

How large should chunks be?

A practical range is 500 to 1000 tokens with 10 to 20 percent overlap. Tune based on evaluation results.

Can I use open-source embeddings?

Yes. Models like all-MiniLM or BGE are viable. Ensure you use the same model for ingest and query.

How do I add citations?

Store source metadata with each chunk and return it alongside the answer. Prompt the model to reference source labels.

What reduces hallucination most?

High-quality retrieval, a clear prompt instruction to use only context, and showing the model when context is insufficient.

Is RAG expensive?

Costs come from embedding ingestion and LLM tokens. Caching and smaller models help. Vector search itself is cheap.

How do I evaluate a RAG system?

Build a golden question set with expected answers. Measure faithfulness, context relevance, and answer correctness using manual or automated scoring.

Conclusion

Building RAG systems with LangChain and vector databases is a repeatable, high-value pattern for grounding LLMs in real data. Start with a simple pipeline, measure retrieval quality, then add reranking, metadata filtering, and monitoring as needed. If you want to go further, read our guide on AI Agents with OpenAI and design your RAG service as a composable microservice for ML workloads.