Introduction to RAG
Retrieval-Augmented Generation (RAG) combines the generative power of LLMs with the precision of information retrieval. Instead of relying solely on what a model learned during training, RAG pulls in relevant, up-to-date documents at query time and feeds them to the LLM as context — giving you accurate, grounded answers on your own data.
Why RAG?
| Problem with plain LLMs | How RAG fixes it |
|---|
| Knowledge cutoff — no data after training | Retrieves current documents at query time |
| Hallucinations — confident but wrong answers | Grounds answers in retrieved source text |
| No access to private / company data | Indexes your own documents in a vector store |
| Context window limits on long documents | Retrieves only the relevant chunks, not everything |
| Expensive fine-tuning to add new knowledge | Just re-index — no model retraining needed |
The RAG pipeline
1. Load
- PDF, Web, CSV, DB
- Document loaders
- Any data source
2. Split
- Chunk into segments
- Preserve context
- Overlap at borders
3. Embed
- Convert text → vector
- Capture semantic meaning
- OpenAI / local models
4. Store
- Save in vector DB
- Chroma, FAISS, Pinecone
- Indexed for fast search
5. Retrieve
- User asks a question
- Embed question
- Similarity search → top-k chunks
6. Generate
- LLM + question + chunks
- Grounded answer
- Can cite sources
Two phases: indexing vs querying
| Phase | When | Steps |
|---|
| Indexing | Once (or on data update) | Load → Split → Embed → Store |
| Querying | Every user request | Embed question → Retrieve → Generate |
Minimal end-to-end RAG
pip install langchain langchain-openai langchain-chroma langchain-community
pip install pypdf python-dotenv tiktoken
import os
from dotenv import load_dotenv
load_dotenv() # OPENAI_API_KEY from .env
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
# ── 1. INDEXING PHASE ──────────────────────────────────────
docs = PyPDFLoader("handbook.pdf").load()
chunks = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200).split_documents(docs)
db = Chroma.from_documents(chunks, OpenAIEmbeddings(), persist_directory="./db")
# ── 2. QUERYING PHASE ──────────────────────────────────────
retriever = db.as_retriever(search_kwargs={"k": 4})
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("system", "Answer using ONLY the context below. If unsure, say 'I don't know'.\n\nContext:\n{context}"),
("human", "{question}"),
])
rag_chain = (
{"context": retriever | (lambda docs: "\n\n".join(d.page_content for d in docs)),
"question": RunnablePassthrough()}
| prompt | llm | StrOutputParser()
)
print(rag_chain.invoke("What is the vacation policy?"))
RAG use cases
| Use case | Documents indexed |
|---|
| Enterprise Q&A bot | Internal wikis, Confluence, HR policies |
| Customer support | Product docs, FAQs, troubleshooting guides |
| Legal / compliance | Contracts, regulations, case law |
| Code assistant | Codebase, API docs, architecture docs |
| Medical reference | Research papers, clinical guidelines |
| Financial analysis | Earnings reports, SEC filings, market data |
RAG vs fine-tuning
| RAG | Fine-tuning |
|---|
| Cost | Low — just indexing | High — GPU compute |
| Update data | Re-index in minutes | Re-train (hours/days) |
| Hallucination control | High — grounded in docs | Low |
| Teach new skills | No | Yes |
| Best for | Knowledge, facts, documents | Style, format, tasks |
Start simple: A basic RAG pipeline — PDF → split → Chroma → retrieve → GPT-4o-mini — often outperforms fine-tuned models on factual Q&A tasks and takes under an hour to build. Only reach for advanced techniques (re-ranking, HyDE, CRAG) after measuring where your baseline falls short.