RAG Systems: Building Knowledge Bases That AI Can Actually Use

The RAG revolution

Retrieval-Augmented Generation (RAG) has become the standard pattern for building AI systems that can answer questions about your specific data. Instead of relying on an LLM’s training data, RAG retrieves relevant documents from your knowledge base and uses them to ground the LLM’s response.

But building a RAG system that actually works in production is harder than the tutorials suggest. Let me share what I’ve learned from building dozens of RAG systems.

The RAG pipeline

A production RAG system has these components:

1. Document processing: Load, clean, and chunk documents
2. Embedding: Convert chunks into vector representations
3. Storage: Store vectors in a vector database
4. Retrieval: Find relevant chunks for a given query
5. Reranking: Reorder retrieved chunks by relevance
6. Generation: Use the LLM to generate an answer from the retrieved chunks
7. Citation: Link the answer back to source documents

Chunking strategies

Chunking is the most underrated part of RAG. How you split your documents determines how well your retrieval works.

Fixed-size chunking

The simplest approach — split text into fixed-size chunks (e.g., 500 tokens) with some overlap.

def chunk_text(text, chunk_size=500, overlap=100):
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - overlap):
        chunk = ' '.join(words[i:i + chunk_size])
        chunks.append(chunk)
    if i + chunk_size < len(words):
        break
    return chunks

Pros: Simple, predictable Cons: Can break sentences and paragraphs awkwardly

Semantic chunking

Split text at natural boundaries — paragraphs, sections, or topic changes. This preserves context better.

def semantic_chunk(text):
    # Split by double newlines (paragraphs)
    paragraphs = text.split('\n\n')
    
    chunks = []
    current_chunk = ""
    for para in paragraphs:
        if len(current_chunk) + len(para) < 2000:
            current_chunk += para + '\n\n'
        else:
            if current_chunk:
                chunks.append(current_chunk.strip())
            current_chunk = para + '\n\n'
    
    if current_chunk:
        chunks.append(current_chunk.strip())
    
    return chunks

My recommendation

For most use cases, I use 500-token chunks with 100-token overlap. This provides a good balance between specificity and context. For structured documents (like knowledge base articles), I chunk by section headers.

Choosing a vector database

Pinecone

• Pros: Fully managed, fast, easy to use
• Cons: Can be expensive at scale, less control
• Best for: Getting started quickly, teams without infrastructure experience

Weaviate

• Pros: Open source, hybrid search built-in, GraphQL API
• Cons: More complex to set up
• Best for: Teams that want hybrid search and self-hosting

pgvector (PostgreSQL)

• Pros: Uses your existing PostgreSQL, ACID compliance, no new infrastructure
• Cons: Slower than dedicated vector databases at scale
• Best for: Projects already using PostgreSQL, smaller datasets

My recommendation

Start with Pinecone for simplicity. If cost becomes an issue, migrate to pgvector (if you already use PostgreSQL) or self-hosted Weaviate.

Embedding models

The embedding model you choose significantly impacts retrieval quality.

OpenAI text-embedding-3-small

• Dimensions: 1536
• Cost: $0.02 per 1M tokens
• Quality: Good for most use cases
• Best for: Getting started, general-purpose applications

OpenAI text-embedding-3-large

• Dimensions: 3072
• Cost: $0.13 per 1M tokens
• Quality: Better than small, especially for complex queries
• Best for: Production systems where quality matters

Open-source models (BGE, E5)

• Cost: Free (but you pay for compute)
• Quality: Comparable to OpenAI for many use cases
• Best for: Data privacy requirements, cost-sensitive applications

The retrieval step

Pure vector search

def retrieve(query, top_k=20):
    query_embedding = embed(query)
    results = pinecone.query(
        vector=query_embedding,
        top_k=top_k,
        include_metadata=True
    )
    return results.matches

Hybrid search (recommended)

Combine vector search with keyword search for better results:

def hybrid_retrieve(query, top_k=20):
    # Vector search
    query_embedding = embed(query)
    vector_results = pinecone.query(
        vector=query_embedding,
        top_k=top_k * 2,
        include_metadata=True
    )
    
    # Keyword search (BM25)
    keyword_results = bm25_search(query, top_k=top_k * 2)
    
    # Merge and deduplicate
    merged = merge_results(vector_results, keyword_results)
    return merged[:top_k]

Reranking: the secret weapon

Reranking is the single biggest improvement you can make to a RAG system. After retrieving top-20 chunks, use a cross-encoder model to rerank them by relevance.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def rerank(query, chunks, top_k=5):
    pairs = [(query, chunk.text) for chunk in chunks]
    scores = reranker.predict(pairs)
    
    ranked = sorted(zip(chunks, scores), key=lambda x: x[1], reverse=True)
    return [chunk for chunk, score in ranked[:top_k]]

In my experience, reranking improves answer quality by 20-30%. It’s the difference between a good RAG system and a great one.

Generating grounded answers

The final step is generating an answer using the retrieved and reranked chunks:

def generate_answer(query, chunks):
    context = '\n\n'.join([
        f"[{i+1}] {chunk.text}\nSource: {chunk.metadata['source']}"
        for i, chunk in enumerate(chunks)
    ])
    
    prompt = f"""Answer the question based on the following context. 
    If the context doesn't contain the answer, say "I don't have enough information to answer this question."
    Always cite your sources using [number] notation.
    
    Context:
    {context}
    
    Question: {query}
    
    Answer:"""
    
    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1
    )
    
    return response.choices[0].message.content

Measuring RAG quality

How do you know if your RAG system is good? I track these metrics:

1. Retrieval accuracy: Are the right documents being retrieved? (Measure with a test set)
2. Answer accuracy: Are the answers correct? (Human evaluation on a sample)
3. Hallucination rate: How often does the system make things up?
4. Citation accuracy: Are the cited sources actually relevant to the answer?
5. Latency: How long does the full pipeline take?

Conclusion

Building a production RAG system requires attention to every step of the pipeline — chunking, embedding, retrieval, reranking, and generation. The biggest improvements come from the unglamorous parts: good chunking, hybrid search, and reranking. Start simple, measure everything, and iterate.