What I learned building a production RAG pipeline

Retrieval-Augmented Generation looks deceptively simple in a tutorial: embed some text, drop it in a vector store, stuff the top matches into a prompt. Getting it to behave in production — across English, Arabic, and Urdu — is a different exercise entirely.

When building the RAG pipeline for sajjad.ai, a platform serving diverse administrative and academic documentation, the basic tutorial approaches broke down immediately. Here are the core engineering insights from shipping a resilient, high-trust RAG system.

Chunking is a product decision

The naive approach is to split documents every N characters or words. That destroys meaning at sentence boundaries and returns fragmented sentences to the model. I moved to structure-aware chunking that respects headings, lists, and paragraphs, with a sliding window overlap so context is not lost at the seams.

For technical documentation, a simple Markdown-header splitter works wonders. Here is a TypeScript example of how we split text by headers while keeping parent headings attached to the chunk context:

interface DocumentChunk {
  content: string;
  metadata: {
    headers: string[];
    sourceFile: string;
  };
}

export function chunkMarkdown(text: string, sourceFile: string): DocumentChunk[] {
  const lines = text.split("\n");
  const chunks: DocumentChunk[] = [];
  let currentHeaders: string[] = [];
  let currentChunkLines: string[] = [];

  for (const line of lines) {
    const headerMatch = line.match(/^(#{1,6})\s+(.*)$/);
    if (headerMatch) {
      // If we have accumulated text, save it as a chunk first
      if (currentChunkLines.length > 0) {
        chunks.push({
          content: currentChunkLines.join("\n").trim(),
          metadata: { headers: [...currentHeaders], sourceFile }
        });
        currentChunkLines = [];
      }
      
      const level = headerMatch[1].length;
      const headerText = headerMatch[2].trim();
      
      // Update header stack based on level
      currentHeaders = currentHeaders.slice(0, level - 1);
      currentHeaders[level - 1] = headerText;
    } else {
      currentChunkLines.push(line);
    }
  }

  // Push final chunk
  if (currentChunkLines.length > 0) {
    chunks.push({
      content: currentChunkLines.join("\n").trim(),
      metadata: { headers: [...currentHeaders], sourceFile }
    });
  }

  return chunks;
}

Retrieval quality beats model size

A bigger model cannot fix bad context. Most of the wins in output quality came from improving what we retrieved:

1. Hybrid Keyword-plus-Semantic Search: BM25 handles exact matches (serial numbers, names, specific codes) which embeddings often wash out. Semantic search handles conceptual matches.
2. Cross-Encoder Re-ranking: We use a fast Bi-Encoder (like BGE-M3) for the initial top-50 retrieval, then pass candidates through a Cross-Encoder (like Cohere Rerank or BGE-Rerank) to select the final top-5. The accuracy boost is massive.

Ground every answer, and say when you cannot

The fastest way to lose trust is a confident, wrong answer. We tackled this with three rules:

• Explicit citations: The prompt requires the model to reference source document IDs inline (e.g. [Doc 1]).
• Low-confidence fallbacks: If the cosine similarity score of the top retrieved chunks drops below a threshold (e.g., 0.65), we bypass the generator and return: “I’m sorry, I cannot find relevant information in the provided documents to answer that question.”
• Strict negative constraints: Instructing the model: “If the answer cannot be fully derived from the provided context, state that you do not know. Do not use external knowledge.”

Takeaway

RAG is a systems engineering problem, not a prompt engineering trick. Treat chunking, retrieval pipeline routing, and validation constraints as first-class software architecture. The LLM is the last and smallest part of the stack.